Video abnormal event detection method and device driven by prototype memory neural network

Through the prototype memory neural network, learn the motion prototype features, match and aggregate features from normal videos, and generate predicted frames, which solves the shortcomings of existing video anomaly detection methods in identifying unknown exceptions and using time contexts, and improves the accuracy of detection and robustness.

CN119068388APending Publication Date: 2024-12-03SUN YAT SEN UNIVERSITY SHENZHEN +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411062989.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing video anomaly detection methods are not accurate when identifying no abnormal events, and the generalization ability of deep neural networks leads to large prediction errors and fails to fully utilize the diversity of time context and normal patterns.

Method used

Prototype memory neural network is adopted to obtain short video sequences and long video sequences, extract dynamic features and build prototype modules, match prototype features and memory features, generate predicted frames, and perform model training to improve the accuracy of video anomaly detection.

Benefits of technology

By learning the motion prototype features in normal videos, the ability to generalize abnormalities is reduced, the utilization of time context is enhanced, and the accuracy and robustness of video anomaly detection are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068388A_ABST
    Figure CN119068388A_ABST
Patent Text Reader

Abstract

The invention discloses a prototype memory neural network driven video abnormal event detection method and device. The method comprises the following steps: acquiring a short video sequence, a long video sequence and a target frame; extracting long sequence dynamic characteristics according to the long video sequence; extracting short sequence dynamic characteristics according to the short video sequence; constructing a prototype module according to the dynamic characteristics of the long sequence; matching the short sequence dynamic features with prototype features of a prototype module; extracting memory features and hidden features according to the short video sequence; performing aggregation operation on the matched prototype features, memory features and hidden features to generate a prediction frame; performing model training according to the target frame, the prediction frame, the prototype module and the short sequence dynamic features to obtain a target model; and inputting the short video sequence into the target model to obtain a video exception score. The method can improve the accuracy of video anomaly detection, and can be widely applied to the technical field of video anomaly detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a method and device for detecting video anomaly events driven by a prototype memory neural network. Background Art

[0002] Current video anomaly detection can be divided into weakly supervised and unsupervised methods, but weakly supervised video anomaly detection can only identify abnormal events included in the training set, but cannot detect unseen anomalies, and although time-intensive video temporal annotations are not required, a large number of abnormal samples still need to be collected. In unsupervised methods, the anomaly detector is only trained using normal data to learn normal patterns. During the testing phase, the learned model will identify behaviors that exceed the learned normal patterns as abnormal events. With the recent progress in deep learning, deep neural network-based methods in video anomaly detection allow reconstruction or prediction of anomalies, but due to the powerful generalization ability of deep neural networks, it cannot guarantee a larger prediction error for abnormal behaviors. In addition, due to the small information gap of ordinary frame predictions, prediction-based methods cannot explicitly utilize temporal context, and the diversity of some normal patterns is also ignored, resulting in low accuracy in video anomaly detection. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a method and device for detecting video abnormal events driven by a prototype memory neural network, in order to solve at least one problem of the prior art. The present invention can improve the accuracy of video abnormality detection.

[0004] To achieve the above object, an embodiment of the present invention provides a method for detecting abnormal video events driven by a prototype memory neural network, the method comprising the following steps:

[0005] Obtain short video sequences, long video sequences and target frames;

[0006] Extracting long sequence dynamic features according to the long video sequence;

[0007] Extracting short sequence dynamic features according to the short video sequence;

[0008] Constructing a prototype module according to the long sequence dynamic characteristics;

[0009] Matching the short sequence dynamic features with the prototype features of the prototype module;

[0010] Extracting memory features and hidden features according to the short video sequence;

[0011] Aggregating the matched prototype features, memory features, and hidden features to generate a prediction frame;

[0012] Performing model training according to the target frame, the predicted frame, the prototype module and the short sequence dynamic features to obtain a target model;

[0013] The short video sequence is input into the target model to obtain a video anomaly score.

[0014] In some embodiments, extracting long sequence dynamic features according to the long video sequence comprises the following steps:

[0015] Acquire a first residual sequence of the long video sequence;

[0016] The first residual sequence is input into a first dynamic feature encoder to obtain a long sequence dynamic feature.

[0017] In some embodiments, extracting short sequence dynamic features according to the short video sequence comprises the following steps:

[0018] Obtaining a second residual sequence of the short video sequence;

[0019] The second residual sequence is input into a second dynamic feature encoder to obtain a short sequence dynamic feature.

[0020] In some embodiments, constructing a prototype module according to the long sequence dynamic features comprises the following steps:

[0021] Acquire a first local dynamic feature of the long sequence dynamic feature;

[0022] constructing a first matching matrix;

[0023] The first local dynamic feature is matched with the prototype feature through the first vector of the first matching matrix to obtain a prototype module.

[0024] In some embodiments, matching the short sequence dynamic feature with the prototype feature of the prototype module comprises the following steps:

[0025] Acquire a second local dynamic feature of the short sequence dynamic feature;

[0026] constructing a second matching matrix;

[0027] The second local dynamic feature is matched with the prototype feature through the second vector in the second matching matrix to obtain the matched prototype feature.

[0028] In some embodiments, extracting memory features and hidden features according to the short video sequence includes the following steps:

[0029] Input each frame of the short video sequence into a spatial encoder to extract features and obtain appearance features;

[0030] The appearance features are input into the spatiotemporal aggregation module to perform a temporal relationship capture operation, and a memory feature and a hidden feature are output.

[0031] In some embodiments, performing an aggregation operation on the matched prototype features, memory features, and hidden features to generate a prediction frame includes the following steps:

[0032] Inputting the matched prototype features and the memory features into a multi-layer perceptron to generate channel attention;

[0033] Obtaining channel-refined prototype features according to the matched prototype features and the channel attention;

[0034] Connecting the channel-refined prototype features and the hidden features to obtain aggregate features;

[0035] The aggregated features are input into a frame decoder to generate a predicted frame.

[0036] In some embodiments, model training is performed according to the target frame, the predicted frame, the prototype module, and the short sequence dynamic features to obtain a target model, comprising the following steps:

[0037] Constructing a prediction loss function, inputting the target frame and the prediction frame into the prediction loss function to obtain a prediction loss value;

[0038] Constructing a prototype dispersion loss function, inputting all prototype items in the prototype module and the mean vector of the prototype items into the prototype dispersion loss function to obtain a prototype dispersion loss value;

[0039] Constructing a feature compactness loss function, obtaining a second local dynamic feature of the short sequence dynamic feature, and obtaining a related prototype item of the second local dynamic feature, inputting the second local dynamic feature and the related prototype item into the feature compactness loss function, and obtaining a feature compactness loss value;

[0040] Constructing a first loss function according to the prediction loss function, the prototype dispersion loss function and the feature compactness loss function;

[0041] Optimizing the first dynamic feature encoder and the prototype module according to the first loss function;

[0042] Constructing a second loss function according to the prediction loss function and the feature compactness loss function;

[0043] According to the second loss function, the second dynamic feature encoder is optimized to obtain a target model.

[0044] To achieve the above object, another aspect of an embodiment of the present invention provides a video abnormal event detection device driven by a prototype memory neural network, the device comprising:

[0045] The first module is used to obtain short video sequences, long video sequences and target frames;

[0046] The second module is used to extract long sequence dynamic features according to the long video sequence;

[0047] The third module is used to extract short sequence dynamic features according to the short video sequence;

[0048] The fourth module is used to construct a prototype module according to the long sequence dynamic characteristics;

[0049] A fifth module is used to match the short sequence dynamic features with the prototype features of the prototype module;

[0050] A sixth module is used to extract memory features and hidden features according to the short video sequence;

[0051] The seventh module is used to perform an aggregation operation on the matched prototype features, memory features and hidden features to generate a prediction frame;

[0052] An eighth module is used to perform model training according to the target frame, the predicted frame, the prototype module and the short sequence dynamic feature to obtain a target model;

[0053] The ninth module is used to input the short video sequence into the target model to obtain a video anomaly score.

[0054] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned prototype memory neural network driven video abnormal event detection method.

[0055] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned prototype memory neural network driven video abnormal event detection method.

[0056] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the aforementioned video abnormal event detection method driven by a prototype memory neural network.

[0057] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a method and device for detecting abnormal video events driven by a prototype memory neural network, which obtains a short video sequence, a long video sequence and a target frame; extracts long sequence dynamic features according to the long video sequence; extracts short sequence dynamic features according to the short video sequence; constructs a prototype module according to the long sequence dynamic features; matches the short sequence dynamic features with the prototype features of the prototype module; extracts memory features and hidden features according to the short video sequence; performs aggregation operations on the matched prototype features, memory features and hidden features to generate a prediction frame; performs model training according to the target frame, the prediction frame, the prototype module and the short sequence dynamic features to obtain a target model; inputs the short video sequence into the target model to obtain a video abnormality score. The present invention learns representative motion prototype features from normal videos through a prototype module, matches the normal prototype features in the prototype module with the motion features of the input video sequence to reconstruct, and then uses the reconstructed features to predict the target frame, which can reduce the generalization ability of the model to abnormalities and enhance the model's use of temporal context, thereby improving the accuracy of video abnormality detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 It is a flow chart of a method for detecting abnormal video events driven by a prototype memory neural network provided by an embodiment of the present invention;

[0060] Figure 2 is a schematic diagram of a process of outputting prototype features provided by an embodiment of the present invention;

[0061] Figure 3 is a schematic diagram of a flow chart of a model test provided by an embodiment of the present invention;

[0062] Figure 4It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention, and they are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0064] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the flow chart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned drawings may be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiment of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0065] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0067] Before describing the embodiments of the present invention in detail, some nouns and terms involved in the embodiments of the present invention are first described. The nouns and terms involved in the embodiments of the present invention are subject to the following explanations.

[0068] Video anomaly event detection is a technology used to automatically identify and detect deviations from normal behavior patterns in surveillance videos. This technology is very important in intelligent surveillance systems to improve safety and respond to abnormal events in a timely manner, such as traffic accidents and criminal activities.

[0069] Deep learning originates from the study of artificial neural networks. It combines low-level features to form more abstract high-level representations of attribute categories or features to discover distributed feature representations of data.

[0070] Deep learning-based video anomaly detection refers to the use of deep neural networks to automatically analyze video content and identify abnormal events that do not conform to normal patterns or expected behaviors. This technology is often used in video surveillance systems to improve security and efficiency. Its working principle is to use deep autoencoders to learn normal patterns and use reconstruction errors to quantify the degree of abnormality.

[0071] Prototype network is a pattern recognition method based on prototype learning. Prototype learning is a machine learning paradigm that captures the distribution characteristics of data by learning a set of representative prototypes. Each prototype represents the characteristics or attributes of a class in the data set and is used for subsequent classification or other tasks.

[0072] Current video anomaly detection can be divided into weakly supervised and unsupervised methods. Among them, weakly supervised video anomaly detection requires normal and abnormal data to train the model, but weakly supervised video anomaly detection can only identify abnormal events included in the training set, but cannot detect unseen anomalies. Although time-intensive video time annotation is not required, a large number of abnormal samples still need to be collected. Unsupervised methods usually formulate video anomaly detection as an anomaly detection problem. The anomaly detector is only trained using normal data. In the test phase, the model will identify behaviors that exceed the learned normal mode as abnormal events.

[0073] Early research on video anomaly detection mainly relied on manually designed features to represent video content, but this method has limited migration capabilities between different scenes. With the development of deep learning technology, methods based on deep neural networks are mainly divided into two frameworks: frame reconstruction and prediction. Although these methods have achieved significant improvements in performance, they still have some problems:

[0074] 1) Previous deep neural network-based methods may perform well in reconstructing or predicting abnormal events, but since the reconstruction target is the original input, the reconstruction-based model may reduce the loss by simply memorizing the pixel-level details of the input, which may lead to abnormal frames being well reconstructed. Although prediction-based methods can avoid this problem to a certain extent, they cannot guarantee a larger prediction error for abnormal behaviors due to the generalization ability of deep neural networks.

[0075] 2) Existing prediction-based methods lack the ability to fully exploit the long-term temporal context of video events. Specifically, these methods can usually only predict short-distance frames (such as the next frame) with small differences from the input sequence, which limits their ability to exploit temporal context. That is, ordinary frame prediction cannot explicitly exploit temporal context due to the small information gap.

[0076] 3) The diversity of normal patterns may have been overlooked in previous studies. For example, a street with no people and a street with a large number of pedestrians are both normal situations, but these two situations have completely different dynamic patterns. This means that when designing anomaly detection models, the diversity and complexity of normal behaviors need to be taken into account.

[0077] In view of this, if Figure 1 As shown, an embodiment of the present invention provides a method for detecting abnormal video events driven by a prototype memory neural network, which may include but is not limited to steps S100 to S900:

[0078] Step S100, obtaining a short video sequence, a long video sequence and a target frame;

[0079] Step S200, extracting long sequence dynamic features according to the long video sequence;

[0080] Step S300, constructing a prototype module according to the long sequence dynamic features;

[0081] Step S400, extracting short sequence dynamic features according to the short video sequence;

[0082] Step S500, matching the short sequence dynamic feature with the prototype feature of the prototype module;

[0083] Step S600, extracting memory features and hidden features according to the short video sequence;

[0084] Step S700, performing an aggregation operation on the matched prototype features, memory features, and hidden features to generate a prediction frame;

[0085] Step S800, performing model training according to the target frame, the predicted frame, the prototype module and the short sequence dynamic feature to obtain a target model;

[0086] Step S900: input the short video sequence into the target model to obtain a video anomaly score.

[0087] In some embodiments, step S200 may include but is not limited to steps S210 to S220:

[0088] Step S210, obtaining a first residual sequence of the long video sequence;

[0089] Step S220: input the first residual sequence into a first dynamic feature encoder to obtain a long sequence dynamic feature.

[0090] In steps S210 to S220 of some embodiments, Figure 2 As shown, get a long video sequence The corresponding first residual sequence The first residual sequence is sent to the first dynamic feature encoder (i.e., the long sequence dynamic feature encoder ε LDC ) to perform long sequence dynamic feature F LDC The extraction of Among them, the long sequence dynamic features are divided into local dynamic context Where h is the height of the image, w is the width of the image, and k is the index of the local dynamic context feature. It is the dynamic feature of long sequence local series.

[0091] In some embodiments, step S300 may include but is not limited to steps S310 to S320:

[0092] Step S310, obtaining a second residual sequence of the short video sequence;

[0093] Step S320: input the second residual sequence into a second dynamic feature encoder to obtain a short sequence dynamic feature.

[0094] In steps S310 to S320 of some embodiments, the short video sequence As the input of the model, the goal is to train the model to learn how to use short video sequences to match the long-term dynamic context stored in the prototype module. Figure 2 As shown, obtain a short video sequence The corresponding second residual sequence The second residual sequence is input into the second dynamic feature encoder (i.e., the short sequence dynamic feature encoder ε SDN ) to perform short sequence dynamic feature F SD The extraction of

[0095] In some embodiments, step S400 may include but is not limited to steps S410 to S430:

[0096] Step S410, obtaining a first local dynamic feature of the long sequence dynamic feature;

[0097] Step S420, constructing a first matching matrix;

[0098] Step S430: Match the first local dynamic feature with the prototype feature through the first vector of the first matching matrix to obtain a prototype module.

[0099] In steps S410 to S430 of some embodiments, a prototype module Γ may be constructed according to the long sequence dynamic features, wherein the prototype module includes B prototype items Γ i , then the expression of the prototype module is Optionally, the first local dynamic feature of the long sequence dynamic feature (i.e., the long sequence local dynamic feature ) as the query item, the prototype module is located by creating a first matching matrix M1, where each vector m k Used to match query terms and all prototype features, the expression of the first matching matrix is Each element value m k,i The matching probability is m k,i , the expression of the matching probability is Where d(·) is the cosine similarity. Then the prototype module can output local prototype features The expression of the local prototype feature is: Aggregating K local prototype features into a set, we have a prototype feature F with long-term dynamic context pro , the expression of the prototype feature is

[0100] In some embodiments, step S500 may include but is not limited to steps S510 to S530:

[0101] Step S510, obtaining a second local dynamic feature of the short sequence dynamic feature;

[0102] Step S520, constructing a second matching matrix;

[0103] Step S530: Match the second local dynamic feature with the prototype feature through the second vector in the second matching matrix to obtain the matched prototype feature.

[0104] In some embodiments, in step S510 to step S530, the second local dynamic feature of the short sequence dynamic feature (i.e., the short sequence local dynamic feature) is converted into ) as the query item to match the normal long-term dynamic prototype learned from the normal long video sequence in the prototype module. The second matching matrix M2 is calculated according to the prototype matching strategy, and the prototype module is located by creating a matching matrix. The expression of the second matching matrix is: Among them, each vector Used to match query terms and all prototype features, each element value The matching probability is The expression of the matching probability is: Where d(·) is the cosine similarity. Then the prototype module can output local prototype features The expression of the local prototype feature is: Aggregate K local prototype features into a set, that is, Then the short input F pro It has a dynamic context that includes a long period of time.

[0105] In steps S400 to S500 of some embodiments, first, the long-term motion prototype learned from the normal long video sequence is stored in the prototype module, and then the short sequence dynamic features extracted from the short video sequence are used to call the normal long-term motion prototype stored in the prototype module. In subsequent steps, the called normal long-term motion prototype will be used to help supplement the missing motion information between the short video input sequence and the long-distance target frame, thereby enhancing the model's utilization of temporal context and reducing the model's generalization ability to abnormalities.

[0106] In some embodiments, step S600 may include but is not limited to steps S610 to S620:

[0107] Step S610, inputting each frame of the short video sequence into a spatial encoder to extract features and obtain appearance features;

[0108] Step S620: input the appearance features into a spatiotemporal aggregation module to perform a temporal relationship capture operation, and output memory features and hidden features.

[0109] In steps S610 to S620 of some embodiments, each frame of the short video sequence is independently input into the spatial encoder ε SP Extract appearance features from Right now The appearance features are then fed into the spatiotemporal aggregation module to capture temporal relationships in time step order, and each unit then outputs a unit memory feature C k and a hidden state feature H k . Wherein, the spatiotemporal aggregation module is formed by stacking convolutional LSTM (ConvLSTM).

[0110] In some embodiments, step S700 may include but is not limited to steps S710 to S740:

[0111] Step S710, inputting the matched prototype features and the memory features into a multi-layer perceptron to generate channel attention;

[0112] Step S720, obtaining a channel-refined prototype feature according to the matched prototype feature and the channel attention;

[0113] Step S730, connecting the channel-refined prototype features and the hidden features to obtain aggregate features;

[0114] Step S740: input the aggregated features into a frame decoder to generate a predicted frame.

[0115] In some embodiments, in steps S710 to S740, the matching prototype features F are connected. pro and memory feature C k And input to the multi-layer perceptron to generate channel attention Right now According to the matched prototype features and the generated channel attention, the channel-refined prototype features can be obtained Right now Connect channel-refined prototype features and hidden features H from ConvLSTM k To embed long-term dynamic context into aggregate features In Due to the aggregation characteristics It can provide a priori of long-term dynamic context for the input short video sequence, which is input to the frame decoder Generate prediction frame Right now

[0116] In some embodiments, step S800 may include but is not limited to steps S810 to S870:

[0117] Step S810, constructing a prediction loss function, inputting the target frame and the prediction frame into the prediction loss function, and obtaining a prediction loss value;

[0118] Step S820, constructing a prototype dispersion loss function, inputting all prototype items in the prototype module and the mean vector of the prototype items into the prototype dispersion loss function to obtain a prototype dispersion loss value;

[0119] Step S830, constructing a feature compactness loss function, obtaining a second local dynamic feature of the short sequence dynamic feature, and obtaining a related prototype item of the second local dynamic feature, inputting the second local dynamic feature and the related prototype item into the feature compactness loss function, and obtaining a feature compactness loss value;

[0120] Step S840, constructing a first loss function according to the prediction loss function, the prototype dispersion loss function and the feature compactness loss function;

[0121] Step S850, optimizing the first dynamic feature encoder and the prototype module according to the first loss function;

[0122] Step S860, constructing a second loss function according to the prediction loss function and the feature compactness loss function;

[0123] Step S870: Optimize the second dynamic feature encoder according to the second loss function to obtain a target model.

[0124] In step S810 of some embodiments, the target frame (ie, the actual frame) is X t+1 And the generated prediction frame Input to the prediction loss function L pre In the above example, we can get the predicted loss value. The expression of the predicted loss function is:

[0125]

[0126] In step S820 of some embodiments, all prototype items Γ in the prototype module are i and the mean vector of all prototype items in the prototype module Input to the prototype dispersion loss function L dis In the above equation, we can get the prototype dispersion loss value. Then the expression of the prototype dispersion loss function is:

[0127]

[0128] Here, Tr(·) represents the trace of the matrix and (·) represents the transpose operation.

[0129] In step S830 of some embodiments, the second local dynamic feature (i.e., the short sequence local dynamic feature) is ) and the most relevant prototype item Input to the feature compactness loss function L com In the above equation, we can get the feature compactness loss value. The expression of the feature compactness loss function is:

[0130]

[0131] In steps S820 to S830 of some embodiments, a feature discrimination module is used to retain the diversity of normal patterns. Feature discrimination includes two parts: prototype dispersion and feature compactness, which are embedded in the process of model optimization to retain the diversity of prototypes and reduce the intra-class differences of normal patterns. Among them, prototype dispersion is that the prototype module learns the representative prototypes of normal patterns by forcing the prototypes to move away from each other during the model optimization process. This process helps the model better capture the intrinsic structure of the data, thereby improving the accuracy of recognition and classification. Feature compactness forces query features to be close to their nearest prototypes in the prototype module to reduce intra-class changes, so that query features with the same dynamic pattern can be accurately matched to the same prototype.

[0132] In step S840 of some embodiments, by predicting the loss function L pre , prototype dispersion loss function L dis And the feature compactness loss function L com , we can get the first loss function L 1st The expression is:

[0133] L 1st =L pre +λ(L dis +L com )

[0134] In the formula, λ represents the hyperparameter used to adjust the weight of the loss function. In the experiment, λ = 0.1.

[0135] In step S850 of some embodiments, the long sequence dynamic feature encoder ε is optimized by a first loss function. LDC , the weights of the prototype module Γ and the prototype memory neural network (referred to as network P), the prototype model Γ can be updated according to the following formula:

[0136]

[0137] In the formula, And in Z i Renormalize the query features, then:

[0138] Where N(·) represents the l2 norm; ω k,i Represents the attention weight of the prototype module; Represents the normalized attention weight; Z i Represents the prototype item Γ of the corresponding prototype module i The query feature set.

[0139] In step S860 of some embodiments, by predicting the loss function L preAnd the feature compactness loss function L com , we can get the second loss function L 2ed The expression is:

[0140] L 2ed =L pre +λL com

[0141] In the formula, λ represents the hyperparameter used to adjust the weight of the loss function. In the experiment, λ = 0.1.

[0142] In step S870 of some embodiments, the short sequence dynamic feature encoder ε is optimized by a second loss function. SDM and the weights of the network P, where the network weights of the prototype module are fixed in this step.

[0143] In step S900 of some embodiments, Figure 3 As shown, the target model is tested. During the test phase, only short video sequences are input into the model. The short-term dynamic features are extracted by the long-term dynamic prototype module to match the long-term dynamic prototype in the prototype module, and then combined with the spatiotemporal feature aggregation module to generate the prediction frame. Use target frame X t+1 and predicted frame The prediction error between the two and the dynamic short sequence feature F SD Its prototype Γ * The anomaly score is calculated by the matching error between mat The calculation formula is as follows:

[0144]

[0145] The calculation formula for the final anomaly score S is:

[0146] S = β(1-M(S pre ))+(1-β)M(S mat )

[0147] Where β is the balancing weight; M(·) is the normalization function applied on the entire video frame; S pre is the peak signal-to-noise ratio.

[0148] For example, Figure 3As shown in the figure, a short video sequence is input into the target model, and each frame of the short video sequence is independently input into the spatiotemporal feature encoder to extract the spatiotemporal features. Then the spatiotemporal features are input into the spatiotemporal aggregation module to capture the temporal relationship in time step order, and then each unit outputs a unit memory feature and a hidden state feature. In addition, the residual sequence of the short sequence video is extracted by the dynamic feature encoder to obtain the short sequence feature, and then the long-term dynamic prototype module extracts the short-term dynamic features to match the long-term dynamic prototype in the prototype module. The unit memory features and the hidden state features are combined to obtain the aggregated features, and then input into the encoder to generate the predicted frame. The anomaly score of the video is calculated by the prediction error between the target frame and the predicted frame, and the matching error between the dynamic short sequence features and their dynamic prototype features.

[0149] The embodiment of the present invention further provides a prototype memory neural network driven video abnormal event detection device, which can implement the above-mentioned prototype memory neural network driven video abnormal event detection method, and the device includes:

[0150] The first module is used to obtain short video sequences, long video sequences and target frames;

[0151] The second module is used to extract long sequence dynamic features according to the long video sequence;

[0152] The third module is used to extract short sequence dynamic features according to the short video sequence;

[0153] The fourth module is used to construct a prototype module according to the long sequence dynamic characteristics;

[0154] A fifth module is used to match the short sequence dynamic features with the prototype features of the prototype module;

[0155] A sixth module is used to extract memory features and hidden features according to the short video sequence;

[0156] The seventh module is used to perform an aggregation operation on the matched prototype features, memory features and hidden features to generate a prediction frame;

[0157] An eighth module is used to perform model training according to the target frame, the predicted frame, the prototype module and the short sequence dynamic feature to obtain a target model;

[0158] The ninth module is used to input the short video sequence into the target model to obtain a video anomaly score.

[0159] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0160] The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned method for detecting abnormal video events driven by a prototype memory neural network is implemented. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0161] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0162] refer to Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0163] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0164] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes a prototype memory neural network driven video abnormal event detection method of the embodiment of the present invention;

[0165] Input / output interface 1003, used to implement information input and output;

[0166] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0167] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0168] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0169] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned prototype memory neural network driven video abnormal event detection method.

[0170] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0171] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the aforementioned video abnormal event detection method driven by a prototype memory neural network.

[0172] In summary, the method and device for detecting abnormal video events driven by a prototype memory neural network according to an embodiment of the present invention have the following advantages:

[0173] 1. In the anomaly detection technology of the embodiment of the present invention, the prototype guides the dynamic matching network, thereby enhancing the discrimination ability and robustness of the anomaly detector (i.e., network P), having a high accuracy in video anomaly detection, minimizing the difference between the predicted frame and the actual frame (i.e., the target frame), and reducing the time consumption of manual anomaly retrieval, allowing the intelligent video surveillance system to automatically generate detection anomaly events, thereby speeding up the processing efficiency of users. The embodiment of the present invention can be applied to video anomaly detection systems, intelligent video surveillance systems, and the like.

[0174] 2. The embodiment of the present invention adopts a novel prototype memory neural network driven and dynamic perception video prediction framework (referred to as network P) to learn representative motion prototypes from normal videos through a prototype module. In the test phase, the motion features of the input video sequence are matched to the normal prototype in the prototype module for reconstruction, and then the reconstructed features are used to predict the target frame. Since the abnormal target frame is also predicted using the normal prototype, the model of the embodiment of the present invention can reduce the generalization ability of the abnormality compared with direct prediction.

[0175] 3. An embodiment of the present invention stores the long-term motion prototypes learned from normal long video sequences into a prototype module (i.e., constructs a prototype module) to learn the long-term dynamic prototypes of normal modes, which helps to predict normal frames over long distances and suppress abnormal frames; then, according to the long-term and short-term dynamic prototype matching learning mechanism, the motion features extracted from the short sequence are used to call the stored normal long-term motion prototypes (i.e., matching prototype modules); finally, the called normal long-term motion prototypes are used to help supplement the missing motion information between the short input sequence and the long-distance target frame, thereby enhancing the model's utilization of temporal context.

[0176] 4. The embodiment of the present invention adopts a feature recognition module to retain the diversity of normal patterns, wherein feature recognition includes two parts, namely prototype dispersion and feature compactness. Embedding it in the process of model optimization can retain the diversity of prototypes and reduce the intra-class differences of normal patterns.

[0177] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0178] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0179] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0180] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0181] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0182] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0183] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0184] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0185] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for detecting abnormal video events driven by a prototype memory neural network, characterized in that: The following steps are involved: Obtain short video sequences, long video sequences and target frames; Extracting long sequence dynamic features according to the long video sequence; Extracting short sequence dynamic features according to the short video sequence; Constructing a prototype module according to the long sequence dynamic characteristics; Matching the short sequence dynamic features with the prototype features of the prototype module; Extracting memory features and hidden features according to the short video sequence; Aggregating the matched prototype features, memory features, and hidden features to generate a prediction frame; Performing model training according to the target frame, the predicted frame, the prototype module and the short sequence dynamic features to obtain a target model; The short video sequence is input into the target model to obtain a video anomaly score.

2. The method for detecting abnormal video events driven by a prototype memory neural network according to claim 1, characterized in that: The step of extracting long sequence dynamic features according to the long video sequence comprises the following steps: Acquire a first residual sequence of the long video sequence; The first residual sequence is input into a first dynamic feature encoder to obtain a long sequence dynamic feature.

3. The method for detecting abnormal video events driven by a prototype memory neural network according to claim 1, characterized in that: The step of extracting short sequence dynamic features according to the short video sequence comprises the following steps: Obtaining a second residual sequence of the short video sequence; The second residual sequence is input into a second dynamic feature encoder to obtain a short sequence dynamic feature.

4. The method for detecting abnormal video events driven by a prototype memory neural network according to claim 1, characterized in that: The construction of the prototype module according to the long sequence dynamic features comprises the following steps: Acquire a first local dynamic feature of the long sequence dynamic feature; constructing a first matching matrix; The first local dynamic feature is matched with the prototype feature through the first vector of the first matching matrix to obtain a prototype module.

5. The method for detecting abnormal video events driven by a prototype memory neural network according to claim 1, characterized in that: The matching of the short sequence dynamic features with the prototype features of the prototype module comprises the following steps: Acquire a second local dynamic feature of the short sequence dynamic feature; constructing a second matching matrix; The second local dynamic feature is matched with the prototype feature through the second vector in the second matching matrix to obtain the matched prototype feature.

6. The method for detecting abnormal video events driven by a prototype memory neural network according to claim 1, characterized in that: The step of extracting memory features and hidden features according to the short video sequence comprises the following steps: Input each frame of the short video sequence into a spatial encoder to extract features and obtain appearance features; The appearance features are input into the spatiotemporal aggregation module to perform a temporal relationship capture operation, and a memory feature and a hidden feature are output.

7. The method for detecting abnormal video events driven by a prototype memory neural network according to claim 1, characterized in that: The step of performing an aggregation operation on the matched prototype features, memory features, and hidden features to generate a prediction frame includes the following steps: Inputting the matched prototype features and the memory features into a multi-layer perceptron to generate channel attention; Obtaining channel-refined prototype features according to the matched prototype features and the channel attention; Connecting the channel-refined prototype features and the hidden features to obtain aggregate features; The aggregated features are input into a frame decoder to generate a predicted frame.

8. The method for detecting abnormal video events driven by a prototype memory neural network according to claim 1, characterized in that: The method of performing model training according to the target frame, the predicted frame, the prototype module and the short sequence dynamic features to obtain a target model comprises the following steps: Constructing a prediction loss function, inputting the target frame and the prediction frame into the prediction loss function to obtain a prediction loss value; Constructing a prototype dispersion loss function, inputting all prototype items in the prototype module and the mean vector of the prototype items into the prototype dispersion loss function to obtain a prototype dispersion loss value; Constructing a feature compactness loss function, obtaining a second local dynamic feature of the short sequence dynamic feature, and obtaining a related prototype item of the second local dynamic feature, inputting the second local dynamic feature and the related prototype item into the feature compactness loss function, and obtaining a feature compactness loss value; Constructing a first loss function according to the prediction loss function, the prototype dispersion loss function and the feature compactness loss function; Optimizing the first dynamic feature encoder and the prototype module according to the first loss function; Constructing a second loss function according to the prediction loss function and the feature compactness loss function; According to the second loss function, the second dynamic feature encoder is optimized to obtain a target model.

9. A video abnormal event detection device driven by a prototype memory neural network, characterized in that: include: The first module is used to obtain short video sequences, long video sequences and target frames; The second module is used to extract long sequence dynamic features according to the long video sequence; The third module is used to extract short sequence dynamic features according to the short video sequence; The fourth module is used to construct a prototype module according to the long sequence dynamic characteristics; A fifth module is used to match the short sequence dynamic features with the prototype features of the prototype module; A sixth module is used to extract memory features and hidden features according to the short video sequence; The seventh module is used to perform an aggregation operation on the matched prototype features, memory features and hidden features to generate a prediction frame; An eighth module is used to perform model training according to the target frame, the predicted frame, the prototype module and the short sequence dynamic feature to obtain a target model; The ninth module is used to input the short video sequence into the target model to obtain a video anomaly score.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 8.