A video anomaly detection method based on a novel memory network

Abstract

The present invention discloses a video anomaly detection method based on a novel memory network. Based on video sample frames arranged in chronological order, a video anomaly frame detection model is constructed, and then the video anomaly frame detection model is applied to complete normal or abnormal detection of the video sample frames. The video anomaly frame detection model includes a preliminary feature extraction network for extracting feature maps of the video sample frames, a deep feature extraction classification network for performing deep feature extraction on a feature map group formed by the feature maps, and based on the components in the label corresponding to the output of the deep feature extraction classification network and the classification loss model, an anomaly score for determining whether each video sample frame in a video sample frame sequence is normal or abnormal is constructed. The method can better extract feature information in the video sample frames and improve the adaptability of the video anomaly frame detection model.

Description

Technical Field

[0001] The present invention relates to a video anomaly detection method, and in particular to a video anomaly detection method based on a novel memory network. Background Art

[0002] Video anomaly detection has always been a hot topic in the field of computer vision and is closely related to video analysis research. The task of abnormal video detection aims to analyze and understand videos and capture abnormal events in videos that are different from expected behaviors. It has great application value and development prospects in the fields of intelligent security, intelligent transportation, and fault correction systems.

[0003] Unlike traditional binary classification tasks, anomaly detection tasks have their own particularities. The probability of anomalies is small, and it is almost impossible to collect balanced normal and abnormal data. Secondly, there are many types of anomalies and anomalies are unbounded. It is difficult to label various anomalies in huge video data. Therefore, anomaly detection is a challenging task.

[0004] With the great success of deep learning algorithms in the field of computer vision in recent years, algorithms based on deep neural networks have gradually been applied to video anomaly detection tasks. Two types of methods have been derived, namely anomaly detection methods based on current frame reconstruction and anomaly detection methods based on future frame prediction. The method based on reconstructing the current frame distinguishes abnormal frames from normal frames based on the idea that the reconstruction error of abnormal frames is large. The method based on future frame prediction decides whether future frames are normal or not based on the idea that anomalies are difficult to predict. Although these two different anomaly detection methods have achieved some results, the idea of ​​using proxy tasks to achieve anomaly detection is essentially flawed. That is, whether it is a reconstruction method or a prediction method, its essence is to output an image that is as similar as possible to the real frame. When the network is trained very well, these two methods are not necessarily able to distinguish between normal and abnormal frames due to their ideological defects. In some special scenarios, the effect is low. Summary of the invention

[0005] The purpose of the present invention is to provide a video anomaly detection method based on a novel memory network, to construct a video anomaly frame detection model based on a preliminary feature extraction network and a deep feature extraction classification network, and then to apply the video anomaly frame detection model to complete normal or abnormal detection of video sample frames.

[0006] To achieve the above functions, the present invention designs a video anomaly detection method based on a novel memory network. According to the following steps S1 to S6, a video anomaly frame detection model is obtained, and then the video anomaly frame detection model is applied to complete the normal or abnormal detection of the video sample frame;

[0007] S1. Obtain video sample frames arranged in chronological order, take each video sample frame as a starting point, select k video sample frames in chronological order to construct a video sample frame group, and obtain a predefined label indicating a normal state or an abnormal state of the video sample frame group;

[0008] S2. Based on the convolutional neural network, a preliminary feature extraction network is constructed with the video sample frame as input and the feature map corresponding to the video sample frame as output;

[0009] S3. Based on the mode of real-time output of feature graphs by the preliminary feature extraction network, a feature graph group consisting of k frames of feature graphs in the historical time direction is used as input, and the label of the video sample frame group corresponding to the feature graph group is used as output. A deep feature extraction classification network is constructed based on a neural network with three fully connected layers, where k is greater than 1.

[0010] S4. Based on the preliminary feature extraction network and the deep feature extraction and classification network, a model for video abnormal frame detection to be trained is constructed. At the same time, based on the participation of each video sample frame in the training, the classification loss model is constructed by applying the preliminary feature extraction network to the deep feature extraction and classification network by introducing the cross entropy loss;

[0011] S5. Based on the video sample frame groups constructed by the video sample frames and the labels corresponding to the video sample frame groups, the video sample frames are used as input, the labels corresponding to the video sample frame groups are used as output, and the classification loss model is combined to train the model to be trained for the video abnormal frame detection to obtain the video abnormal frame detection model;

[0012] S6. For each video sample frame in each video sample frame group, construct an abnormality score for determining whether each video sample frame in the video sample frame group is normal or abnormal, and determine a video sample frame with an abnormality score greater than a preset value as an abnormal video frame, otherwise it is a normal video frame.

[0013] As a preferred technical solution of the present invention: the preliminary feature extraction network is a U-Net encoder.

[0014] As a preferred technical solution of the present invention: in step S3, when the number of feature map frames output by the preliminary feature extraction network is less than the preset frame number k, the feature map output by the preliminary feature extraction network in real time is copied by k frames to construct a feature map group, and the k-frame feature map group is used as the input of the deep feature extraction classification network.

[0015] As a preferred technical solution of the present invention: in the step S3, when the number of feature map frames output by the preliminary feature extraction network is less than the preset frame number k, the n frames of feature maps output by the preliminary feature extraction network and the feature maps output by the preliminary feature extraction network in real time are copied to kn frames to construct a feature map group, and the k-frame feature map group is used as the input of the deep feature extraction classification network.

[0016] As a preferred technical solution of the present invention: in the step S3, when the number of feature map frames output by the preliminary feature extraction network is less than the preset frame number k, wait until the number of feature map frames output by the preliminary feature extraction network is k, construct a feature map group with the k-frame feature maps, and use it as the input of the deep feature extraction classification network.

[0017] As a preferred technical solution of the present invention: step S4 comprises the following steps:

[0018] S41: For the output of the deep feature extraction classification network, the label is constructed as follows:

[0019] (y 0 ,y 1 ,y 2 ,y 3 ,...y 2k )

[0020] Among them, y i is the i-th component of the label, and the number of components of the label is 2k;

[0021] The labels corresponding to the preset normal video sample frame groups are as follows:

[0022] (1, 0, 1, 0, ... 0)

[0023] Among them, the number of components of this tag is 2k;

[0024] The labels corresponding to the preset abnormal video sample frame groups are as follows:

[0025] (0, 1, 0, 1, ... 1)

[0026] Among them, the number of components of this tag is 2k;

[0027] The labels corresponding to the first set of video sample frame groups are preset as follows:

[0028] (0, 1, 0, 1, ... 1)

[0029] Among them, the number of components of this tag is 2k;

[0030] S42: Based on the method of inputting the feature map group formed by the feature maps corresponding to the video sample frames output by the preliminary feature extraction network into the deep feature extraction classification network, the label corresponding to the output of the deep feature extraction classification network is as follows:

[0031] (d×epoch, 1-d×epoch, d×epoch, 1-d×epoch, ...1-d×epoch)

[0032] Among them, epoch is the number of training times, d is a preset label parameter less than 1, and the number of components of this label is 2k;

[0033] S43: Based on the output of the deep feature extraction classification network and its corresponding label, the classification loss model Loss is constructed according to the cross entropy formula as follows:

[0034] Loss = ∑-y i ln(x i )

[0035] In the formula, x i is the output of the i-th neuron in the output layer of the deep feature extraction classification network, y i is the i-th component of the label.

[0036] As a preferred technical solution of the present invention: the label is composed of 2k components, and the component group composed of two adjacent components in the label represents the normal or abnormal state of each video sample frame in the video sample frame group. Based on the classification losses corresponding to the 2kth and 2k-1th components of the label, an abnormal score of the video sample frame for judging the real-time input video abnormal frame detection model is constructed.

[0037] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0038] The present invention designs a video anomaly detection method based on a new memory network. The method is inspired by the human brain's recognition, understanding and identification of anomalies, and proposes a new anomaly detection method based on the idea of ​​"what has been seen is normal, what has not been seen is abnormal". This method abandons the basic ideas of the two previous methods, overcomes the defects of the original ideas, and realizes a new method for video anomaly detection by learning and recognizing video content. The framework of this method is divided into two parts. The first part sends the real-time video sample frame to the first preliminary feature extraction network to extract the primary feature information of the real-time input video sample frame. The second part is used to send the primary feature information of the past few frames to the second deep feature extraction classification network, extract the deep information of this sequence and perform another integrated judgment, and finally output the abnormal score to judge whether the video sample frame in this video sequence is normal or not. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 4 is a schematic diagram of a video abnormal frame detection model provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0041] Reference Figure 1 , a video anomaly detection method based on a novel memory network provided by an embodiment of the present invention is characterized in that, according to the following steps S1 to S6, a video anomaly frame detection model is obtained, and then the video anomaly frame detection model is applied to complete normal or abnormal detection of video sample frames;

[0042] S1. Obtain video sample frames arranged in chronological order, take each video sample frame as a starting point, select k video sample frames in chronological order to construct a video sample frame group, and obtain a predefined label indicating a normal state or an abnormal state of the video sample frame group;

[0043] S2. Based on the convolutional neural network, a preliminary feature extraction network is constructed with the video sample frame as input and the feature map corresponding to the video sample frame as output;

[0044] The input of the video sample frame adopts a single video sample frame sequential input mode. In one embodiment, when the t-th video sample frame is input, I t Represents this video sample frame, and the video sample frame I t Input the preliminary feature extraction network to extract a feature map representing the preliminary feature information.

[0045] Before the video sample frame is input into the preliminary feature extraction network, it is normalized to a H×W×C video sample frame, where H is the height of the video sample frame, W is the width of the video sample frame, and C is the number of channels of the video sample frame. The preliminary feature extraction network consists of three basic layers and three pooling layers. Each basic layer contains 2 convolutional layers, 2 Relu activation layers, and 2 batch normalization layers.

[0046] In one embodiment, a 256×256×3 video sample frame passes through the preliminary feature extraction network to generate a 32×32×512 feature map to represent the primary feature information of the video sample frame. Suppose the output of the video sample frame passing through the preliminary feature extraction network at time t is f(I t ).

[0047] S3. Based on the mode of real-time output of feature graphs by the preliminary feature extraction network, a feature graph group consisting of k frames of feature graphs in the historical time direction is used as input, and the label of the video sample frame group corresponding to the feature graph group is used as output. A deep feature extraction classification network is constructed based on a neural network with three fully connected layers, where k is greater than 1.

[0048] The deep feature extraction classification network is different from the preliminary feature extraction network. The input of this network is not a single feature map, but a feature map group of the past k frames, i.e., f(I t )、f(I t-1 )、f(I t-2 )......f(I t-k ).

[0049] In one embodiment, a FIFO queue is used to store the first k-1 frames of feature maps output by the preliminary feature extraction network in real time. The feature map output by the preliminary feature extraction network is inserted into the end of the FIFO queue, and the first feature map that enters the queue, i.e., the earliest feature information, will be discarded. The FIFO queue needs to be updated all the time to keep storing feature maps of several past frames.

[0050] In one embodiment, k=4, and the queue length of the FIFO is 3, which ensures that the first element will be dequeued when the FIFO queue is updated. The feature map output by the preliminary feature extraction network will first be superimposed on the channel before being input into the deep feature extraction and classification network, that is, the dimension of the feature map input into the deep feature extraction and classification network in this embodiment is 1×2048×32×32. The network consists of three fully connected layers, all of which are used to reduce the dimension. The number of output nodes of the first fully connected layer is 512, the number of nodes of the second fully connected layer is 32, and the number of output nodes of the third fully connected layer is 8. There is a Relu activation function between each layer of the neural network. Every two output neural nodes of the output node are used as a pair to evaluate the anomaly score of the corresponding frame of this video sequence. In this way, the following output is generated through the processing of the deep feature extraction and classification network: g(f(I t ), f(I t-1 ), f(I t-2 ), f(I t-3 )), where the g function represents the processing of the deep feature extraction classification network.

[0051] In the video anomaly detection method based on a novel memory network provided by an embodiment of the present invention, when the number of feature map frames output by the preliminary feature extraction network is less than a preset number of frames k, the feature map output by the preliminary feature extraction network in real time is copied by k frames to construct a feature map group, and the k-frame feature map group is used as the input of the deep feature extraction classification network.

[0052] In the video anomaly detection method based on a novel memory network provided by an embodiment of the present invention, when the number of feature map frames output by the preliminary feature extraction network is less than the preset frame number k, the n frames of feature maps output by the preliminary feature extraction network and the feature maps output by the preliminary feature extraction network in real time are copied kn frames to construct a feature map group, and the k-frame feature map group is used as the input of the deep feature extraction classification network.

[0053] In the video anomaly detection method based on a novel memory network provided by an embodiment of the present invention, when the number of feature map frames output by the preliminary feature extraction network is less than the preset frame number k, the number of feature map frames output by the preliminary feature extraction network is waited to be k, and a feature map group is constructed with the k-frame feature maps, which are used as the input of the deep feature extraction classification network.

[0054] S4. Based on the preliminary feature extraction network and the deep feature extraction and classification network, a model for video abnormal frame detection to be trained is constructed. At the same time, based on the participation of each video sample frame in the training, the classification loss model is constructed by applying the preliminary feature extraction network to the deep feature extraction and classification network by introducing the cross entropy loss;

[0055] The video anomaly detection method based on a novel memory network provided by the embodiment of the present invention abandons the training idea of ​​the existing method, that is, during training, the labels of the training data set are kept constant and correspond to the positive example labels, but a new training strategy is adopted, that is, the output of the deep feature extraction classification network is not allowed to correspond to the normal label from the beginning to the end, but the deep feature extraction classification network is allowed to learn to recognize the normal video sample frames it has seen through multiple training. Therefore, in different training times, the labels corresponding to the output of the deep feature extraction classification network are different, so that a deep feature extraction classification network that slowly recognizes the normal video sample frames can be obtained, which specifically includes the following steps:

[0056] S41: For the output of the deep feature extraction classification network, the label is constructed as follows:

[0057] (y 0 ,y 1 ,y 2 ,y 3 ,...y 2k )

[0058] Among them, y i is the i-th component of the label, and the number of components of the label is 2k;

[0059] The labels corresponding to the preset normal video sample frame groups are as follows:

[0060] (1, 0, 1, 0, ... 0)

[0061] Among them, the number of components of this tag is 2k;

[0062] The labels corresponding to the preset abnormal video sample frame groups are as follows:

[0063] (0, 1, 0, 1, ... 1)

[0064] Among them, the number of components of this tag is 2k;

[0065] The labels corresponding to the first set of video sample frame groups are preset as follows:

[0066] (0, 1, 0, 1, ... 1)

[0067] Among them, the number of components of this tag is 2k;

[0068] S42: Based on the method of inputting the feature map group formed by the feature maps corresponding to the video sample frames output by the preliminary feature extraction network into the deep feature extraction classification network, the label corresponding to the output of the deep feature extraction classification network is as follows:

[0069] (d×epoch, 1-d×epoch, d×epoch, 1-d×epoch, ...1-d×epoch)

[0070] Among them, epoch is the number of training times, d is a preset label parameter less than 1, and the number of components of this label is 2k;

[0071] In one embodiment, k=4, and the label corresponding to the output of the deep feature extraction classification network is: 0 ,y 1 ,y 2 ,y 3 ,...y 7 ), when the deep feature extraction and classification network sees the feature map group corresponding to this video sample frame group for the first time, the label corresponding to its output should be (0, 1, 0, 1, 0, 1, 0, 1). As the number of training times increases, the deep feature extraction and classification network becomes more and more certain that the video sample frame group has been seen before. At this time, the label corresponding to the output should slowly increase to (0.75, 0.25, 0.75, 0.25, 0.75, 0.250.75, 0.25). As the number of training times increases, the deep feature extraction and classification network has seen this normal video sample frame group more and more times. At this time, the label value of this normal video sample frame group should be (1, 0, 1, 0, 1, 0, 1, 0). Experiments show that training the video abnormal frame detection model to be trained with training times epoch=51 and d=0.02 can obtain better training results. In this embodiment, when the video abnormal frame detection model to be trained is trained for the kth time with normal video sample frames as input, the label corresponding to its output is:

[0072] (0.02×epoch, 1-0.02×epoch, 0.02×epoch, 1-0.02×epoch, 0.02×epoch, 1-0.02×epoch, 0.02×epoch, 1-0.02×epoch, 0.02×epoch, 1-0.02×epoch)

[0073] S43: Based on the output of the deep feature extraction classification network and its corresponding label, the classification loss model Loss is constructed according to the cross entropy formula as follows:

[0074] Loss = ∑-y i ln(x i )

[0075] In the formula, x i is the output of the i-th neuron in the output layer of the deep feature extraction classification network, y i is the i-th component of the label.

[0076] S5. Based on the video sample frame groups constructed by the video sample frames and the labels corresponding to the video sample frame groups, the video sample frames are used as input, the labels corresponding to the video sample frame groups are used as output, and the classification loss model is combined to train the model to be trained for the video abnormal frame detection to obtain the video abnormal frame detection model;

[0077] S6. For each video sample frame in each video sample frame group, construct an abnormality score for determining whether each video sample frame in the video sample frame group is normal or abnormal, and determine a video sample frame with an abnormality score greater than a preset value as an abnormal video frame, otherwise it is a normal video frame.

[0078] In the video anomaly detection method based on the novel memory network provided by an embodiment of the present invention, the preliminary feature extraction network may adopt a U-Net encoder or a HRNet convolutional neural network.

[0079] In the video anomaly detection method based on a novel memory network provided by an embodiment of the present invention, the label is composed of 2k components, and a component group composed of two adjacent components in the label represents the normal or abnormal state of each video sample frame in the video sample frame group. Based on the classification losses corresponding to the 2kth and 2k-1th components of the label, an abnormal score of the video sample frame for judging the real-time input video abnormal frame detection model is constructed.

[0080] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A video anomaly detection method based on a novel memory network, characterized in that: According to the following steps S1 to S6, a video abnormal frame detection model is obtained, and then the video abnormal frame detection model is applied to complete the normal or abnormal detection of the video sample frame; S1. Obtain video sample frames arranged in chronological order, take each video sample frame as a starting point, select k video sample frames in chronological order to construct a video sample frame group, and obtain a predefined label indicating a normal state or an abnormal state of the video sample frame group; S2. Based on the convolutional neural network, a preliminary feature extraction network is constructed with the video sample frame as input and the feature map corresponding to the video sample frame as output; S3. Based on the mode of real-time output of feature graphs by the preliminary feature extraction network, a feature graph group consisting of k frames of feature graphs in the historical time direction is used as input, and the label of the video sample frame group corresponding to the feature graph group is used as output. A deep feature extraction classification network is constructed based on a neural network with three fully connected layers, where k is greater than 1. S4. Based on the preliminary feature extraction network and the deep feature extraction and classification network, a model for video abnormal frame detection to be trained is constructed. At the same time, based on the participation of each video sample frame in the training, the classification loss model is constructed by applying the preliminary feature extraction network to the deep feature extraction and classification network by introducing the cross entropy loss; Step S4 includes the following steps: S41: For the output of the deep feature extraction classification network, the label is constructed as follows: <h2 style=";text-align:left;direction:ltr">(y0, y1, y2, y3,…y<h2 style=";text-align:left;direction:ltr"> 2k <h2 style=";text-align:left;direction:ltr"> ) Among them, y i is the i-th component of the label, and the number of components of the label is 2k; The labels corresponding to the preset normal video sample frame groups are as follows: (1，0，1，0，…0) Among them, the number of components of this tag is 2k; The labels corresponding to the preset abnormal video sample frame groups are as follows: (0，1，0，1，…1) Among them, the number of components of this tag is 2k; The labels corresponding to the first set of video sample frame groups are preset as follows: (0，1，0，1，…1) Among them, the number of components of this tag is 2k; S42: Based on the method of inputting the feature map group formed by the feature maps corresponding to the video sample frames output by the preliminary feature extraction network into the deep feature extraction classification network, the label corresponding to the output of the deep feature extraction classification network is as follows: (d×epoch, 1-d×epoch, d×epoch, 1-d×epoch, …1-d×epoch) where epoch is the number of training times, d is a preset label parameter less than 1, and the number of components of the label is 2k; S43: Based on the output of the deep feature extraction classification network and its corresponding label, the classification loss model Loss is constructed according to the cross entropy formula as follows: Loss=∑-y i ln(x i ) In the formula, x i is the output of the i-th neuron in the output layer of the deep feature extraction classification network, y i is the i-th component of the label; S5. Based on the video sample frame groups constructed by the video sample frames and the labels corresponding to the video sample frame groups, the video sample frames are used as input, the labels corresponding to the video sample frame groups are used as output, and the classification loss model is combined to train the model to be trained for the video abnormal frame detection to obtain the video abnormal frame detection model; S6. For each video sample frame in each video sample frame group, construct an abnormality score for determining whether each video sample frame in the video sample frame group is normal or abnormal, and determine a video sample frame with an abnormality score greater than a preset value as an abnormal video frame, otherwise it is a normal video frame.

2. The video anomaly detection method based on the novel memory network as claimed in claim 1, characterized in that: The preliminary feature extraction network is a U-Net encoder.

3. The video anomaly detection method based on the novel memory network as claimed in claim 1, characterized in that: In step S3, when the number of feature map frames output by the preliminary feature extraction network is less than the preset frame number k, the feature map output by the preliminary feature extraction network in real time is copied to k frames to construct a feature map group, and the k-frame feature map group is used as the input of the deep feature extraction classification network.

4. The video anomaly detection method based on the novel memory network as claimed in claim 1, characterized in that: In step S3, when the number of feature map frames output by the preliminary feature extraction network is less than the preset frame number k, the n frames of feature maps output by the preliminary feature extraction network and the feature maps output by the preliminary feature extraction network in real time are copied to kn frames to construct a feature map group, and the k-frame feature map group is used as the input of the deep feature extraction classification network.

5. The video anomaly detection method based on the novel memory network as claimed in claim 1, characterized in that: In step S3, when the number of feature map frames output by the preliminary feature extraction network is less than the preset frame number k, wait until the number of feature map frames output by the preliminary feature extraction network is k, construct a feature map group with the k-frame feature maps, and use it as the input of the deep feature extraction classification network.

6. The video anomaly detection method based on the novel memory network as claimed in claim 1, characterized in that: The label is composed of 2k components, and a component group composed of two adjacent components in the label represents the normal or abnormal state of each video sample frame in the video sample frame group. Based on the classification losses corresponding to the 2kth and 2k-1th components of the label, an abnormal score of the video sample frame for judging the real-time input video abnormal frame detection model is constructed.

Images