A video anomaly detection method based on a causal time relationship module

Abstract

The application discloses a video anomaly detection method based on a causal temporal relationship module, which comprises the following steps: first, collecting video data as a training set and a test, dividing the video data into non-overlapping segments, regarding the video as a bag and the segment as an example; inputting the video example into a feature extraction network I3D to extract original features thereof; using a causal temporal relationship module to extract enhanced features containing time dependence, inputting the enhanced features into a full connection layer to obtain an anomaly score of the example, and training the model through a designed loss function; in the test stage, inputting the original features of the video example into the optimal anomaly detection model, comparing the prediction result with the label of the test set to obtain the accuracy of anomaly detection. The method is based on a video level label, uses only the current and historical frame information to extract time correlation through the causal temporal relationship module, is beneficial to online detection of anomalies, and improves the accuracy of anomaly detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video anomaly detection, and particularly relates to a video anomaly detection method based on a cause-effect time relationship module. BACKGROUND

[0002] In recent years, with the enhancement of people's safety awareness and the popularization of security knowledge, the anomaly detection technology based on intelligent monitoring video has gradually attracted attention. At the same time, the task of manual anomaly detection is becoming more and more difficult, so the video anomaly detection technology based on computer vision emerges as the times require.

[0003] The goal of the video anomaly behavior detection task is to identify and locate the abnormal target object or abnormal behavior in the monitoring video through various computer vision methods, such as vehicles reversing on the road, sudden robbery events occurring on the square, and so on. Compared with the classic target detection tasks such as pedestrian re-identification and natural language processing, the video anomaly detection task has the following special features: (1) The concept of anomaly is abstract, and its definition is not specific enough (i.e. "violation of the norm"), and there is a lack of clear dividing line between abnormal samples and normal samples. (2) The definition of abnormal events has a certain relativity. Whether the same event is an abnormal event depends on the scene it is in, such as a soldier training a gun in the army is a normal event, but if someone carries a gun in the park, it is an abnormal event. (3) The rarity of abnormal events. The frequency of abnormal events in reality is much lower than that of normal cases, so data collection can be difficult, and the available data is imbalanced between normal and abnormal categories. (4) The real environment is complex and changeable, so there are many types of abnormal events that cannot be listed one by one. All of the above unique features make the video anomaly event detection a very challenging task.

[0004] In the prior art, the detection of video abnormal behavior has the problems of high frame-by-frame labeling cost, low recognition accuracy, and important time sequence relationship in the video being ignored. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a video anomaly detection method based on a cause-effect time relationship module. First, the I3D network is used to extract the RGB features and the optical flow features of the video respectively, and the two are spliced to obtain the original features of the video, and then input into a cause-effect time relationship module to capture the local time dependence relationship of the video segment, and then the abnormal score of the learning example is guided by multiple losses to realize anomaly detection under weak supervision, solve the problems of high labeling cost, low recognition accuracy, and important time sequence relationship in the video being ignored in the existing detection methods, and achieve the purpose of reducing the artificial labeling cost while improving the accuracy of abnormal event detection.

[0006] In order to achieve the above object, the application provides a video anomaly detection method based on a causal time relationship module, comprising the following steps:

[0007] S1: First, collect video data, divide it into a training set and a test set according to a 3:1 ratio, then divide each video into a plurality of mutually non-overlapping segments with the same number of frames, regard each video as a package, define the abnormal video and the normal video as positive packages and negative packages respectively, and regard the segments as individual examples;

[0008] S2: Then, input the RGB images and the optical flow images of the video examples of the training set and the test set into an inflation convolution network I3D to obtain the visual features and the optical flow features, and splice the two to obtain the original features of the video examples;

[0009] S3: Input the original features obtained in S2 into a causal time convolution module, use the features of the current example and the previous examples to enhance the interaction between the current frame information and the historical frame information, and then obtain the time dependence relationship of the video to obtain enhanced features containing time information;

[0010] S4: Input the enhanced features obtained in S3 into a fully connected layer network to obtain the anomaly scores of each example, and the greater the score value, the greater the possibility of being an abnormal segment;

[0011] S5: Select the k examples (in this method, the value of k is uniformly set to 4) with the maximum anomaly scores in each video package, use the classification loss, the smoothing loss and the sparse loss to train the video anomaly detection model in S3 and S4 to obtain a group of optimal model parameters;

[0012] S6: Compare the anomaly scores of each example obtained with the set threshold value, and if the anomaly score is greater than the threshold value, it is determined to be abnormal, otherwise it is determined to be normal; the labels of the test set are frame-level, and the prediction results are compared with the real labels to obtain the accuracy rate of video anomaly detection.

[0013] Further, in the step S2, the step of extracting video features specifically comprises:

[0014] S2-1: The inflation convolution network I3D comprises two parallel 3D convolution modules, the RGB images of all examples of each video are input into one branch of the I3D, and the visual features of each example are extracted;

[0015] S2-2: The optical flow images of the video examples are input into another 3D convolution branch of the I3D, and the motion features of the video are extracted;

[0016] S2-3: The RGB features and the motion features are spliced to obtain the original features of the video.

[0017] Further, in the step S3, the method for obtaining the enhanced features from the original features of the video through the causal temporal relationship module specifically comprises:

[0018] S3-1: The causal temporal relationship module aims to aggregate useful information from historical information and current features through a temporal attention mechanism. Unlike the ordinary self-attention mechanism, it only uses the information of the current frame and the previous frames to capture the temporal dependence in the local range, which is more suitable for online tasks. The formula is as follows:

[0019]

[0020] where X n,t represents the current feature, X n,t-Δt:t ={X n,t-Δt ,X n,t-Δt+1 ,...,X n,t} represents the features of the previous Δt examples, and Δt represents the number of examples in this interval. The semantic similarity between the current example X n,t and the previous Δt examples X n,t-Δt:t is calculated by a dot product operation, and the final feature similarity matrix is calculated by softmax normalization.

[0021] S3-2: R represents the relative position, i.e. {-Δt,...,-1,0}, and the final position prior matrix is calculated by softmax normalization, which indicates that the farther the distance from the current distance, the more the information decays. The position prior matrix is a constant constant that does not change with the input features;

[0022] S3-3: The semantic similarity matrix and the position prior matrix are concatenated, and then multiplied element-wise with the features of the example to obtain enhanced features containing local temporal dependencies of the video.

[0023] Further, in the step S4, the method for obtaining the anomaly score is as follows: first, input the enhanced features containing temporal information obtained in step S3 into a fully connected network, which contains three fully connected layers and a sigmoid layer. Each example will get an anomaly score between 0 and 1, and the larger the score, the more likely the model considers the segment to be an abnormal segment.

[0024] Further, in the step S5, the method for training the model specifically comprises:

[0025] S5-1: Obtain the anomaly score of each example in step S4, and select the top k examples from all video examples according to the anomaly score. Then, use the classification loss L c to train the model combined with the video-level label;

[0026] S5-2: A temporal smoothing loss Lsm such that adjacent segments in the abnormal video have similar abnormal scores;

[0027] S5-3: constrain the abnormal scores of abnormal videos and the number of abnormal segments with sparse loss L sp ;

[0028] S5-4: add the classification loss, the smoothing loss and the sparse loss to obtain the global loss function L of the model L c = L sm + L sp , guide the training of the video anomaly detection model, and obtain optimal model parameters.

[0029] Further, in the step S6, the test process includes: the test set video examples pass through the step S2 to obtain original features thereof, pass through the step S3 to obtain enhanced features thereof, and then are input into the optimal anomaly detection model obtained in the step S5 to obtain abnormal scores of each test video example, the frame-level scores of each example are consistent with the example-level. Then, the frame-level scores are compared with a threshold value, and if the frame-level scores are greater than the threshold value, it is determined that the video is abnormal, and otherwise, it is determined that the video is normal. Since the test set labels are frame-level, the predicted results can be compared with the real labels to obtain the accuracy of video anomaly detection.

[0030] By means of the above technical solutions, the application provides a video anomaly detection method and system based on a causal time relationship module, and at least has the following beneficial effects:

[0031] 1. The application is based on a weakly supervised setting, and regards videos as packets and segments as examples. In the training stage, only video-level labels are used for training and optimization of the model, thereby relieving the pressure of frame-by-frame labeling which needs to consume a large amount of manpower and material resources.

[0032] 2. The application adopts a causal convolution network, and only uses current and historical information to capture local time dependence of the video, which is beneficial to the promotion of the online real-time anomaly detection task and the capture of important time correlation in the video.

[0033] 3. The model is trained by selecting k segments instead of one segment from each video, which can more fully utilize the video data, enhances the robustness to normal segments in abnormal videos, and improves the effect of anomaly event detection. DETAILED DESCRIPTION

[0034] Figure 1 is a flowchart of the video anomaly detection method based on the causal time relationship module in the application;

[0035] Figure 2 is a preprocessing diagram of the video in the application;

[0036] Figure 3A structural diagram of a cause-effect time relationship module in the application;

[0037] Figure 4 A visualization result diagram of abnormal behavior detection in the application; DETAILED DESCRIPTION

[0038] For the convenience of understanding of those skilled in the art, the application is further described below in combination with the drawings.

[0039] As Figures 1-4 shown, the application is a video abnormality detection method based on a cause-effect time relationship module, specifically comprising the following steps:

[0040] S1: As Figure 1 shown, first, video data is collected, the ShanghaiTech dataset used in the method is a public dataset commonly used in the field of abnormality detection, which is shot by multiple scene and angle cameras, has complex lighting conditions and camera angles. The video is divided into a training set and a test set, the number of which is 238 and 199 respectively, the training set provides video-level labels, and the test set provides frame-level labels. The video is divided into segments with the same number of frames and no overlap, each video is regarded as a bag by using a multiple instance learning method, the segments are regarded as instances in the bag, and the video containing abnormal behavior is regarded as a positive bag B a , and the video not containing abnormal behavior is regarded as a negative bag B n .

[0041] S2: As Figure 2 shown, the training set video is cut into instances v i , and then input into the I3D network (Inflated 3D ConvNet, I3D) pre-trained on the Kinetic-400 dataset, so that the original feature X n of the video can be obtained.

[0042] The step S2 comprises the following steps:

[0043] S2-1: The I3D contains two parallel 3D convolution modules, the RGB image of each video instance is input into one branch of the I3D, and the visual feature of each instance is extracted;

[0044] S2-2: The optical flow image of each video instance is input into another 3D convolution branch of the I3D, and the motion feature of the video can be extracted;

[0045] S2-3: The visual feature and the motion feature are spliced to obtain the original feature X n,t of the video instance.

[0046] S3: input the original features of the video obtained in S2 into the designed causal temporal relationship module to obtain enhanced features, the specific steps comprising:

[0047] S3-1: in the video anomaly behavior detection task, there should be no information leakage from the past to the future, so the standard time modeling method is not suitable for this task. In order to solve this problem, a causal temporal relationship module is designed, the idea of which is to enhance the interaction between the current frame information and the historical frame information only with the features of the current and historical examples after obtaining the original features, and then capture the local time dependence to enhance the corresponding features. As shown in Figure 3 , the causal temporal relationship module has causality and locality at the same time, which aims to aggregate useful information from historical information and current features through a time attention mechanism. This local dependence is a kind of global dependence, and the module can be represented by the following formula:

[0048]

[0049] In the formula, represents the feature of the example at this moment, X n,t-Δt:t ={X n,t-Δt ,X n,t-Δt+1 ,...,X n,t}, X n,t-Δt:t ∈R Δt×D represents the features of the previous Δt examples, and Δt represents the number of examples in this interval. In this method, Δt is uniformly set to 9. First, X n,t and X n,t-Δt:t are projected into the embedding space by a fully connected layer with learnable weights W θ and . [] represents the dot product operation, which aims to calculate the semantic similarity between the example X n,t at this moment and the features of each example X n,t-Δt:t . Finally, the final semantic similarity matrix is calculated using softmax normalization in the local range, which is 1x Δt;

[0050] S3-2: R represents the relative position, that is, {- Δt,...,-1,0}, so that the module can capture the position prior that the self-attention network does not have. Then, the final position prior matrix is calculated using softmax normalization, and the farther the distance from the current feature, the more the information decays. Unlike the semantic similarity matrix, the position prior matrix is a constant constant that does not change with the change of the input feature;

[0051] S3-3: concatenate the semantic similarity matrix and the position prior matrix to obtain the modified similarity matrix, and multiply it with the original features of the example to obtain the enhanced features Y n,t, which contains both the current information and the interaction information between the current example feature and the previous example features. For the subsequent convolution operation, Y n,t is transformed from 2 x D to 1 x 2D by the following formula to obtain Z n,t :

[0052] Z n,t = Dropout(ReLU(Conv1D(Y n,t ))

[0053] S4: The method for obtaining the anomaly score of the video example is as follows: first, input the enhanced feature Z n,t containing time information obtained in step S3 into a fully connected network, which contains three fully connected layers and a layer of sigmoid, and each example obtains an anomaly score between 0 and 1, and the greater the score value, the greater the possibility that the model considers the segment to be an abnormal segment.

[0054] S5: After constructing the anomaly detection model based on the causal convolutional network, the model is trained by using a classification loss smoothing loss and a sparse loss, which includes the following steps:

[0055] S5-1: The anomaly score p of the example obtained in step S4 is selected, and the k examples with the highest anomaly score in each video package (in this method, the value of k is uniformly set to 4) are selected, and the average of the k scores is taken as the anomaly score of the video package. In combination with the video-level label of the training set, a brand-new classification loss is used to train the model:

[0056]

[0057] wherein ε is a threshold value for distinguishing the anomaly score of the normal and abnormal videos, p i is the top k of the anomaly score in the entire video example, is the video-level anomaly score, and y is the video-level label (the label of the abnormal video is 1, and the label of the normal video is 0). A more flexible score mapping is used here, instead of being mapped to 1 as in the standard BCE loss, and the purpose is to hope that the anomaly score of the abnormal video exceeds the set threshold.

[0058] S5-2: Since the video is a time sequence, the occurrence of the abnormal event lasts for a period of time, and therefore the segment-level anomaly score changes slowly in time. Based on this, we propose a time smoothing loss for the abnormal video, hoping that the adjacent segments in the abnormal video have similar anomaly scores, and the form is as follows:

[0059]

[0060] wherein T represents the number of examples in the video.

[0061] S5-3: Because the abnormal event occurs rarely, the abnormal segments only occupy a small part of the abnormal video, so the video-level abnormal score of the abnormal video will not be very large. In the training process, we use the following sparse loss to constrain the abnormal score of the abnormal video and the number of abnormal segments:

[0062]

[0063] S5-4: Add the classification loss, the smoothing loss and the sparse loss to obtain the global loss function L = L c + L sm + L sp , guide the training of the video anomaly detection model, and obtain the optimal model parameters.

[0064] S6: The flow of the test includes: the test set video examples obtain their original features through the S2 step, obtain their enhanced features through the S3 step, and then input into the optimal anomaly detection model obtained in the S5 step to obtain the abnormal score of each test video example, so that the frame-level score of the video is consistent with the score of the example in which the video is located. Then compare the frame-level score with the threshold value, if greater than the threshold value, it is determined to be abnormal, otherwise it is determined to be normal. The test set label is frame-level, so the predicted result can be compared with the real label to obtain the accuracy rate of video anomaly detection. The optimal anomaly detection model of the method obtains a frame-level AUC result of 96.56% on the ShanghaiTech dataset; as shown in the visualization result of Figure 4 , the curve is the abnormal score predicted by the model, and the shaded part is the real label, it can be seen that the occurrence of the abnormal event can be accurately predicted by using the method.

[0065] The above is only an example and description of the present application, and those skilled in the art of the present technology can make equivalent modifications or supplements to the described specific embodiments or replace them with similar ways, as long as they do not deviate from the structure of the invention or exceed the scope defined by the present claims, and should belong to the protection scope of the present application.

Claims

1.A method for video anomaly detection based on a cause-effect temporal relationship module, characterized in that, The method comprises the following steps: S1: first collect video data, divide it into a training set and a test set, then divide each video into mutually non-overlapping segments with the same number of frames, treat each video as a bag, and define abnormal videos and normal videos as positive bags and negative bags respectively, and treat the segments as individual examples; S2: then input the RGB images and optical flow images of the video examples in the training set and the test set into the inflated convolutional network I3D to obtain the visual features and motion features thereof, and perform splicing operations on the two to obtain the original features of the video; S3: input the original features obtained in S2 into the causal temporal convolution module to enhance the interaction between the current frame information and the historical frame information by using the current example features and the previous example features, thereby obtaining the temporal dependence relationship of the video and obtaining enhanced features containing time information; S4: input the enhanced features containing time information obtained in S3 into the fully connected layer network to obtain the abnormal score of each example, and the greater the score value, the greater the possibility of being an abnormal segment; S5: select the k examples with the largest abnormal scores in each video bag, wherein k is uniformly set to 4, and use the classification loss, the smoothing loss and the sparsity loss to train the video anomaly detection model in S3 and S4 to obtain a group of optimal model parameters; S6: in the test stage, compare the abnormal score of each example obtained with the set threshold value, and if the score is greater than the threshold value, it is determined to be abnormal, otherwise it is determined to be normal; the test set label is frame-level, and the prediction result is compared with the true label to obtain the accuracy of video anomaly detection; In the step S2, the step of extracting video features specifically comprises: S2-1: the inflated convolutional network I3D comprises two parallel 3D convolution modules, the RGB images of all examples of each video are input into one branch of the I3D, and the visual features of each example are extracted; S2-2: the optical flow images of the video examples are input into another 3D convolution branch of the I3D, and the motion features of the video are extracted; S2-3: the visual features and the motion features are spliced to obtain the original features of the video; The original features are input into the causal temporal relationship module to obtain enhanced features, which specifically comprises: S3-1: the causal temporal relationship module aims to aggregate useful information from historical information and current features through a time attention mechanism, and the formula is as follows: Among them, X n,t X represents the characteristics at this moment. n,t-Δt:t ={X n,t-Δt ,X n,t-Δt+1 ,...,X n,t } represents the features of the first Δt examples, where Δt represents the number of examples in this interval. The feature of example X at this moment is calculated using the dot product operation. n,t Compared with the first Δt examples X n,t-Δt:t The semantic similarity is then calculated, and the final feature similarity matrix is ​​obtained by normalizing the results using softmax. S3-2: R represents the relative position, i.e. {-Δt,...,-1,0}, and the final position prior matrix is calculated using softmax normalization, the farther the distance from the current distance, the more the information decays, and the position prior matrix is a constant constant that does not change with the input features; S3-3: the semantic similarity matrix and the position prior matrix are spliced, and then multiplied by the features of the example to obtain enhanced features containing local temporal dependence of the video. 2.The video anomaly detection method based on the cause-effect time relation module according to claim 1, wherein, In the step S4, the enhanced features containing time information obtained in step S3 are input into the fully connected network, which comprises three fully connected layers and a sigmoid layer, each example will obtain an abnormal score between 0 and 1, and the greater the score value, the greater the possibility that the model considers the segment to be an abnormal segment. 3.The video anomaly detection method based on the cause-effect time relationship module according to claim 1, wherein, The step S5 comprises: S5-1: Step S4 obtains the anomaly score of each example, and then selects the top k from the anomaly scores of all video examples, and uses the classification loss L combined with the video-level label to train the model c ; S5-2: A temporal smoothing loss L is proposed sm such that adjacent segments in an anomalous video have similar anomaly scores; S5-3: constrain the anomaly scores of the abnormal videos and the number of abnormal segments with a sparsity loss L sp S5-3: constrain the anomaly scores of the abnormal videos and the number of abnormal segments with a sparsity loss L S5-4: Add the classification loss, the smoothing loss and the sparsity loss to obtain the global loss function L of the model L = L c + L sm + L sp , guide training of the video anomaly detection model, and obtain optimal model parameters.

Images