Anomaly Frame Detection Method for Video Temporal Series Based on Unsupervised Frame Correlation

Abstract

An unsupervised video time-series anomaly frame detection method based on frame correlation is proposed. This method constructs training pairs containing normal and pseudo-anomaly videos. Semantic and positional encoding is performed on the video frames in the training pairs to obtain dimensionality-reduced features with temporal information. Correlation calculation is then performed based on the encoded features to obtain the global and local correlations of the video frames. The invention employs two attention distribution maps, analogous to the human process of anomaly judgment, to achieve an anomaly detection process based on similar comparisons within a video window. Anomaly scores for each frame of the target sequence are obtained by comparing the attention distribution maps. Specifically, in each target distribution map, two types of EM algorithm-like single-loop iterative updates are constructed according to actual characteristics. The anomaly model is then corrected using reconstruction loss and the k-min-max rule to obtain the patterns present in normal samples and their implicit patterns. When the input during the inference stage is anomaly, accurate anomaly detection can be achieved.

Description

Technical Field

[0001] This invention relates to a technology in the field of video processing, specifically a method for detecting abnormal frames in video time series based on unsupervised frame correlation. Background Technology

[0002] In recent years, with the deployment of more and more surveillance cameras, video anomaly detection has played an increasingly important role in intelligent monitoring systems to reduce manual labor in on-site monitoring. However, in most cases, anomalies account for a very small proportion of real-world data, so the collected data often consists mostly of normal data. In most solutions, the use of samples is limited to normal samples, i.e., unsupervised learning, defining normal patterns by acquiring the regularity of normal samples, and defining samples that deviate from normal patterns as anomalies. However, due to the lack of supervisory signals from anomaly samples, the model is difficult to be sufficiently sensitive to anomalies. An improved solution is to maintain the unsupervised mode that conforms to the conditions of real-world data collection, and in the instance-related anomaly definition mode, to use pseudo-anomaly samples constructed from normal samples to provide supervisory signals to the model instead of the unavailable true anomaly samples. At the same time, pseudo-anomaly samples and normal samples are used for joint training, thereby solving the problem of missing anomaly samples during model training.

[0003] A search of existing technologies revealed that Chinese patent document CN112084887A, published on December 15, 2020, discloses an adaptive video classification method and system based on an attention mechanism. This method first acquires an unsupervised video stream as the target domain video stream; then acquires labeled image and video data as the source domain dataset; next, it uses a pre-defined domain adaptive model to train features on the target domain video stream and the source domain dataset to output feature parameters for the target domain video stream; finally, it classifies the target domain video stream based on these feature parameters to generate category labels. However, this technique still requires fully labeled data as a training set, and in most scenarios, it is difficult to obtain abnormal samples, resulting in a dataset containing only normal samples, which cannot solve the unsupervised training task. Furthermore, when extracting features from the target and source domain video streams, it still maintains a pixel semantic learning approach, causing the model to fail to learn temporal correlation information during training. In addition, the learning objective of this technique is to determine normality and abnormality through category labels, which cannot solve the problem of abnormal judgment when out-of-category events occur in the test set. Summary of the Invention

[0004] To address the aforementioned shortcomings of existing technologies, this invention proposes an anomaly detection method for video time series based on unsupervised frame correlation. By employing two attention distribution maps and analogizing the human process of anomaly judgment, it implements an anomaly detection process based on similar comparisons within a video window. The anomaly score for each frame of the target sequence is obtained by comparing the attention distribution maps. Specifically, for each target distribution map, two types of EM algorithm-like single-loop iterative updates are constructed based on actual characteristics. The anomaly model is then corrected using reconstruction loss and the k-min-max rule to obtain the patterns present in normal samples and their latent patterns. When the input during the inference stage is anomaly, accurate anomaly detection can be achieved.

[0005] This invention is achieved through the following technical solution:

[0006] This invention relates to an anomaly frame detection method for video time series based on unsupervised frame correlation. The method constructs training pairs containing normal and pseudo-anomaly videos. Semantic and positional encoding are performed on the video frames in the training pairs to obtain dimensionality-reduced features with temporal information. Correlation is calculated based on the encoded features to obtain the global and local correlations of the video frames. Correlation constraints are constructed for the local correlation features of normal and pseudo-anomaly videos. To avoid model failure due to excessively rapid convergence of local correlations, separate global correlation models are constructed for normal videos and pseudo-anomaly videos. and The global correlation feature focuses on the correlation of long-distance frames. For training pairs consisting of normal videos and pseudo-abnormal videos, a k-min-max rule is constructed to further constrain the global correlation feature using a hinge loss function. At the same time, the video frames selected by the k-min-max strategy are fed into the instance scoring model to calculate the scoring loss.

[0007] The training set is obtained as follows: the input original video is decomposed into several frames using ffmpeg as the original data; overexposed images, images with ghosting, and other poor image quality are removed from the original data; the original images are decomposed into several time windows according to a preset time window and step size, and each video segment will generate... The data augmentation is achieved by using multiple video clips, with the resulting video clips being F = [[x1, x2, x3, ..., x...]. k ], [x 1+s x 2+s x 3+s , ..., x k+s ],...,[x T-k+1 x T-k+2 x T-k+3 , ..., x TConstructing abnormal video segments from normal video segments: Replace the current scene's video segment with a normal video segment from another scene to construct a pseudo-abnormal video F = [x1, x2, x3, ..., x...]. j-1 g1, g2, g3, ..., g m x j+m , ..., x k ], where: k is the normal video length, and: frames j to j+m-1 are replaced with, [g1, g2, g3, ..., g m Record the location of the anomaly.

[0008] The process of performing semantic encoding and positional encoding to obtain dimensionality-reduced features with positional information specifically involves: Where: t k For video frame x k Input network We get x k The result after semantic encoding It is a convolutional layer. p k The result of the position encoder, with x k The specific vector values ​​are irrelevant. The implementation uses a trigonometric function feature encoder; specifically... Where s is the frame number and i is the feature number. This identifies the value of the i-th dimension vector of the feature in the s-th frame. k increases sequentially from 1 to d / 2, where d represents x. k The feature dimension is used to obtain the final extracted feature f. k =p k +t k .

[0009] The global and local correlations of the video frames are calculated using the following method: Where: f l The same as described above refers to the feature output of the l-th layer model. N represents the video segment window size, and Q, K, and V represent the query, key, and value in the attention model, respectively. σ is the variance of the normal distribution learned by the model in the i-th frame. Let be the model parameters corresponding to the unknowns in the l-th layer. l represents the number of model layers.

[0010] The aforementioned abnormal frame detection method specifically includes the following steps:

[0011] The first step is to use a portion of the attention map in the attention model to calculate the global correlation between frames. in: N is the size of the video window.

[0012] Step 2: Solve for local correlations Where: Exp is the exponential function, i and j are the frame numbers, and N is the size of the video window. σ i The variance of the i-th video frame obtained from the model is calculated under a normal distribution.

[0013] Step 3: Through the reconstruction project Calculate the difference between global correlation and local correlation Where: KL is the KL divergence calculation. For the i-th frame of the l-th frame and the global correlation within the video window, For the i-th frame of the l-th layer model, score(G, C; F) is the set of abnormal evaluations obtained for each frame in the current video window. Using this as input to the (l+1)th layer model, the above calculations are performed again to obtain the global relevance features of the (l+1)th layer. and local correlation features

[0014] When the input is a normal video, F = [x1, x2, x3, ..., x...]. k Each x in ] i All of these are normal. Therefore, considering the overall reconstruction error, a local correlation constraint loss is constructed to constrain C. score(G detach C)||1, where ||·||1 is the 1-norm calculation. F is the feature input of the original video sequence. For video sequence reconstruction, ||·|| fro λ is the Frobenius norm, used to balance reconstruction error and outlier score evaluation.

[0015] Similarly, a global correlation constraint loss for normal video is constructed to constrain G. Let the pseudo-anomaly video be F = [x1, x2, x3, ..., x...]. j-1 g1, g2, g3, ..., g m x j+m , ..., x k ], where: k is the normal video length, and: frames j to j+m-1 are replaced with, [g1, g2, g3, ..., g mAt this point, global and local correlations are still calculated. For normal frames x in pseudo-abnormal videos i For G and C, the optimization is the same as for frames in normal video. For abnormal frames g in pseudo-abnormal video... i For the optimization of C, the local correlation constraint loss is used in the same way as for frames in normal video. For G, a global correlation constraint loss for pseudo-anomaly videos is constructed to constrain it. Where: E(·) is the value of all frame numbers of the specified feature type in global correlation.

[0016] For the global correlation G of pseudo-anomaly videos ab Global correlation G with normal video no Calculate the L2 norm value for each frame in the video and set the min-max rule for the pseudo-anomaly video G. ab In the process, the global features of the top k video frames with smaller L2 norms are selected. For normal videos, the global correlation G... no Global features are selected from the top k video frames with the largest L2 norm. The hinge loss function, Loss, is then calculated. dis =D-(g score (G ab )-g score (G no ), where: D is a preset value, ensuring Loss dis Greater than 0. ||·||2 is the 2-norm, Γ k (G ab ) is the set of global features of the first k video frames with smaller L2 norms, and N is the number of video frames.

[0017] Similarly, Ω k (G ab ) is the set of global features of the first k video frames with larger L2 norms.

[0018] For instance classification, let the pseudo-anomaly videos be F = [x1, x2, x3, ..., x...]. j-1 g1, g2, g3, ..., g m x j+m , ..., x k For a normal video frame x i Set the label to 0 for pseudo-abnormal video frames g. i Set the label to 1. Calculate the classification loss.

[0019] Where: f γ Calculate the anomaly score for the linear layer. The above annotations are for different video frames; other identifiers have the same meaning as those mentioned above.

[0020] This invention relates to a system for implementing the above-mentioned method, comprising: a pseudo-anomaly generation module, a feature encoding module, a correlation calculation module, a correlation constraint module, a k-min-max constraint module, and an anomaly scoring module. The pseudo-anomaly generation module performs video reassembly processing based on normal video information in different scenarios to obtain training pairs of pseudo-anomaly videos and normal videos, along with their annotation results. The feature encoding module performs semantic and positional encoding processing based on the pseudo-anomaly videos and normal videos to obtain pseudo-anomaly video features and normal video features with temporal positional and semantic information. The correlation calculation module obtains global correlation features based on the encoded features using an attention model, and then obtains the correlation features by fitting a Gaussian distribution. The local correlation feature and correlation constraint module apply local correlation feature constraints based on global and local correlation features, as well as global correlation constraints for normal and pseudo-anomaly videos, to obtain local correlation features that focus more on the neighborhood and global correlation features that focus more on the entire domain. The k-min-max constraint module uses the hinge loss function based on the global correlation features obtained by the correlation calculation module to obtain a global correlation calculation model that is sensitive to pseudo-anomalies. The anomaly scoring module uses the anomaly scoring model based on the global correlation features of normal and pseudo-anomaly videos, and uses the annotation results obtained by the pseudo-anomaly generation module as model constraints to obtain an anomaly score for each video frame.

[0021] Technical effect

[0022] This invention transforms unsupervised video anomaly detection into a supervised anomaly detection problem by constructing pseudo-anomalies. Anomalies are defined by correlation analysis of frames under global video conditions, and different correlation loss constraints are defined based on the different characteristics of normal and pseudo-anomalies in the global correlation. The k-min-max rule is constructed to constrain the norm of the extracted features, which improves the detection accuracy when the model does not know the anomaly signal. Compared with existing methods, the anomaly detection recognition rate is increased by 3.8%. Attached Figure Description

[0023] Figure 1 This is a flowchart of the present invention;

[0024] Figure 2 This is a schematic diagram of an embodiment;

[0025] Figure 3 This is a schematic diagram of a pseudo-exception construction module. Detailed Implementation

[0026] The data in this embodiment comes from the publicly available video anomaly detection dataset ShanghaiTech. During training, the training set consists of 175 normal videos; the test set consists of 199 videos, including 155 normal videos and 44 anomalous videos. The video window size is set to 20, the model layer number l is set to 3, the stride is set to 1, k in the k-min-max rule is set to 3, λ is set to 3, and the batch size is set to 16. The semantic feature extractor is 2048*512. During training, Adam is selected as the optimizer, and the training epochs are set to 100. The model parameters are obtained through backpropagation. In the testing phase, the input is a video sequence with a length equal to the video window value. After calculating the global relevance features, the anomaly score for each video frame is obtained through the instance scoring model fγ.

[0027] like Figure 1 As shown in the figure, this embodiment relates to an anomaly frame detection method for video time series based on unsupervised frame correlation, including:

[0028] Step 1: Obtain multiple consecutive normal video sequences as a training set, and remove segments with poor quality such as overexposure and ghosting;

[0029] Step 2: For the normal video processed in Step 1, extract segments from other normal videos with different scenes and replace them with segments of the same length from the normal videos to form pseudo-abnormal videos. The pseudo-abnormal videos and normal videos form training pairs.

[0030] Step 3: Use convolutional networks Semantic features are extracted from pseudo-abnormal videos and normal videos respectively, and positional features are calculated. The semantic features and positional features are added together to obtain the training features of pseudo-abnormal videos and normal videos.

[0031] Step 4: Input the pseudo-anomaly training features and normal training features into the global and local feature models to obtain global features, and then apply the correlation-constrained loss function. Constraining the local features of pseudo-abnormal videos and normal videos, and applying the global correlation loss function of normal videos. Constraining the global features of normal videos, and using the global correlation loss function of pseudo-abnormal videos. Constrain the global features of pseudo-anomaly videos.

[0032] Step 5: Use the k-min-max rule to filter global features between pseudo-abnormal videos and normal videos, and then apply the features of the selected video frames to the hinge loss function. dis Constrain global features.

[0033] Step 6: Input the features of the video frames selected in Step 5 into the training scorer model fγ to obtain global features.

[0034] Step 7: Construct reconstruction terms for normal and pseudo-abnormal videos using global features. Input the reconstruction terms into the global and local feature models and repeat steps 4-6 until the number of iterations reaches the preset model layer number l, thus achieving the training of the global and local feature models.

[0035] Step 8: In the online phase, abnormal frame detection is performed using the trained global and local feature models.

[0036] like Figure 2 As shown, the abnormal frame detection system implementing the above method in this embodiment includes: a pseudo-anomaly generation module, a feature encoding module, a correlation calculation module, a correlation constraint module, a k-min-max constraint module, and an anomaly scoring module. Specifically: the pseudo-anomaly generation module performs video reassembly processing based on normal video information in different scenarios to obtain training pairs of pseudo-anomaly videos and normal videos, along with their annotation results; the feature encoding module performs semantic and positional encoding processing based on the pseudo-anomaly videos and normal videos to obtain pseudo-anomaly video features and normal video features with temporal positional and semantic information; the correlation calculation module uses an attention model to obtain global correlation features based on the encoded features, and then fits a Gaussian score... The local correlation features are obtained. The correlation constraint module performs local correlation feature constraints based on the global correlation features and local correlation features, as well as global correlation constraints for normal videos and pseudo-anomaly videos, to obtain local correlation features that focus more on the neighborhood and global correlation features that focus more on the whole domain. The k-min-max constraint module uses the hinge loss function based on the global correlation features obtained by the correlation calculation module to obtain a global correlation calculation model that is sensitive to pseudo-anomalies. The anomaly scoring module uses the anomaly scoring model based on the global correlation features of normal videos and pseudo-anomaly videos, and uses the annotation results obtained by the pseudo-anomaly generation module as model constraints to obtain the anomaly score for each video frame.

[0037] The aforementioned abnormal frame detection system defines anomalies as video frames in a video window that have low correlation with other frames, making it feasible to construct pseudo-anomalies using normal video frames. At the same time, different constraints are applied to normal videos and pseudo-abnormal videos, so that the model's learning of global features can be applied to different videos to achieve anomaly judgment for video frames. The comparison with other unsupervised video anomaly detection methods and the contribution effect of each module are shown in the table. The index is the general anomaly detection index AUC.

[0038]

[0039] Specific practical experiments demonstrate that this method outperforms common unsupervised video anomaly detection algorithms. In the table above, the "None loss" term represents the performance when no constraints are imposed on the global correlation of the model. This indicates that only local correlation is used to constrain global correlation, only(Loss dis This indicates that only the k-min-max rule is used to constrain global correlation. As can be seen from the above experiments... The optimal performance is achieved by applying both constraints simultaneously, which is superior to other unsupervised video anomaly detection methods. Furthermore, both constraints optimize the None loss term.

[0040] Compared to existing technologies, this invention uses only normal samples for training, reducing the difficulty of acquiring the dataset. The definition of anomalies differs from traditional methods that define pixel anomalies or optical flow anomalies. This method defines anomalies as video frames whose global correlation within the video window differs from other samples. This definition fully integrates temporal information into the features. Furthermore, when events outside the training set occur in the test set, anomaly samples can still be identified using correlation features.

[0041] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A method for detecting abnormal frames in a video time series based on unsupervised inter-frame correlation, characterized in that, Comprise: Step one: obtain a plurality of continuous normal video sequences as a training set, eliminate the poor quality of the segment, specifically: the input original video is decomposed into several frames as original data by ffmpeg, and the image quality of the original data is deleted; Step two: for the normal video processed in step one, extract the segments in other normal videos with different scenes from it, replace the same length segments in the normal video to form a pseudo abnormal video, and form a training pair with the normal video, specifically: according to the preset time window and step, it is decomposed into several time windows, each video segment will produce video segments, achieving the effect of data augmentation, and the video segments obtained from a single video are ; construct abnormal video segments through normal video segments: use normal video segments in other scenes to replace video segments in the current scene to construct pseudo abnormal videos , wherein: k is the length of the normal video, wherein: the jth frame to the j+m-1 frame is replaced by , record the position of the anomaly; Step three: using convolutional network The semantic features of the pseudo abnormal video and the normal video are extracted respectively, and the position features are calculated, and the semantic features and the position features are added to obtain the training features of the pseudo abnormal video and the normal video, specifically: Wherein: is the video frame Input the convolutional network Obtained, that is The result after semantic coding, , is the result of the position encoder, and The specific vector value is irrelevant, and the implementation is a trigonometric function feature encoder, specifically, , Wherein is the frame number, is the feature serial number, Indicates the first Frame feature first Dimensional vector value, From 1 to , Indicates Feature dimension, the final extracted feature is ; Step four: input the pseudo-abnormal training features and normal training features into the global feature and local feature model to obtain the global feature, and constrain the local features of the pseudo-abnormal video and the normal video through the global correlation loss function of the normal video constrain the global feature of the normal video through the global correlation loss function of the pseudo-abnormal video constrain the global feature of the normal video through the global correlation loss function of the pseudo-abnormal video constrain the global feature of the pseudo-abnormal video Step five: using k-min-max rule to screen the global features of pseudo abnormal videos and normal videos, and passing the selected video frame features through the hinge loss function Constrained global features; Step six: input the features of the video frame selected in step five into the trained scorer model to obtain global features ​ Step seven: build the reconstruction term of normal video and pseudo abnormal video through global feature, input the reconstruction term into global feature and local feature model and repeat step four-step six until the cycle number reaches the preset model layer number , realize the training of global feature and local feature model; Step eight: in the online phase, the global feature and local feature model trained are used for abnormal frame detection.

2. The method of claim 1, wherein the step of detecting the abnormal frame is characterized by, The normal video segment refers to: the input original video is decomposed into several frames as original data by ffmpeg, and the images with poor image quality are deleted from the original data; the original images are decomposed into several time windows according to a preset time window and step length, and each video segment will generate video segments, achieving the effect of data augmentation, and the video segments obtained from a single video are .

3. The method of claim 1, wherein the step of detecting the abnormal frame is characterized by, The global and local correlations of the video frames are calculated using Q, K, V. The method is as follows: ,in: For the first Feature output of layer model , The size of the video segment window. These represent the query, key, and value in the attention model. , Let be the variance of the normal distribution learned by the model for the i-th frame. , For the first The model parameters corresponding to the layers are unknowns. This represents the number of model layers.

4. An abnormal frame detection system for implementing the abnormal frame detection method of any one of claims 1-3 based on unsupervised lower frame correlation, characterized in that, Comprise: The pseudo anomaly generation module, the feature encoding module, the correlation calculation module, the correlation constraint module, the k-min-max constraint module and the anomaly scoring module, wherein: the pseudo anomaly generation module generates video recombination processing according to the normal video information in different scenes, obtains the pseudo anomaly video and normal video training pair and its label result, the feature encoding module encodes and position encodes the pseudo anomaly video and normal video to obtain the pseudo anomaly video feature and normal video feature with time sequence position information and semantic information, the correlation calculation module obtains global correlation features by using an attention model according to the encoded features, and obtains local correlation features by fitting a Gaussian distribution, the correlation constraint module performs local correlation feature constraint, normal video global correlation constraint and pseudo anomaly video global correlation constraint according to the global correlation features and the local correlation features, to obtain local correlation features that pay more attention to the neighborhood and global correlation features that pay more attention to the whole, the k-min-max constraint module obtains a global correlation calculation model sensitive to pseudo anomalies by using a hinge loss function according to the global correlation features obtained by the correlation calculation module, and the anomaly scoring module uses an anomaly scoring model according to the global correlation features of the normal video and the pseudo anomaly video, uses the label result obtained by the pseudo anomaly generation module as a model constraint, and obtains the anomaly score of each video frame.

5. The system of claim 4, wherein, The abnormal frame detection system defines the anomaly as a video frame with low correlation with other frames in the video window, so that it is feasible to use normal video frames to construct pseudo anomalies, and different constraint conditions are used for normal videos and pseudo anomaly videos, so that the learning of the model for global features can be used in different videos, the abnormality of the video frame is judged, and the comparison with other unsupervised video anomaly detection methods and the contribution effect of each module are shown in the table, and the index is the abnormal detection general index AUC.

Images