Video anomaly detection device and detection method
By improving the self-attention mechanism and feature generation model, combining the conditional variational automatic encoder and the contrast language-image pre-training model to generate generalized features, the problem of insufficient modeling of different abnormal features of video anomaly detection models is solved, and significantly improved Detection accuracy and performance.
Patent Information
- Application Number
- CN202411116866.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-15
AI Technical Summary
The existing video anomaly detection model has weak modeling of different anomaly features, resulting in poor detection performance.
A video anomaly detection device is designed, including a feature generation model and a video anomaly detection model. Through the feature extraction module, feature enhancement module and anomaly scoring module, the global and local timing feature enhancement modules are used to improve the self-attention mechanism, combined with the conditional variable segmentation Autoencoder and contrast language-image pretrained models to generate generalized features to improve detection accuracy.
It effectively improves the generalization ability and accuracy of video anomaly detection, can better model the difference between abnormal features and normal features, reduces the risk of overfitting abnormal features, and improves detection performance.
Smart Images

Figure CN118644810B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and more specifically, to a video anomaly detection device and a detection method. Background Art
[0002] Surveillance cameras are everywhere in cities. As an important part of the city's security system, video surveillance exists to record everything that happens in the city. Abnormal event detection based on surveillance videos is an important means to improve the level of public safety.
[0003] The video anomaly detection algorithm based on deep learning uses the powerful learning ability of deep learning to learn the behavior patterns and scene information of normal and abnormal videos, and judges the abnormal phenomena in the video based on this. The video anomaly detection method based on strong supervision requires the dataset to provide video clip-level labels for the training data during training. This setting requires a huge labeling cost, while the method based on weak supervision only requires the dataset to have video-level labels. Although the detection efficiency is lower than that of the anomaly detection method based on strong supervision, it can reduce the cost of video labeling. Therefore, the anomaly detection method based on weak supervision is more applicable in actual scenarios.
[0004] Most of the existing weakly supervised video anomaly detection methods are based on multi-instance learning methods, that is, a video is divided into several non-overlapping video clips, each video clip is taken as an instance, and all instances in a video constitute a package. The video packages are divided into positive packages and negative packages according to whether they contain abnormal instances (positive packages contain abnormal instances, while negative packages are all normal instances). Then, each instance in the package is ranked by abnormality score, and several instances with the highest scores in positive and negative packages are selected, and the loss function is designed to widen the gap between them.
[0005] At present, there are still some defects in the video anomaly detection method based on weak supervision. On the one hand, the data sets used by the existing video anomaly detection models usually use pre-extracted features. The common pre-extracted features are I3D features. Most of them are based on the self-attention mechanism to enhance the I3D features and obtain temporal features with temporal information. At the same time, in the self-attention mechanism, the relative position relationship between video clips, that is, the position encoding information, is also considered. However, due to the sparsity and continuity of abnormal clips in abnormal videos, the current position encoding information cannot model the relationship between video clips well. Therefore, there is still room for improvement in the position encoding information. On the other hand, most video anomaly detection models only consider the distinction between normal features and abnormal features, and increase the difference between normal features and abnormal features through the loss function, but ignore the separability between abnormal features, such as fighting and running. In addition, another part of the video anomaly detection models consider the separability between abnormal features, but these models will cause the abnormal features to overfit when solving this problem, so it is necessary to model the abnormal features more accurately.
[0006] After searching, the Chinese patent application, application number 202110993367.8, published on September 24, 2021, discloses a video anomaly detection model training method, a video anomaly detection method and a device. The scheme includes: obtaining a training video frame sequence, corresponding sequence feature information and target feature information of a training screening video frame corresponding to the training video frame sequence; inputting the training video frame sequence into the initial video anomaly detection model to obtain the prediction feature information corresponding to the training screening video frame; generating a temporal positive training sample of a temporal discriminant model corresponding to the initial video anomaly detection model based on the target feature information and the sequence feature information, and generating a temporal negative training sample based on the sequence feature information and the prediction feature information; conducting adversarial training on the initial video anomaly detection model and the temporal discriminant model based on the target feature information, the prediction feature information, the temporal positive training sample and the temporal negative training sample until the convergence condition is met to obtain the target video anomaly detection model. The scheme can improve the accuracy of the video anomaly detection model, but the scheme does not take into account the problem of overfitting when the video anomaly detection model models different abnormal features. Summary of the invention
[0007] 1. Technical issues to be solved
[0008] In view of the problem that the video anomaly detection model in the prior art has weak modeling of different abnormal features, the present invention provides a video anomaly detection device and a detection method, which can achieve more generalized modeling of abnormal features and improve video anomaly detection performance.
[0009] 2. Technical solution
[0010] The purpose of the present invention is achieved through the following technical solutions.
[0011] A video anomaly detection device, comprising: a feature generation model and a video anomaly detection model;
[0012] The feature generation model includes a feature extraction module, a first feature enhancement module and a feature sampling module;
[0013] The feature extraction module extracts the original features of the video clip in the video to be detected; the original features are input into the first feature enhancement module for feature enhancement to obtain the first time series features; the feature sampling module reconstructs the first time series features to obtain the generalized features;
[0014] The video anomaly detection model includes a feature extraction module, a second feature enhancement module and an anomaly scoring module;
[0015] The feature extraction module extracts the original features of the video clip in the video to be detected; the original features are input into the second feature enhancement module for feature enhancement to obtain the second time series features; the generalized features obtained by the feature generation model and the second time series features are aligned, and the second time series features are constrained by the generalized features; the anomaly score module calculates the second time series features to obtain the anomaly score, and determines whether the video clip is abnormal based on the anomaly score.
[0016] Further, the first feature enhancement module includes a global temporal feature enhancement module and a local temporal feature enhancement module;
[0017] The global timing feature enhancement module generates a global timing feature;
[0018] The local time series feature enhancement module generates local time series features;
[0019] The global timing feature and the local timing feature are added together to obtain a first timing feature.
[0020] Furthermore, the global timing feature enhancement module generates global timing features, and the steps include:
[0021] The original features are input into the fully connected layer of the global temporal feature enhancement module to extract the global query features, global key features and global value features;
[0022] Perform matrix multiplication of the global query feature and the transposed global key feature to obtain a relevance matrix;
[0023] The correlation matrix is enhanced by using position coding to obtain the final correlation matrix;
[0024] Normalize the final correlation matrix to generate a global attention weight map;
[0025] Perform matrix multiplication of the global value feature and the global attention weight map to obtain the global temporal weight value;
[0026] The global time series weight value and the original feature are matrix-added to obtain the global time series feature.
[0027] Furthermore, the calculation formula for enhancing the correlation matrix using position coding is:
[0028]
[0029] Among them, G represents the position correlation matrix, i and j represent the sequence number of the video clip, G i,j represents the degree of position correlation on (i, j) in the position correlation matrix, exp represents the exponential function, γ and β represent the learnable parameters in training, H represents the feature norm difference matrix, and H i,j Represents the feature norm difference at (i, j) in the feature norm difference matrix, -H i,j Represents the negative of the feature norm difference at (i, j) in the feature norm difference matrix, H i represents the feature norm of the i-th video clip, H j represents the feature norm of the jth video clip, C represents the total number of channels of the video clip feature, c represents the number of channels of the video clip feature, and f i,c The eigenvalue representing the c-th channel number of the ith video clip.
[0030] Furthermore, the local temporal feature enhancement module includes a multi-head attention module; the local temporal feature enhancement module generates local temporal features, and the steps include:
[0031] The original features are input into the fully connected layer of the local temporal feature enhancement module, and local query features, local key features and local value features are obtained through feature extraction;
[0032] Set the attention mask matrix in the multi-head attention module;
[0033] Input local query features, local key features and local value features into the multi-head attention module, calculate the attention weights of local query features, local key features and local value features according to the attention mask matrix, and obtain the local temporal weight values;
[0034] The local time series weight value and the original feature are matrix-added to obtain the local time series feature.
[0035] Furthermore, the attention weights of the local query features, local key features, and local value features are calculated according to the attention mask matrix. The calculation formula is:
[0036]
[0037] Among them, mask represents the attention mask matrix, i and j represent the sequence number of the video clips, T represents the number of video clips, and mask i,j Indicates whether to perform local attention calculation on the i-th video clip and the j-th video clip, Attention indicates the local attention weight value, m indicates the head number of the multi-head attention module, h indicates the total number of heads of the multi-head attention module, and f qb represents the local query feature, f kb represents the local key feature, f vb represents the local value feature, represents the local query feature of the mth head, represents the local key feature of the mth head, represents the local value feature of the mth head, softmax() represents the softmax activation function, d k represents the feature dimension, τ represents the transposition operation, and Concat represents the dimension concatenation. Represents the local timing weight value.
[0038] Furthermore, the feature sampling module includes a conditional variational autoencoder and a contrastive language-image pre-training model; the feature sampling module performs feature reconstruction on the first temporal feature to obtain a generalized feature, and the steps include:
[0039] Setting a semantic information label, concatenating the first temporal feature and the semantic information label to obtain a pre-reconstruction feature;
[0040] The pre-reconstructed features are input into the encoder of the conditional variational autoencoder to obtain the mean and variance;
[0041] Random sampling from a normal distribution and combining with the mean and variance to generate an intermediate latent variable;
[0042] Concatenate the intermediate latent variables with the semantic information labels to obtain the intermediate features;
[0043] Input the intermediate features into the decoder of the conditional variational autoencoder to obtain generalized features;
[0044] By comparing the language-image pre-training model, semantic features are generated, and generalization features are constrained by semantic features.
[0045] Furthermore, the loss function of generalizing features by constraining semantic features is expressed as:
[0046]
[0047] Among them, L1 represents the one-stage alignment loss function, max represents the maximum value function, d represents the feature distance, and f token represents the semantic feature, fin represents the first time series feature, f out represents the generalization feature, d(f token ,f in ) represents the distance between the semantic feature and the first temporal feature, d(f token ,f out ) represents the distance between the semantic feature and the generalized feature, f1 represents the representative feature, f2 represents another representative feature, d(f1,f2) represents the distance between the two representative features, C represents the total number of channels of the video clip feature, c represents the number of channels of the video clip feature, It represents the eigenvalue of the feature on the c channel. represents the eigenvalue of another feature on the c channel. represents the two-norm.
[0048] A video anomaly detection method comprises the following steps:
[0049] Phase 1:
[0050] Obtain the video to be detected, and the feature extraction module extracts the original features of the video clip in the video to be detected;
[0051] The feature enhancement module performs feature enhancement on the original feature to obtain the first time series feature;
[0052] The feature sampling module reconstructs the first time series feature to obtain a generalized feature;
[0053] Phase 2:
[0054] Obtain the video to be detected, and the feature extraction module extracts the original features of the video clip in the video to be detected;
[0055] The second feature enhancement module performs feature enhancement on the original feature to obtain a second time series feature;
[0056] Perform feature alignment operation on the generalized features obtained in the first stage and the second time series features, and constrain the second time series features through the generalized features;
[0057] The anomaly score module calculates the second temporal feature to obtain an anomaly score, and determines whether the video clip is abnormal according to the anomaly score.
[0058] Furthermore, the loss function of constraining the second time series feature by the generalized feature obtained in the first stage is expressed as:
[0059]
[0060] Among them, L alignrepresents the loss function of the second temporal feature constrained by the generalized feature, C represents the total number of channels of the video clip feature, c represents the number of channels of the video clip feature, T represents the total number of video clips, i represents the i-th video clip, and f out represents the generalization feature, represents the feature value of the generalized feature of the i-th video clip on the c-th channel, represents the two-norm, represents the second timing characteristic, Represents the feature value of the second temporal feature of the i-th video clip on the c-th channel.
[0061] 3. Beneficial effects
[0062] Compared with the prior art, the advantages of the present invention are:
[0063] (1) A video anomaly detection device and detection method of the present invention, because the abnormal segments in the video to be detected are sparse and continuous, by constructing a feature enhancement module and improving the position encoding information in the self-attention mechanism, the video anomaly detection device can pay more attention to the relationship between adjacent video segments, effectively improving the efficiency of video anomaly detection.
[0064] (2) A video anomaly detection device and detection method of the present invention, by constructing a feature sampling module, can randomly sample from a normal distribution based on a conditional variational autoencoder to obtain more generalized features. The generalized features have the modeling of both temporal features and semantic features, and the temporal features pay more attention to the influence of the feature norm, so that the feature generation model can pay more attention to the difference between abnormal features and normal features. The semantic features are derived from a normal distribution and are therefore more generalized, which helps the two-stage video anomaly detection model to more accurately model the abnormal features and effectively improve the accuracy of video anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 This is a schematic diagram of the structure of a video anomaly detection device according to an embodiment of the present invention;
[0066] Figure 2 This is a schematic diagram of the structure of the first feature enhancement module according to an embodiment of the present invention;
[0067] Figure 3 This is a schematic diagram of the structure of a feature sampling module according to an embodiment of the present invention;
[0068] Figure 4 This is a schematic diagram of the structure of an abnormality scoring module according to an embodiment of the present invention;
[0069] Figure 5 It is a schematic diagram for guiding the implementation of the present invention;
[0070] Figure 6Schematic diagram of L1 loss function according to an embodiment of the present invention;
[0071] Figure 7 Schematic diagram of the L2 loss function of an embodiment of the present invention. DETAILED DESCRIPTION
[0072] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0073] Example 1
[0074] like Figure 1 As shown, a video anomaly detection device provided by the present embodiment. A video anomaly detection device includes a feature generation model and a video anomaly detection model. The feature generation model includes a feature extraction module, a first feature enhancement module and a feature sampling module; the video anomaly detection model includes a feature extraction module, a second feature enhancement module and an anomaly scoring module. The feature extraction module extracts the original features of the video clip in the video to be detected, inputs the original features into the first feature enhancement module for feature enhancement to obtain the first time series features, and the feature sampling module performs feature reconstruction on the first time series features to obtain generalized features; the second feature extraction module extracts the original features of the video clip in the video to be detected, inputs the original features into the second feature enhancement module for feature enhancement to obtain the second time series features, performs feature alignment operation on the generalized features obtained by the feature generation model and the second time series features, constrains the second time series features by the generalized features, and the anomaly scoring module calculates the second time series features to obtain the anomaly score, and determines whether the video clip is abnormal according to the anomaly score.
[0075] Specifically in this embodiment, the feature generation model includes a feature extraction module, a first feature enhancement module and a feature sampling module.
[0076] Specifically, for the feature extraction module, the original features of the video clips in the video to be detected are extracted by the feature extraction module. In this embodiment, the feature extraction module uses the pre-trained I3D network, C3D network or ResNet network in the prior art as the network structure of the feature extraction module to perform feature extraction. Furthermore, in this embodiment, the I3D network is used to extract the original feature X of the video clip in the video to be detected, and the feature dimension of the original feature X is [T, D], where T represents the number of video clips in the video to be detected, and D represents the feature dimension of each video clip.
[0077] In the prior art, video anomaly detection generally uses a 3D convolutional neural network pre-trained on an action recognition dataset as a feature extractor to extract features, and then sends the extracted features to the anomaly score calculation model to calculate the video anomaly score. However, due to the differences between action recognition and anomaly detection tasks, the extracted features are suboptimal when applied to anomaly detection tasks. At the same time, given the applicability and effectiveness of the self-attention mechanism in capturing key video features, the prior art uses the self-attention mechanism to enhance the original features, making the original features more suitable for the field of video anomaly detection. However, due to the sparseness and continuity of abnormal clips in the video, the prior art cannot make good use of position encoding information and semantic information to model the relationship between video clips, resulting in low accuracy in video anomaly detection.
[0078] It should be noted that the abnormal segments in the video have sparseness and continuity, which means that in the existing weakly supervised video anomaly detection, the normal video only contains normal segments, and the abnormal video contains normal segments and abnormal segments. Since the number of abnormal segments accounts for a relatively small number of the entire abnormal video, it has sparseness, and abnormal events often occur in a continuous time, so it also has continuity. Therefore, the first feature enhancement module provided in this embodiment is a feature enhancement module based on position coding.
[0079] like Figure 2 As shown, for the first feature enhancement module, the first feature enhancement module includes a global timing feature enhancement module and a local timing feature enhancement module. In this embodiment, the first feature enhancement module is composed of a global timing feature enhancement module and a local timing feature enhancement module in parallel. The global timing feature enhancement module generates a global timing feature, the local timing feature enhancement module generates a local timing feature, and the global timing feature and the local timing feature are added to obtain the first timing feature.
[0080] For the global temporal feature enhancement module, the global temporal feature enhancement module is composed of a self-attention mechanism and position encoding. In this embodiment, the global temporal feature is generated by the global temporal feature enhancement module, and the steps include: inputting the original feature into the fully connected layer of the global temporal feature enhancement module to obtain global query features, global key features and global value features through feature extraction; performing matrix multiplication of the global query feature and the transposed global key feature to obtain a correlation matrix; using position encoding to enhance the correlation matrix to obtain the final correlation matrix; normalizing the final correlation matrix to generate a global attention weight map; performing matrix multiplication of the global value feature and the global attention weight map to obtain a global temporal weight value; performing matrix addition of the global temporal weight value and the original feature to obtain a global temporal feature.
[0081] Specifically, in the global temporal feature enhancement module, the original feature X is input into the fully connected layer Q, fully connected layer K and fully connected layer V of the global temporal feature enhancement module for feature extraction to obtain the global query feature f q (X), global key feature f k (X) and the global value feature f v (X). It should be noted that since the network structures of the fully connected layers Q, K and V are the same, the global query feature f q (X), global key feature f k (X) and the global value feature f v The feature dimensions of (X) are also the same, both [T, D]. The global query feature f q (X), global key feature f k (X) and the global value feature f v (X) is used for the calculation of the self-attention mechanism in the subsequent steps. Further, the global query feature f q (X) and the transposed global key feature f k (X) to obtain the correlation matrix M of the video clip. In this embodiment, the global query feature f q (X) and the transposed global key feature f k (X) The calculation formula for the correlation matrix M obtained by matrix multiplication is:
[0082]
[0083] Among them, τ represents the transpose operation, Represents the transposed global key features.
[0084] The characteristic dimension of the correlation matrix M is [T, T]. In the correlation matrix M, each element value represents the importance of other video clips to this video clip. The larger the value, the more relevant the two video clips are.
[0085] Furthermore, in order to make the video anomaly detection device pay more attention to abnormal features and make the correlation between adjacent video clips higher, in this embodiment, it is particularly proposed to use position coding to enhance the correlation matrix M to obtain the final correlation matrix M f Specifically, position coding PE is introduced into the correlation matrix M to model the relative distance of video clips, thereby improving the correlation between adjacent clips. In this embodiment, the calculation formula for enhancing the correlation matrix M using position coding is:
[0086]
[0087] Among them, G represents the position correlation matrix, i and j represent the sequence number of the video clip, Gi,j represents the degree of position correlation on (i, j) in the position correlation matrix, exp represents the exponential function, γ and β represent the learnable parameters in training, H represents the feature norm difference matrix, and H i,j represents the feature norm difference at (i, j) in the feature norm difference matrix. In this embodiment, the feature norm difference is used to represent the correlation between the two video clips. The larger the feature norm difference is, the less correlated the two video clips are. i,j Represents the negative of the feature norm difference at (i, j) in the feature norm difference matrix, H i represents the feature norm of the i-th video clip, H j represents the feature norm of the jth video clip, C represents the total number of channels of the video clip feature, c represents the number of channels of the video clip feature, and f i,c The eigenvalue representing the c-th channel number of the ith video clip.
[0088] In this embodiment, since the feature norm of the abnormal video clip feature is larger than the feature norm of the normal video clip feature, it is necessary to find the feature norm difference matrix H between adjacent video clips. The larger the feature norm difference matrix H is, the more unrelated the two video clips are, and the smaller the value of the corresponding element of the position correlation matrix G is. In this embodiment, the feature dimension of the position correlation matrix G is [T, T]. Each element in the position correlation matrix G represents the relative distance between each video clip and other video clips, that is, the degree of correlation. The larger the correlation value, the more related the two are. Therefore, in this embodiment, the position correlation matrix G and the correlation matrix M are added element by element to obtain the final correlation matrix M f , the calculation formula is:
[0089]
[0090] The final correlation matrix M f The feature dimension of is [T,T].
[0091] Furthermore, the final correlation matrix M f The global attention weight map A is generated by normalizing the Softmax activation function g , global attention weight map A g The feature dimension of is [T, T]. It should be noted that the normalization operation by the Softmax activation function in this embodiment is a prior art. In this embodiment, the Softmax activation function can make the matrix elements equally transformed along a certain dimension, and transform all element values to between 0-1, thereby indicating the weight of each element in the dimension, that is, the percentage of relevance.
[0092] Furthermore, the global attention weight map Ag With the global value feature f v (X) Perform matrix multiplication to obtain the global temporal weight value of each video segment , the calculation formula is:
[0093]
[0094] Global timing weight value The feature dimension of is [T,D]. Therefore, through the global time series weight value Indicates the weight value of each video clip, the global temporal weight value The larger the value, the more likely the video clip is to be an abnormal video.
[0095] Furthermore, the global timing weight value Add the matrix to the original feature X to get the global time series feature X g , the calculation formula is:
[0096]
[0097] Global timing characteristics X g The feature dimension is [T,D].
[0098] Therefore, in this embodiment, the global time series feature enhancement module performs global time series feature enhancement on the original feature X in the time dimension to obtain the global time series feature X g , making the video anomaly detection device pay more attention to abnormal feature changes.
[0099] For the local temporal feature enhancement module, the local temporal feature enhancement module is composed of a multi-head attention module. In this embodiment, the local temporal feature is generated by the local temporal feature enhancement module, and the steps include: inputting the original feature into the fully connected layer of the local temporal feature enhancement module to obtain local query features, local key features and local value features through feature extraction; setting the attention mask matrix in the multi-head attention module; inputting the local query feature, local key feature and local value feature into the multi-head attention module, and calculating the attention weights of the local query feature, local key feature and local value feature according to the attention mask matrix to obtain the local temporal weight value; performing matrix addition of the local temporal weight value and the original feature to obtain the local temporal feature.
[0100] Specifically, in the local temporal feature enhancement module, the original features X are input into the fully connected layer Q of the local temporal feature enhancement module respectively. b , fully connected layer K b and the fully connected layer V b Feature extraction is performed in the local query feature f qb , local key feature f kband the local value feature f vb It should be noted that due to the fully connected layer Q b , fully connected layer K b and the fully connected layer V b The network structure is the same as that of qb , local key feature f kb and the local value feature f vb The feature dimensions of are also the same, both [T, D]. Furthermore, the local query feature f qb , local key feature f kb and the local value feature f vb Input into the multi-head attention module to enhance local features and obtain local temporal weight values . It should be noted that, in this embodiment, the multi-head attention module is a module in the Transformer, which is essentially a self-attention mechanism and has the ability to extract local temporal features. Therefore, in order to make the multi-head attention module applicable to the field of video anomaly detection, this embodiment sets an attention mask matrix mask. In the attention mask matrix mask, the mask is a built-in parameter in the multi-head attention function, and its function is to allow the multi-head attention module to calculate the attention weights of the video fragment blocks. Specifically, the attention mask matrix mask is set. Since the characteristic dimension of the attention weight map is [T, T], the characteristic dimension of the attention mask matrix mask is set to [T, T], so that the video anomaly detection device can calculate the local attention mechanism for adjacent video fragments according to the design of specific element values. Furthermore, by setting the initial value of each element in the attention mask matrix mask to negative infinity, the multi-head attention module does not calculate the attention weights for the video fragments at the negative infinity position. According to actual needs, in this embodiment, the attention weights are calculated for three adjacent video clips. Therefore, the corresponding positions in the attention mask matrix mask are set to 0, so that the multi-head attention module calculates the attention weights for the corresponding number of video clips, thereby obtaining the local temporal weight value .
[0101] In this embodiment, the local query feature f is quantized by the attention mask qb , local key feature f kb and the local value feature f vb Calculate the attention weight using the following formula:
[0102]
[0103] Among them, mask represents the attention mask matrix, i and j represent the sequence numbers of video clips, T represents the number of video clips, and mask i,jIndicates whether to perform local attention calculation on the i-th video clip and the j-th video clip. In this embodiment, if mask i,j is 0, local attention calculation is performed. If mask i,j If it is negative infinity, no local attention calculation is performed. Attention represents the local attention weight value, m represents the head number of the multi-head attention module, h represents the total number of heads of the multi-head attention module, h is a manually set value, and f qb represents the local query feature, f kb represents the local key feature, f vb represents the local value feature, represents the local query feature of the mth head, represents the local key feature of the mth head, represents the local value feature of the mth head, softmax() represents the softmax activation function, d k Represents the feature dimension, divided by d in the formula k The purpose is to prevent the feature from being too large, Concat means dimension concatenation, and τ means transposition operation. Represents the local temporal weight value, and the feature dimension is [T,D].
[0104] In this embodiment, the local timing weight value Add the matrix to the original feature X to get the local time series feature X l , the calculation formula is:
[0105]
[0106] Therefore, the local temporal feature enhancement module is used to enhance the local temporal feature of the original feature X in the time dimension to obtain the local temporal feature X l , making the video anomaly detection device pay more attention to abnormal feature changes.
[0107] Finally, the global time series feature X g and local temporal features X l Add together to get the first time series feature X', the calculation formula is:
[0108]
[0109] The feature dimension of the first time series feature X' is [T, D].
[0110] like Figure 3As shown, for the feature sampling module, the feature sampling module includes a conditional variational autoencoder (Condition VAE, CVAE) and a contrastive language-image pre-training (Contrastive Language-Image Pre-training, CLIP) model. It should be noted that in this embodiment, the contrastive language-image pre-training model is an existing pre-training model based on contrastive text-image pairs, and corresponding semantic information features can be generated by passing semantic information into it. In this embodiment, the probability distribution of each abnormal event and normal event in the video clip can be obtained through the feature sampling module, and more accurate abnormal features and normal features can be generated through the feature sampling module, so as to serve as a teacher model to guide the student model for training and improve the detection performance.
[0111] In this embodiment, the feature sampling module performs feature reconstruction on the first temporal feature X' to obtain a generalized feature, and the steps include: setting a semantic information label label, concatenating the first temporal feature X' and the semantic information label label to obtain a pre-reconstructed feature f input ; The pre-reconstructed feature f input Input into the encoder of the conditional variational autoencoder to get the mean μ and variance σ 2 . Further, ε is randomly sampled from the normal distribution N(0,1) and compared with the mean μ and variance σ 2 Combine the generated intermediate latent variable z and concatenate the intermediate latent variable z with the semantic information label label to obtain the intermediate feature f mid , the intermediate feature f mid Input into the decoder of the conditional variational autoencoder to obtain the generalized feature f out At the same time, the semantic features f are generated by comparing the language-image pre-training model token , so that through the semantic feature f token Constrained generalization feature f out .
[0112] Specifically, in the abnormal video to be detected, the first temporal feature X' obtained from the first feature enhancement module is top-k selected. For the first temporal feature X' of the abnormal video, the three video clip features with the largest feature norm in the channel dimension are selected and averaged to obtain the abnormal temporal feature f that is most likely to be the abnormal feature. abn , abnormal time series characteristics f abn The feature dimension is [1, D]; in the normal video of the video to be detected, the first temporal feature X' obtained from the first feature enhancement module is selected, and a video clip feature with the largest feature norm in the channel dimension is obtained to obtain the normal temporal feature f nor , normal timing characteristics f norThe feature dimension of is [1, D]. Therefore, in this embodiment, the first time series feature X' includes the abnormal time series feature f abn and normal timing characteristics f nor , let the first time series feature X' be f in , whose feature dimension is [1,D].
[0113] Furthermore, we set the semantic information label label and transform the first temporal feature f in Concatenate the semantic information label label in the channel dimension to obtain the pre-reconstructed feature f input , pre-reconstruction feature f input The feature dimension of is [1, D+n], where n represents the vector dimension of the semantic information label label. The vector size of the semantic information label label depends on the number of abnormal events and normal events. In this way, the features with corresponding label information of the input conditional variational autoencoder can be obtained.
[0114] In this embodiment, the conditional variational autoencoder is composed of an encoder and a decoder, and each of the encoder and the decoder is composed of three fully connected layers. The three fully connected layers of the encoder are: c1 、f c2 and f c3 , the three fully connected layers of the decoder are: c4 、f c5 and f c6 In this embodiment, the fully connected layer plays a role in dimensionality reduction and dimensionality increase. It should be noted that the third fully connected layer f c3 There are two fully connected layers with the same network structure, one for generating the mean μ and the other for generating the variance σ 2 In the encoder, the pre-reconstructed features f input Through the first fully connected layer f c1 The feature dimension is [1, ], through the second fully connected layer f c2 The feature dimension is [1, ], after the third fully connected layer f c3 Then we get the output variable mean μ and variance σ of the encoder 2 , whose feature dimensions are [1, ], the mean μ and variance σ 2 Represents the mean and variance of features with label information.
[0115] It should be noted that since sampling is not reversible and cannot be back-propagated, the conditional variational autoencoder introduces a reparameterization operation to make the sampling process reversible, that is, it does not directly start from the mean μ and variance σ 2Instead of sampling from the standard normal distribution N(0,1), ε is sampled from the standard normal distribution N(0,1) to indirectly achieve random sampling, that is, the intermediate latent variable z of the conditional variational autoencoder is obtained, and its feature dimension is [1, ], the calculation formula is:
[0116]
[0117] Therefore, the sampling operation does not need to participate in gradient descent, so the video anomaly detection device can be trained.
[0118] Furthermore, if Figure 3 As shown, the intermediate latent variable z is concatenated with the semantic information label label in the channel dimension to obtain the intermediate feature f mid , whose feature dimension is [1, +n], thus, the generalized features f corresponding to the label information are generated through the three fully connected layers of the decoder of the conditional variational autoencoder out Specifically, the intermediate feature f mid After the first fully connected layer f of the decoder c4 After dimensionality increase, its feature dimension is [1, ], and then passes through the second fully connected layer f of the decoder c5 After dimensionality increase, its feature dimension is [1, ], and finally passes through the third fully connected layer f of the decoder c6 Perform dimension increase to obtain the final generalized feature f with label information out , whose feature dimension is [1,D].
[0119] At the same time, in order to make the generalization feature f out Generate features that more accurately represent label information, and use the pre-trained contrastive language-image pre-training model to generate semantic features f corresponding to the label token , whose feature dimension is [1,D], making it consistent with the generalized feature f out Perform feature alignment, which is achieved through the constraints of the loss function. At the same time, the semantic feature f is constrained token With the first timing characteristic f in Perform feature alignment to obtain more accurate generalized features with label information.
[0120] like Figure 5 As shown in the figure, after the feature sampling module is trained, the encoder is discarded, and the intermediate latent variable z is randomly sampled from the standard normal distribution, and it is concatenated with the obtained feature label in the channel dimension and sent to the trained decoder to obtain the corresponding generalized feature with label information. The generalized feature is sent to the second-stage training, which can guide the second-stage video anomaly detection model to learn modeling with semantic label information, thereby improving the performance of anomaly detection.
[0121] In this embodiment, the video anomaly detection model includes a feature extraction module, a second feature enhancement module and an anomaly scoring module.
[0122] Regarding the feature extraction module, it should be noted that, in this embodiment, the feature extraction module in the video anomaly detection model and the feature extraction module in the feature generation model have the same structure and the same parameters.
[0123] As for the second feature enhancement module, it should be noted that, in this embodiment, the second feature enhancement module in the video anomaly detection model and the first feature enhancement module in the feature generation model have the same structure but different parameters. Furthermore, by setting different parameters, it is possible to avoid overfitting of the original features of the second feature enhancement module, and effectively reduce the impact on the detection performance.
[0124] Thus, the feature extraction module extracts the original feature X of the video clip in the video to be detected, and the second feature enhancement module enhances the original feature X to obtain the second temporal feature f t .
[0125] like Figure 4 As shown, for the anomaly scoring module, the anomaly scoring module consists of two fully connected layers. The second temporal feature f t is the input feature of the anomaly score module, whose feature dimension is [T, D]. The second time series feature f t The data is input into two fully connected layers, and finally an anomaly score is output, whose feature dimension is [T, 1], that is, the anomaly score corresponding to each video clip. Then, a top-k selection is performed, and the largest anomaly scores of the first three video clips are selected. The average value is taken to get the final anomaly score s.
[0126] Therefore, the video anomaly detection device provided by this embodiment, because the abnormal segments in the video to be detected have the characteristics of sparseness and continuity, by constructing a feature enhancement module, improving the position encoding information in the self-attention mechanism, so that the video anomaly detection device can pay more attention to the relationship between adjacent video segments, thereby effectively improving the efficiency of video anomaly detection. In addition, by constructing a feature sampling module, based on the conditional variational autoencoder, random sampling from the normal distribution can be obtained to obtain more generalized abnormal features, achieve more accurate modeling of abnormal features, and effectively improve the accuracy of video anomaly detection.
[0127] Example 2
[0128] This embodiment also provides a video anomaly detection method, comprising the following steps:
[0129] Phase 1: Obtain the video to be detected, and the feature extraction module extracts the original features of the video clips in the video to be detected; the first feature enhancement module enhances the original features to obtain the first time series features; the feature sampling module reconstructs the first time series features to obtain generalized features.
[0130] The second stage: obtain the video to be detected, and the feature extraction module extracts the original features of the video clip in the video to be detected; the second feature enhancement module enhances the original features to obtain the second time series features; the generalized features obtained in the first stage are aligned with the second time series features, and the second time series features are constrained by the generalized features; the anomaly score module calculates the second time series features to obtain the anomaly score, and determines whether the video clip is abnormal based on the anomaly score.
[0131] It should be noted that, in the first stage, since the weakly supervised video anomaly detection only provides the video level label "0" or "1", in order to adapt to the training of the video anomaly detection device of this embodiment, in this embodiment, Figure 6 As shown in Figure 1, according to the abnormal events in each video of the dataset, the semantic feature information, such as "fighting", "cycling", etc., is used to feed it into the comparative language-image pre-training model to obtain the semantic feature f corresponding to the label of the semantic information. token , and combine it with the generalized feature f obtained by the first feature enhancement module and the feature sampling module out Perform feature alignment and make the semantic feature f token and the first time series feature f obtained by the first feature enhancement module in Therefore, the loss function of the first stage is based on triplet loss, and the loss function of generalization features constrained by semantic features is expressed as:
[0132]
[0133] Where L1 represents the one-stage alignment loss function, max represents the maximum value function. In this embodiment, the maximum value between the two functions is taken, d represents the feature distance, and f token represents the semantic feature, f in represents the first time series feature, f out represents the generalization feature, d(f token ,f in ) represents the distance between the semantic feature and the first temporal feature, d(f token ,f out ) represents the distance between the semantic feature and the generalized feature, f1 represents the representative feature, f2 represents another representative feature, d(f1,f2) represents the distance between the two representative features, C represents the total number of feature channels, c represents the current number of channels, It represents the eigenvalue of the feature on the c channel. represents the eigenvalue of another feature on the c channel. represents the two-norm.
[0134] Therefore, the generalized feature f out , the enhanced first time series feature f in With the semantic feature f token Alignment, so that the feature sampling module can use the label information to sample more accurate generalized features f out , and then guide the video anomaly detection device to improve the detection performance.
[0135] In order to sample the intermediate latent variable z directly from the normal distribution after training, it is necessary to set the mean μ and variance σ 2 Constrain it to make it tend to normal distribution. Figure 7 As shown, the loss function of the first stage is the KL divergence loss function, and the loss function is expressed as:
[0136]
[0137] Among them, L2 represents the one-stage alignment loss function.
[0138] The KL divergence loss function makes the mean μ and variance σ 2 The distribution of is close to normal distribution, with the mean μ close to 0 and the variance σ 2 When it is close to 1, the encoder of the conditional variational autoencoder can generate the mean and variance of the normal distribution characteristics with label information features, so that when sampling, it can be sampled directly from the normal distribution, which increases the generalization of the features.
[0139] Therefore, the final loss function of the first stage is expressed as:
[0140]
[0141] Among them, L 一阶段 Represents the final loss function of one stage.
[0142] In the second stage, since only the video level is labeled in the field of weakly supervised video anomaly detection, it is uncertain whether the video clip in the video to be detected is a normal video clip or an abnormal video clip. Therefore, the commonly used binary cross entropy loss constrained student model is used to calculate the loss function of the anomaly score s output by the anomaly score module and the video level label. The calculation formula is:
[0143]
[0144] Among them, L bce represents the binary cross entropy loss, i represents a natural number, N represents the number of videos during training, and yi represents the label of each video. In this embodiment, “0” represents a normal video clip and “1” represents an abnormal video clip. i Represents the predicted anomaly score value.
[0145] At the same time, in the second stage, in order to make the generalized features f generated in the first stage out The second temporal feature f that can guide the two-stage generation t , using the loss function to constrain, by generalizing the feature f out Constrain the second timing characteristic f t The loss function is expressed as:
[0146]
[0147] Among them, L align represents the loss function of the second temporal feature constrained by the generalized feature, C represents the total number of channels of the video clip feature, c represents the number of channels of the video clip feature, T represents the total number of video clips, i represents the i-th video clip, and f out represents the generalization feature, represents the feature value of the generalized feature of the i-th video clip on the c-th channel, represents the two-norm, represents the second timing characteristic, Represents the feature value of the second temporal feature of the i-th video clip on the c-th channel.
[0148] Therefore, the final loss function formula for the second stage is as follows:
[0149]
[0150] Among them, L 二阶段 Represents the final loss function of the second stage.
[0151] It should be noted that after the two-stage training is completed, during detection, it is only necessary to use the two-stage video anomaly detection model to detect the video to be detected, and the anomaly score s of each video clip of the video to be detected can be obtained.
[0152] In this embodiment, a threshold λ is set artificially. During detection, when the anomaly score s output by the video anomaly detection model of a video clip in the video to be detected is greater than or equal to the threshold λ, the video clip is an abnormal video clip; when the anomaly score s output by the video anomaly detection model of a video clip in the video to be detected is less than the threshold λ, the video clip is a normal video clip.
[0153] Therefore, the video anomaly detection method provided in this embodiment can significantly improve the anomaly detection performance by modeling the abnormal features in a more generalized manner; at the same time, it effectively solves the problem of poor supervision accuracy of the video anomaly detection algorithm, and has strong practicality and wide applicability.
[0154] Example 3
[0155] This embodiment also provides a computer device. A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video anomaly detection method when executing the computer program.
[0156] Example 4
[0157] This embodiment also provides a computer-readable storage medium. A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a video anomaly detection method described in this embodiment is executed. The computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device; the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0158] The above schematically describes the invention and its implementation methods, which is not restrictive. Without departing from the spirit or basic features of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited thereto, and any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and an embodiment similar to the technical solution are designed without creativity, which should all belong to the protection scope of the present invention. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate the name, and do not indicate any specific order.
Claims
1. A video anomaly detection device, characterized in that: include: Feature generation model and video anomaly detection model; The feature generation model includes a feature extraction module, a first feature enhancement module and a feature sampling module; The feature extraction module extracts the original features of the video clip in the video to be detected; the original features are input into the first feature enhancement module for feature enhancement to obtain the first temporal features; the features The sampling module reconstructs the first time series feature to obtain a generalized feature; The video anomaly detection model includes a feature extraction module, a second feature enhancement module and an anomaly scoring module; the second feature enhancement module in the video anomaly detection model and the first feature enhancement module in the feature generation model have the same structure but different parameters; The feature extraction module extracts the original features of the video clip in the video to be detected; the original features are input into the second feature enhancement module for feature enhancement to obtain the second time series features; the generalized features obtained by the feature generation model are aligned with the second time series features, and the second time series features are constrained by the generalized features; The anomaly score module calculates the constrained second temporal feature to obtain an anomaly score, and determines whether the video clip is abnormal based on the anomaly score.
2. A video anomaly detection device according to claim 1, characterized in that: The first feature enhancement module includes a global temporal feature enhancement module and a local temporal feature enhancement module; The global timing feature enhancement module generates a global timing feature; The local time series feature enhancement module generates local time series features; The global timing feature and the local timing feature are added together to obtain a first timing feature.
3. A video anomaly detection device according to claim 2, characterized in that: The global timing feature enhancement module generates global timing features, and the steps include: The original features are input into the fully connected layer of the global temporal feature enhancement module to extract the global query features, global key features and global value features; Perform matrix multiplication of the global query feature and the transposed global key feature to obtain a relevance matrix; The correlation matrix is enhanced by using position coding to obtain the final correlation matrix; Normalize the final correlation matrix to generate a global attention weight map; Perform matrix multiplication of the global value feature and the global attention weight map to obtain the global temporal weight value; The global time series weight value and the original feature are matrix-added to obtain the global time series feature.
4. A video anomaly detection device according to claim 3, characterized in that: The calculation formula of the correlation matrix enhanced by position coding is: Among them, G represents the position correlation matrix, i and j represent the sequence number of the video clip, G i,j represents the degree of position correlation on (i, j) in the position correlation matrix, exp represents the exponential function, γ and β represent the learnable parameters in training, H represents the feature norm difference matrix, and H i,j Represents the feature norm difference at (i, j) in the feature norm difference matrix, -H i,j Represents the negative of the feature norm difference at (i, j) in the feature norm difference matrix, H i represents the feature norm of the i-th video clip, H j represents the feature norm of the jth video clip, C represents the total number of channels of the video clip feature, c represents the number of channels of the video clip feature, and f i,c The eigenvalue representing the c-th channel number of the ith video clip.
5. A video anomaly detection device according to claim 4, characterized in that: The local temporal feature enhancement module includes a multi-head attention module; the local temporal feature enhancement module generates local temporal features, and the steps include: The original features are input into the fully connected layer of the local temporal feature enhancement module, and local query features, local key features and local value features are obtained through feature extraction; Set the attention mask matrix in the multi-head attention module; Input local query features, local key features and local value features into the multi-head attention module, calculate the attention weights of local query features, local key features and local value features according to the attention mask matrix, and obtain the local temporal weight values; The local time series weight value and the original feature are matrix-added to obtain the local time series feature.
6. A video anomaly detection device according to claim 5, characterized in that: According to the attention mask matrix, the attention weights of the local query features, local key features and local value features are calculated using the following formula: Among them, mask represents the attention mask matrix, i and j represent the sequence number of the video clips, T represents the number of video clips, and mask i,j Indicates whether to perform local attention calculation on the i-th video clip and the j-th video clip, Attention indicates the local attention weight value, m indicates the head number of the multi-head attention module, h indicates the total number of heads of the multi-head attention module, and f qb represents the local query feature, f kb represents the local key feature, f vb represents the local value feature, represents the local query feature of the mth head, represents the local key feature of the mth head, represents the local value feature of the mth head, softmax() represents the softmax activation function, d k represents the feature dimension, τ represents the transposition operation, and Concat represents the dimension concatenation. Represents the local timing weight value.
7. A video anomaly detection device according to claim 6, characterized in that: The feature sampling module includes a conditional variational autoencoder and a contrastive language-image pre-training model; The feature sampling module performs feature reconstruction on the first time series feature to obtain a generalized feature, and the steps include: Setting a semantic information label, concatenating the first temporal feature and the semantic information label to obtain a pre-reconstruction feature; The pre-reconstructed features are input into the encoder of the conditional variational autoencoder to obtain the mean and variance; Random sampling from a normal distribution and combining with the mean and variance to generate an intermediate latent variable; Concatenate the intermediate latent variables with the semantic information labels to obtain the intermediate features; Input the intermediate features into the decoder of the conditional variational autoencoder to obtain generalized features; By comparing the language-image pre-training model, semantic features are generated, and generalization features are constrained by semantic features.
8. A video anomaly detection device according to claim 7, characterized in that: The loss function of generalizing features constrained by semantic features is expressed as: Among them, L1 represents the one-stage alignment loss function, max represents the maximum value function, d represents the feature distance, and f token represents semantic features, f in represents the first time series feature, f out represents the generalization feature, d(f token ,f in ) represents the distance between the semantic feature and the first temporal feature, d(f token ,f out ) represents the distance between the semantic feature and the generalized feature, f1 represents the representative feature, f2 represents another representative feature, d(f1,f2) represents the distance between the two representative features, C represents the total number of channels of the video clip feature, c represents the number of channels of the video clip feature, It represents the eigenvalue of the feature on the c channel. represents the eigenvalue of another feature on the c channel. represents the two-norm.
9. A video anomaly detection method, characterized in that: Using a video anomaly detection device according to any one of claims 1 to 8, comprising the following steps: Phase 1: Obtain the video to be detected, and the feature extraction module extracts the original features of the video clip in the video to be detected; The first feature enhancement module performs feature enhancement on the original feature to obtain a first time series feature; The feature sampling module reconstructs the first time series feature to obtain a generalized feature; Phase 2: Obtain the video to be detected, and the feature extraction module extracts the original features of the video clip in the video to be detected; The second feature enhancement module performs feature enhancement on the original feature to obtain a second time series feature; Perform feature alignment operation on the generalized features obtained in the first stage and the second time series features, and constrain the second time series features through the generalized features; The anomaly score module calculates the constrained second temporal feature to obtain an anomaly score, and determines whether the video clip is abnormal based on the anomaly score.
10. A video anomaly detection method according to claim 9, characterized in that: The loss function of constraining the second temporal feature by the generalized feature obtained in the first stage is expressed as: Among them, L align represents the loss function of the second temporal feature constrained by the generalized feature, C represents the total number of channels of the video clip feature, c represents the number of channels of the video clip feature, T represents the total number of video clips, i represents the i-th video clip, and f out represents the generalization feature, represents the feature value of the generalized feature of the i-th video clip on the c-th channel, represents the two-norm, represents the second timing characteristic, Represents the feature value of the second temporal feature of the i-th video clip on the c-th channel.
