Video anomaly detection method based on causal normality learning

Abstract

The invention discloses a video anomaly detection method based on causal normality learning. The video anomaly detection method comprises the following steps: acquiring a monitoring video of multiple scenes and cutting the monitoring video into b frames of continuous fragments with a uniform size as training samples; a CRCL model is constructed, the CRCL model comprises an M1 scene encoder, an M3 motion perception feature extractor, a scene depolarization learning module, a causality heuristic normality learning module and an M5 clustering module, the training samples are input into the CRCL model for training, a correlation matrix and loss are calculated, and model parameters are optimized through gradient back propagation; after a preset round is reached, calculating a clustering center of causal representation through an M5 clustering module, and minimizing clustering loss to obtain an optimal cluster; and the rounds of training are repeatedly carried out until the drop-out value of the loss function between two adjacent rounds of iteration in the training meets a set threshold value or reaches the maximum number of iterations, and the trained CRCL model is output. According to the invention, the video anomaly detection performance in a multi-scene environment can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a video anomaly detection method based on causal normality learning. Background Art

[0002] Video Anomaly Detection (VAD) aims to use deep learning technology to automatically analyze the spatiotemporal patterns of surveillance videos to detect any abnormal events that may cause personal injury or economic loss, such as traffic accidents, illegal operations and violent behaviors. It has broad application prospects in the fields of public safety protection and information forensics. Due to the diversity of abnormal instances in the real world and their low frequency compared to normal events, it is unrealistic to collect a sufficient number of all kinds of anomalies for modeling. Therefore, existing methods usually regard VAD as a semi-supervised out-of-distribution detection task. Such methods do not need to collect and annotate abnormal samples. They only use easy-to-collect regular events (negative samples, labeled 0) to train deep neural networks to characterize the prototype spatiotemporal patterns of normal videos, that is, normality. In the test phase, the anomaly score is quantitatively calculated by measuring the deviation between the input sample and the learned normality. However, such deep learning methods aim to establish a statistical association between regular events and negative labels, and lack the exploration of the causal relationship of normality learning. As a result, the trained deep neural network model may have the ability to characterize abnormal samples due to its strong generalization performance, resulting in missed detection.

[0003] Although memory networks have been introduced into VAD in recent years to weaken the deep model's ability to represent unseen anomalies, the improved model can only process surveillance videos of a single scene and cannot cope with label-independent data bias in diverse normal events, resulting in a high false alarm rate when facing multi-scene data. Researchers have proposed using object detection or instance segmentation models to separate noteworthy foreground objects from background scenes and analyze their appearance and motion patterns separately to learn normality, but the additional data processing work significantly increases the computational and storage costs of the model. In addition, the target-level method requires all foreground objects to be defined in advance, which is unrealistic for complex scenes involving multiple subjects or unknown new scenes. In contrast, causal learning attempts to explore the potential causal relationship between data domain labels, and has been proven to be more effective than statistical associations established by deep learning in tasks such as domain adaptation. Although some studies have introduced causality into the VAD task, it still does not take into account the scene bias that is prevalent in the real world. It only performs well on a single scene dataset and suffers from significant performance degradation when facing multiple scenes or new scenes. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a video anomaly detection method based on causal normality learning, which uses causal representation learning to improve the detection performance of video anomaly detection in realistic settings such as multi-scenario environments, so as to solve the problems of high false alarms caused by label-irrelevant data bias such as scene bias in deep representation learning methods, significantly improve the deployability of unsupervised VAD methods in realistic applications, and provide a new solution for multi-scenario VAD.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] A video anomaly detection method based on causal normality learning comprises the following steps:

[0007] Obtain surveillance videos of multiple scenes and crop them into b-frame continuous segments of uniform size as training samples;

[0008] A CRCL model is constructed, wherein the CRCL model includes an M1 scene encoder, an M3 motion perception feature extractor, a scene debiasing learning module, a causal-inspired normality learning module and an M5 clustering module, wherein the scene debiasing learning module includes an M2-1 scene classifier and an M2-2 scene classifier, wherein the causal-inspired normality learning module includes an M4-1 memory network, an M4-2 prototype decomposer and an M4-3 causal-inspired characterizer, wherein training samples are input into the CRCL model for training, a correlation matrix and a loss function are calculated and model parameters are optimized using gradient back propagation; after reaching a preset round, the cluster center of the causal characterization is calculated by the M5 clustering module and the clustering loss is minimized to obtain the best cluster; rounds of training are repeatedly performed until the decrease value of the loss function between two adjacent rounds of iterations in the training meets a set threshold or the maximum number of iterations is reached, and the training is stopped, and the trained CRCL model is output;

[0009] The video to be detected is input into the M3 motion perception feature extractor, causality-inspired normality learning module and M5 clustering module in the trained CRCL model, and the deviation from the learned causal representation consistency is calculated and the anomaly score in the range of [0, 1] is calculated. If the anomaly score is greater than the set threshold, the video to be detected is an abnormal event.

[0010] Preferably, the method further comprises the following steps: after determining that an abnormal event occurs, a voice alarm mechanism is activated to alert management personnel and the corresponding segments are automatically captured and stored.

[0011] Preferably, the processing process of the CRCL model specifically includes the following steps:

[0012] The M1 scene encoder and M3 motion-aware feature extractor take the same batch of b video clips as input and output scene features F sce and entanglement representation F ent;

[0013] M4-1 Memory Network with F ent As input, the memory entry is updated and the prototype feature F' is output. The M4-2 prototype decomposer takes F ent and F' as input to calculate the private feature F p and shared features F s , the private feature F p and shared features F s It is simultaneously input to the M4-3 causality-inspired characterizer to calculate the corresponding causal representation and correlation matrix. The M5 clustering module uses the causal representation as input to calculate the cluster center.

[0014] M2-1 scene classifier is based on scene feature F sce As input to perform scene classification, the M2-2 scene classifier uses F ent is used as input to perform scene classification; scene features F sce Input causality-inspired normality learning module and determine the entanglement representation F ent The resulting correlation matrices are then used to simulate the TDE process to remove the scenario bias.

[0015] Preferably, the M4-1 memory network is based on F ent As input, the memory entry is updated and the prototype feature F' is output. The M4-2 prototype decomposer takes F ent and F' as input to calculate the private feature F p and shared features F s , the private feature F p and shared features F s It is simultaneously input to the M4-3 causality-inspired characterizer to calculate the corresponding causal characterization and correlation matrix, which specifically includes the following steps:

[0016] The M4-2 prototype decomposer includes two pooling operations and two multi-layer perceptrons with the same structure. The two multi-layer perceptrons with the same structure are composed of a 3-layer fully connected network with sigmoid as the activation function of the last layer; the entanglement representation F ent And prototype feature F' are averaged and globally pooled along the channel dimension to obtain f avg ,f' avg ,f max ,f' max , and then input it into the multi-layer perceptron to calculate the difference coefficients α and β:

[0017] α=MLP(f avg -f′ avg ;θ1),β=MLP(f max -f′ max ;θ2)

[0018] Where MLP represents a multi-layer perceptron used to calculate the difference coefficients α and β, and its corresponding learnable parameters are denoted as θ1 and θ2 respectively;

[0019] Perform multiplication along the channel dimension to calculate the private feature F p and shared features F s :

[0020]

[0021] The M4-3 causality-inspired representation is composed of a deep neural network, with F p and F s As input, the features of the two pointing to the same regular event and thus having the same semantic label are used to calculate the causal representation. The process is as follows:

[0022]

[0023] Where R and Respectively represent F s and F p The causal representation of , the dimension is b × n, and the causal loss is used to optimize the M4-3 causality-inspired representation, the formula is as follows:

[0024]

[0025] Where C1, C2, and C3 represent The correlation matrix of the corresponding column vector of R, that is, C1(i,j) represents the correlation between the i-th column of R and The dimension of the correlation matrix is ​​n×n, I is the unit matrix, and λ is the balancing hyperparameter.

[0026] Preferably, the M2-1 scene classifier is based on the scene feature F sce As input to perform scene classification, the M2-2 scene classifier uses F ent is used as input to perform scene classification; scene features F sce Input causality-inspired normality learning module and determine the entanglement representation F ent The difference in the resulting correlation matrix is ​​used to simulate the TDE process to remove the scene bias, as follows:

[0027] After receiving the scene features from the same batch of b video clips, the M2-1 scene classifier is optimized with the following cross entropy loss function under the supervision of the scene labels:

[0028]

[0029] where y ijIndicates whether sample i belongs to category j, where 0 means it belongs and 1 means it does not belong, p ij represents the probability that sample i belongs to category j predicted by the M2-1 scene classifier, N s Represents the total number of scenes contained in the dataset; M2-2 scene classifier uses entanglement representation F ent As input, mutual learning is performed through classification tasks to perceive F ent Scene bias in:

[0030]

[0031] in and represent the scene classification probabilities of the M2-1 scene classifier and the M2-2 scene classifier respectively;

[0032] Eliminate F by simulating the TDE process ent The bias in F sce It is also input into CiNL to calculate the correlation matrix, recorded as {C'1, C'2, C'3}. The optimization equation for scene debiasing learning is as follows:

[0033] min(d({C1, C2, C3}, I)-d({C′1, C′2, C′3}, I)+ε, 0)

[0034] Where d() represents the distance metric function and ε is the margin parameter that controls the minimum distance.

[0035] Preferably, the anomaly score s t , calculation formula:

[0036]

[0037] where D represents the distance between the causal representation of the input sample and the nearest cluster center, and g() represents the maximum and minimum normalization performed on the deviations of all frames of the same video.

[0038] Based on the above technical solution, the beneficial effects of the present invention are:

[0039] 1) The present invention uses causal representation learning to mine the potential causality of normality learning, and improves the detection performance of the model on single-scene real surveillance videos: The present invention uses a motion-aware feature extractor to obtain the spatiotemporal features of surveillance videos and uses a causal-inspired normality learning module to mine causal normality. Compared with deep representation learning methods based on reconstruction or prediction, it can eliminate the influence of non-causal variables and achieve a better balance between efficiently describing routine events and limited generalization of abnormal instances. Tests on the single-scene datasets UCSD Ped2 and CUHKAvenue show that the proposed causal representation consistency learning achieved frame-level AUCs of 99.1% and 92.9%, respectively, which are better than the current best deep learning methods, such as STM-AE (98.1%, 89.8), HN-MUM (97.1%, 88.2%), and MAAM-Net (97.7%, 90.9%);

[0040] 2) The present invention proposes scene debiasing learning to eliminate scene bias and learn normal endogenous features, significantly improving the detection performance on multi-scene datasets: The present invention constructs a structural causal model for scene-robust VAD and proposes scene debiasing learning to eliminate the scene bias entangled in the spatiotemporal representation obtained by the deep encoder. Scene debiasing learning provides more discriminative normal endogenous features for the causal-inspired normality learning module for subsequent modeling learning, significantly improving the deployment potential of CRCL in real-world environments. Quantitative experimental results on the multi-scene dataset Shanghaitech (containing 13 scenes) show that the frame-level AUC of the proposed CRCL is 81.7%, leading the existing deep representation learning methods by at least 6%. In addition, CRCL achieved a frame-level AUC of 77.3% on a hybrid dataset based on UCSD Ped2h and CUHKAvenue, while the current most advanced deep VAD model (STM-AE) is only 68.3%, demonstrating the significant advantages of the proposed CRCL in processing videos with scene bias;

[0041] 3) The present invention uses causal consistency rather than reconstruction or prediction errors to detect video anomalies, which not only improves the model's ability to detect minor anomalies but also significantly reduces the model's dependence on a large number of normal videos for training: Deep learning methods usually require a large number of normal videos to train generative models to characterize the spatiotemporal patterns of regular events, and treat samples that cannot be effectively characterized (i.e., cause large reconstruction and prediction errors) as abnormalities during the test phase. However, the premise of its establishment is that the model can learn the prototype pattern of any regular event, so a sufficient number of various possible normal videos are usually required for model training. The CRCL proposed in the present invention uses the label consistency and causal principle of features to learn video normality. Experiments show that the proposed method can still learn a high-performance detection model when only limited normal videos are available. Case studies on the CUHKAvenue dataset show that the frame-level AUC of CRCL when only 80% and 50% of the training data are available are 92.1% and 90.1%, respectively, with a decrease of 0.8% and 2.8%, respectively, which is better than the 1.6% and 10.6% decrease of STM-AE. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of a structural causal model in an embodiment, wherein FIG. (a) is a structural causal model for analyzing an unsupervised video anomaly detection task from a causal perspective; FIG. (b) is a structural causal model for eliminating scene bias;

[0043] Figure 2 The present invention is a structural framework diagram of a causal representation consistency learning model in a video anomaly detection method based on causal normality learning in an embodiment. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0045] This embodiment uses causal representation learning technology to improve the robustness and detection performance of video anomaly detection tasks in realistic settings such as multiple scenes, and provides a video anomaly detection method based on causal normality learning. The method applies a causal representation consistency learning (CRCL) model, which includes an M1 scene encoder, an M3 motion perception feature extractor, a scene debiasing learning (SdL) module, a causal-inspired normality learning (CiNL) module, and an M5 clustering module. Its scene debiasing learning module includes an M2-1 scene classifier and an M2-2 scene classifier, and its causal-inspired normality learning module includes an M4-1 memory network, an M4-2 prototype decomposer, and an M4-3 causal-inspired characterizer (CiC). The CRCL model can simultaneously achieve efficient characterization of the diverse spatiotemporal patterns of routine events and limited generalization of various types of minor anomalies.

[0046] First, a structural causal model of unsupervised VAD for analyzing causal perspectives is constructed. It is pointed out that the spatiotemporal representations obtained by existing deep learning models contain non-causal factors, so the statistical associations between regular events and negative labels established by them are false and fragile, and cannot resist the label-independent data bias that is prevalent in regular events. In response to this, a causal-inspired normality learning module is proposed. Specifically, CiNL uses a memory network and a prototype decomposer to separate the common semantics and individual semantics of regular events, denoted as shared features F s and private feature F p , and then guide the causal-inspired representation using Fs and F according to the causal principle p A set of low-dimensional and orthogonal causal variables are learned by using features pointing to the same negative label. In the training phase, only normal videos are used to optimize the parameters of each module in CiNL. In the test phase, abnormal events that have never been seen will not have the above orthogonality, that is, they will not have causal representation consistency. In addition, in response to the scene bias that is prevalent in multi-scene videos, another structural causal model is constructed to analyze how scene bias affects normality learning by affecting entangled representation. In this regard, scene-debiasing learning (SdL) is proposed, which uses causal inference to collaborate with CiNL to eliminate scene bias in the entangled representation obtained by the deep encoder. The detailed technical solution is as follows:

[0047] 1. Structural Causal Model

[0048] Two structural causal models were constructed, such as Figure 1 (a) and Figure 1 (b) is used to describe the unsupervised video anomaly detection task from a causal perspective and analyze the negative impact of scene bias, where X, Y, U, Z, and F are sce ,F ent ,F n They represent normal videos, negative labels, non-causal variables, causal variables, scene biases (scene features), entanglement representations, and endogenous features of normality, respectively.

[0049] for Figure 1 In the structural causal model in (a), existing deep learning methods attempt to establish a statistical association between X and Y (as shown by the dotted one-way arrow in the figure), but ignore the impact of the non-causal variable U on the process, making the learned model unable to cope with the data bias and personalized semantic features that are prevalent in diverse routine events. According to the principle of common causality, it is pointed out that there is a causal variable Z that has a causal effect on X and Y that have a statistical association (such as Figure 1 (a) shows the solid one-way arrow, and can explain all the correlations between the two when Z is used as a condition. Therefore, the key to the proposed video anomaly detection task is to mine the causal variable Z in normal learning. For video data, Z is not directly observable. However, according to the causal independence mechanism, a set of causal elements {z1,…,z n} are independent of each other, do not know each other and do not affect each other, that is

[0050]

[0051] In the formula, the joint probability distribution P(z1,…,z n ) represents all random variables z1,…,z n The joint probability distribution of i |PA i ) means that at a given z i The parent node PA i In the case of i In other words, z i The value of depends on the value of its parent node; the product symbol It means that for each (from 1 to n), the corresponding conditional probability must be calculated, and then all these conditional probabilities are multiplied together.

[0052] We exploit the orthogonal independence of causal elements to construct a causal loss to guide a causality-inspired characterizer (CiC) to mine {z1,…,z nThe sparse shift mechanism hypothesis states that non-essential distribution changes of observable data are presented in a sparse or local form in the learned causal consistency without affecting all elements, which provides a basis for learning {z1,…,z n It also provides a theoretical basis for detecting video anomalies using representation consistency, namely, diverse regular events all have this consistency, while the causal representations of abnormal events with different semantic labels will show significant changes.

[0053] In addition, the existing video anomaly detection models based on deep representation learning use deep convolutional networks to directly obtain spatiotemporal features from the original video sequence, while ignoring the possible interference of background scene information, so that the features input to the normality model are usually entangled with scene compilation, which limits the model's learning ability for normality and abnormal event detection performance. Figure 1 (b) shows the structural causal model to analyze how scene bias affects normal learning. The ideal feature extraction and normal learning process is F n →F ent →Y, while existing methods are usually negatively affected by scene bias in the first step, that is, the normal endogenous features and scene bias are entangled with each other, affecting F ent The discriminative ability of the model is used to limit the model's ability to express diverse regular events with label-independent data bias, that is, (F sce ,F n ) → F ent →Y. According to causal inference, F can be truncated by backdoor operation do() sce The impact on normal learning is as follows:

[0054]

[0055] Where P(C=F sce ) represents the negative impact of scene bias, and traditional backdoor operation methods must clearly identify this impact, which is impractical for unsupervised VAD tasks where only negative samples can be used for model training. In this regard, the total direct effect (TDE) analysis is used to eliminate F from the result level. sce Impact:

[0056]

[0057] in and Y c They represent the detection results of general deep representation learning and only using scene bias. Through the cooperation between SdL and CiNL, the proposed CRCL can gradually perceive the scene bias in the entangled representation and eliminate it using the TDE process.

[0058] 2. Causality-inspired normality learning module

[0059] The causality-inspired normality learning module consists of a memory network, a prototype decomposer, and a causality-inspired characterizer (CiC), such as Figure 2 During the training phase, the M3 motion-aware feature extractor extracts the entangled representation F of the same batch of b video clips. ent (To simplify the formula expression, it is also recorded as F), and input it into the memory network, which writes the prototype features of regular events into the memory entries through write operations, and then reconstructs the input entanglement representation F through read operations using the stored memory entries, recorded as the prototype feature F'. Since the memory network is constantly updated during the training process, and only the detailed information of new events and stored entries is recorded during the memory entry update process, it can be considered that F' contains a large number of prototype features of regular events, and F also includes the personalized features of a single abnormal event.

[0060] The prototype decomposer includes two pooling operations and two multi-layer perceptrons with the same structure, the latter of which is denoted as {θ1,θ2}, which consists of a 3-layer fully connected network with sigmoid as the activation function of the last layer. The F output from the M3 motion-aware feature extractor and the F' output from the M4-1 memory network are first averaged and globally pooled along the channel dimension to obtain f avg ,f' avg ,f max ,f' max , and then input it into the multi-layer perceptron to calculate the difference coefficients α and β:

[0061] α=MLP(f avg -f′ avg ;θ1),β=MLP(f max -f′ max ;θ2)

[0062] Where MLP represents a multi-layer perceptron used to calculate the difference coefficients α and β, and its corresponding learnable parameters are denoted as θ1 and θ2 respectively;

[0063] By performing multiplication along the channel dimension, the private feature F can be calculated. p and shared features F s :

[0064]

[0065] Furthermore, CiC is composed of a deep neural network, which uses F p and F sAs input, the features of the two pointing to the same regular event and thus having the same semantic label are used to calculate the causal representation. The process is as follows:

[0066]

[0067] Where R and Respectively represent F s and F p The causal representation of is b×n in dimension, that is, an n-dimensional vector is used to express the private / shared features of the original higher dimension. According to the principle of independent causality, vectors of different dimensions should be orthogonal and independent, and according to F s and F p The consistency of the labels of , the causal representations of the two should be similar. To this end, the following causal loss is proposed to optimize CiC:

[0068]

[0069] Where C1, C2, and C3 represent The correlation matrix of the corresponding column vector of R, that is, C1(i,j) represents the correlation between the i-th column of R and The cosine correlation between the jth column vectors of . Therefore, the dimension of each correlation matrix is ​​n×n. In addition, I is a unit matrix and λ is a balancing hyperparameter. Finally, C1, C2, and C3 are input into the clustering module to obtain a more compact causal representation, such as Figure 2 shown.

[0070] 3. Scene debiasing learning

[0071] Scene debiasing learning includes two scene classification models (M2-1 scene classifier and M2-2 scene classifier), which are based on the scene features F output from the M1 scene encoder. sce and the entangled representation of the output of the M3 motion-aware feature extractor F ent The M1 scene encoder is composed of a simple 2D convolutional network. Compared with the M3 motion-aware feature extractor that uses temporal attention, which focuses on background scene information, the M3 motion-aware feature extractor is a 4-layer encoder based on a 3D convolutional neural network, and temporal attention is embedded between layers to efficiently explore motion information of different temporal lengths and actively focus on important temporal semantics. After receiving scene features from the same batch of b video clips, the M2-1 scene classifier performs parameter optimization under the supervision of the scene label through the following cross-entropy loss function:

[0072]

[0073] where y ijIndicates whether sample i belongs to category j, where 0 means it belongs and 1 means it does not belong, and p ij N represents the probability that sample i belongs to category j predicted by the M2-1 scene classifier. s represents the total number of scenes in the dataset, so SdL can only work on multi-scene datasets. In contrast, the M2-2 scene classifier uses the entangled representation F ent As input, mutual learning is performed through classification tasks to perceive F ent Scene bias in:

[0074]

[0075] in and They represent the scene classification probabilities of the M2-1 scene classifier and the M2-2 scene classifier respectively. Finally, the F ent The bias in . Figure 2 As shown, F sce It is also input into CiNL to calculate the correlation matrix, recorded as {C'1, C'2, C'3}. Since the scene features do not have the causal consistency of the learned video normality, they do not satisfy the causal principle. Therefore, the optimization equation for scene debiasing learning is as follows:

[0076] min(d({C1, C2, C3}, I)-d({C′1, C′2, C′3}, I)+ε, 0)

[0077] Where d() represents the distance metric function, and ε is the margin parameter that controls the minimum distance. Through the cooperation between SdL and CiNL, CRCL can gradually perceive and remove the scene bias in the entanglement representation and use CiC to mine the causal video normality.

[0078] 4. Consistency-based anomaly detection

[0079] In the test phase, the CRCL model detects abnormal events by measuring whether the input sample has the causality learned in the training set. Specifically, since the CRCL model is only trained on normal videos, its M4-1 memory network only stores the prototype features of regular events, while the M4-2 prototype decomposer and the M4-3 causality-inspired characterizer can only express the causal pattern of normal videos. Therefore, anomalies are identified by quantitatively measuring causal consistency and clustering distance, as shown below:

[0080]

[0081] Where D represents the distance between the causal representation of the input sample and the nearest cluster center, and g() represents the maximum and minimum normalization performed on the deviations of all frames of the same video. Therefore, the final anomaly score s for all frames of the same video is t The range is [0, 1], and a higher score indicates that the frame is more abnormal.

[0082] In this embodiment, multi-scenario abnormal event detection for video IoT is specifically oriented. Nvidia 3090 and Pytorch deep learning framework are used to implement the model, the optimizer is Adam, the initial learning rate is 8e-4, and the batch size is 16. All input videos are first cropped to 256x256 pixels. The M5 clustering module is implemented by the nearest neighbor algorithm (K-means), and it starts working after 100 rounds of training to update the clustering center. In this embodiment, a deep residual network ResNet50 is used to construct a causal-inspired characterizer. The balanced hyperparameters λ on the UCSD Ped2, CUHKAvenue, and ShanghaiTech datasets are 10, 18, and 20, respectively.

[0083] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0084] The above is only a preferred implementation of a video anomaly detection method based on causal normality learning disclosed in the present invention, and is not intended to limit the protection scope of the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of this specification shall be included in the protection scope of the embodiments of this specification.

Claims

1. A video anomaly detection method based on causal normality learning, characterized in that: The steps include: Obtain surveillance videos of multiple scenes and crop them into b-frame continuous segments of uniform size as training samples; A CRCL model is constructed, wherein the CRCL model includes an M1 scene encoder, an M3 motion perception feature extractor, a scene debiasing learning module, a causal-inspired normality learning module and an M5 clustering module, wherein the scene debiasing learning module includes an M2-1 scene classifier and an M2-2 scene classifier, wherein the causal-inspired normality learning module includes an M4-1 memory network, an M4-2 prototype decomposer and an M4-3 causal-inspired characterizer, wherein training samples are input into the CRCL model for training, a correlation matrix and a loss function are calculated and model parameters are optimized using gradient back propagation; after reaching a preset round, the cluster center of the causal characterization is calculated by the M5 clustering module and the clustering loss is minimized to obtain the best cluster; rounds of training are repeatedly performed until the decrease value of the loss function between two adjacent rounds of iterations in the training meets a set threshold or the maximum number of iterations is reached, and the training is stopped, and the trained CRCL model is output; The video to be detected is input into the M3 motion perception feature extractor, causality-inspired normality learning module and M5 clustering module in the trained CRCL model, and the deviation from the learned causal representation consistency is calculated and the anomaly score in the range of [0, 1] is calculated. If the anomaly score is greater than the set threshold, the video to be detected is an abnormal event.

2. The video anomaly detection method based on causal normality learning according to claim 1 is characterized in that: The method also includes the following steps: after determining that an abnormal event occurs, a voice alarm mechanism is activated to alert management personnel and corresponding segments are automatically captured and stored.

3. The video anomaly detection method based on causal normality learning according to claim 1 is characterized in that: The processing process of the CRCL model specifically includes the following steps: The M1 scene encoder and M3 motion-aware feature extractor take the same batch of b video clips as input and output scene features F sce and entanglement representation F ent ; M4-1 Memory Network with F ent As input to update the memory entry and output the prototype feature F', the M4-2 prototype decomposer takes F ent and F' as input to calculate the private feature F p and shared features F s , the private feature F p and shared features F s It is simultaneously input to the M4-3 causality-inspired characterizer to calculate the corresponding causal representation and correlation matrix. The M5 clustering module uses the causal representation as input to calculate the cluster center. M2-1 scene classifier is based on scene feature F sce As input to perform scene classification, the M2-2 scene classifier uses F ent is used as input to perform scene classification; scene features F sce Input causality-inspired normality learning module and determine the entanglement representation F ent The resulting correlation matrices are then used to simulate the TDE process to remove the scenario bias.

4. The video anomaly detection method based on causal normality learning according to claim 3 is characterized in that: The M4-1 memory network is based on F ent As input, the memory entry is updated and the prototype feature F' is output. The M4-2 prototype decomposer takes F ent and F' as input to calculate the private feature F p and shared features F s , the private feature F p and shared features F s It is simultaneously input to the M4-3 causality-inspired characterizer to calculate the corresponding causal characterization and correlation matrix, which specifically includes the following steps: The M4-2 prototype decomposer includes two pooling operations and two multi-layer perceptrons with the same structure. The two multi-layer perceptrons with the same structure are composed of a 3-layer fully connected network with sigmoid as the activation function of the last layer; the entanglement representation F ent And prototype feature F' are averaged and globally pooled along the channel dimension to obtain f avg ,f' avg ,f max ,f' max , and then input it into the multi-layer perceptron to calculate the difference coefficients α and β: α=MLP(f avg -f′ avg ;θ1), β=MLP(f max -f′ max ;θ2) Where MLP represents a multi-layer perceptron used to calculate the difference coefficients α and β, and its corresponding learnable parameters are denoted as θ1 and θ2 respectively; Perform multiplication along the channel dimension to calculate the private feature F p and shared features F s : The M4-3 causality-inspired representation is composed of a deep neural network, with F p and F s As input, the features of the two pointing to the same regular event and thus having the same semantic label are used to calculate the causal representation. The process is as follows: Where R and Respectively represent F s and F p The causal representation of , the dimension is b × n, and the causal loss is used to optimize the M4-3 causality-inspired representation, the formula is as follows: Where C1, C2, and C3 represent The correlation matrix of the corresponding column vector of R, that is, C1(i,j) represents the correlation between the i-th column of R and The dimension of the correlation matrix is ​​n×n, I is the unit matrix, and λ is the balancing hyperparameter.

5. The video anomaly detection method based on causal normality learning according to claim 3 is characterized in that: The M2-1 scene classifier is based on the scene feature F sce As input to perform scene classification, the M2-2 scene classifier uses F ent is used as input to perform scene classification; scene features F sce Input causality-inspired normality learning module and determine the entanglement representation F ent The difference in the resulting correlation matrix is ​​used to simulate the TDE process to remove the scene bias, as follows: After receiving the scene features from the same batch of b video clips, the M2-1 scene classifier is optimized with the following cross entropy loss function under the supervision of the scene labels: where y ij Indicates whether sample i belongs to category j, where 0 means it belongs and 1 means it does not belong, p ij represents the probability that sample i belongs to category j predicted by the M2-1 scene classifier, N s Represents the total number of scenes contained in the dataset; M2-2 scene classifier uses entanglement representation F ent As input, mutual learning is performed through classification tasks to perceive F ent Scene bias in: in and represent the scene classification probabilities of the M2-1 scene classifier and the M2-2 scene classifier respectively; Eliminate F by simulating the TDE process ent The bias in F sce It is also input into CiNL to calculate the correlation matrix, recorded as {C'1, C'2, C'3}. The optimization equation for scene debiasing learning is as follows: min(d({C1, C2, C3}, I)-d({C′1, C′2, C′3}, I)+ε, 0) Where d() represents the distance metric function and ε is the margin parameter that controls the minimum distance.

6. The video anomaly detection method based on causal normality learning according to claim 5 is characterized in that: The anomaly score s t , calculation formula: where D represents the distance between the causal representation of the input sample and the nearest cluster center, and g() represents the maximum and minimum normalization performed on the deviations of all frames of the same video.

Images