Weakly supervised video anomaly detection method and system based on multi-head spectral residual gating

Abstract

The application provides a weakly supervised video anomaly detection method and system based on a multi-head spectral residual gating. The method comprises the following steps: segmenting an input video, and extracting features by using a pre-trained feature extractor; dividing a global feature sequence into a predetermined number of subspace heads along a channel dimension; constructing a local autocorrelation matrix, and estimating a background principal component direction corresponding to each subspace by using a differentiable power iteration algorithm; inputting a spectral residual feature into a semantic perception gating network to generate a mask, and enhancing the global feature sequence by using the mask to obtain an enhanced output feature. In the feature manifold, the application models a normal background as a low-rank principal component subspace, and models an anomaly as a sparse and high-energy spectral residual, so that the semantic decoupling of the background and the anomaly is effectively realized.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and video analysis technology, and in particular to a weakly supervised video anomaly detection method and system based on multi-head spectral residual gating. Background Technology

[0002] With the widespread deployment of intelligent surveillance systems, video anomaly detection (VAD) plays an increasingly important role in public safety maintenance. Due to the extreme scarcity and infinite diversity of real-world anomaly events (such as traffic accidents), traditional fully supervised learning methods struggle to obtain sufficient frame-level annotations. In recent years, weakly supervised video anomaly detection (WSVAD) has gradually become the mainstream research paradigm. The model is trained using only video-level binary labels (normal / abnormal) to infer frame-level anomaly localization.

[0003] Most existing WSVAD methods follow the Multiple Instance Learning (MIL) framework. These methods typically utilize pre-trained models (such as I3D or CLIP) to extract features and optimize the classifier through sparsity constraints on Top-K high-scoring segments. Despite progress, existing methods still face key challenges:

[0004] (1) False alarms caused by feature entanglement: Existing feature extractors are often highly sensitive to salient objects in the scene (such as moving vehicles and crowds) rather than focusing on the abnormal behavior itself. The MIL classifier is prone to confusing "dynamic background" with real "abnormal events" and has difficulty decoupling the foreground action from the background manifold.

[0005] (2) The static assumptions of background modeling fail: Early reconstruction methods assumed that normal samples could be well reconstructed by autoencoders, while anomalous samples could not. However, deep neural networks can often perfectly reconstruct anomalous frames (i.e., the "identity mapping" problem). In addition, traditional subspace methods (such as PCA) are usually computed offline and cannot adapt to the real-time changing background distribution in video streams. Summary of the Invention

[0006] In view of the above situation, the main objective of this invention is to propose a weakly supervised video anomaly detection method and system based on multi-head spectrum residual gating, so as to solve the above-mentioned technical problems.

[0007] This invention proposes a weakly supervised video anomaly detection method based on multi-head spectral residual gating, the method comprising the following steps:

[0008] Step 1: Construct an anomaly detection model based on the feature extraction module, multi-head manifold background estimation module, spectral residual generation and gating module, and spectral contrast energy optimization module; use the feature extraction module to segment the input video and use a pre-trained feature extractor to extract features to obtain a global feature sequence;

[0009] Step 2: Input the global feature sequence into the multi-head manifold background estimation module, divide it into a predetermined number of subspace heads along the channel dimension to obtain subspace features; for each subspace feature, construct a local autocorrelation matrix, and use a differentiable exponential iterative algorithm to estimate the background principal component direction corresponding to each subspace in the local autocorrelation matrix;

[0010] Step 3: Project the subspace features onto the corresponding background principal component direction using the spectral residual generation and gating module to obtain the reconstructed background features. Subtract the subspace features from the reconstructed background features to calculate the residual features. Then, stitch together the residual features of all subspaces to obtain the spectral residual features.

[0011] Step 4: Input the spectral residual features into the semantic awareness gating network in the spectral residual generation and gating module to generate a mask. Use the mask to enhance the global feature sequence to obtain the enhanced output features.

[0012] Step 5: Define the spectral escape rate based on the reconstructed background features and the spectral residual features; input the enhanced output features into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain the anomaly probability score of the video segment; construct the multi-instance learning classification loss based on the anomaly probability score of the video segment; construct the spectral contrast energy loss based on the spectral escape rate; jointly optimize the anomaly detection model using the spectral contrast energy loss and the multi-instance learning classification loss to obtain the optimized anomaly detection model;

[0013] The input video is fed into the optimized anomaly detection model to obtain the final anomaly probability score of the video. The final anomaly detection result is determined based on the final anomaly probability score of the video.

[0014] This invention also proposes a weakly supervised video anomaly detection system based on multi-head spectral residual gating, the system comprising:

[0015] Extraction module, used for:

[0016] An anomaly detection model is constructed based on a feature extraction module, a multi-head manifold background estimation module, a spectral residual generation and gating module, and a spectral contrast energy optimization module. The feature extraction module is used to segment the input video, and a pre-trained feature extractor is used to extract features to obtain a global feature sequence.

[0017] The estimation module is used for:

[0018] The global feature sequence is input into the multi-head manifold background estimation module, and is divided into a predetermined number of subspace heads along the channel dimension to obtain subspace features. For each subspace feature, a local autocorrelation matrix is ​​constructed, and the background principal component direction corresponding to each subspace in the local autocorrelation matrix is ​​estimated using a differentiable exponential iterative algorithm.

[0019] The spectral residual module is used for:

[0020] The subspace features are projected onto the corresponding background principal component direction using the spectral residual generation and gating module to obtain the reconstructed background features. The difference between the subspace features and the reconstructed background features is calculated to obtain the residual features. The residual features of all subspaces are then concatenated to obtain the spectral residual features.

[0021] The spectral residual features are input into the semantic-aware gating network in the spectral residual generation and gating module to generate a mask. The mask is then used to enhance the global feature sequence to obtain the enhanced output features.

[0022] The optimization module is used for:

[0023] The spectral escape rate is defined based on the reconstructed background features and the spectral residual features; the enhanced output features are input into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain the anomaly probability score of the video segment; the multi-instance learning classification loss is constructed based on the anomaly probability score of the video segment; the spectral contrast energy loss is constructed based on the spectral escape rate; the anomaly detection model is jointly optimized using the spectral contrast energy loss and the multi-instance learning classification loss to obtain the optimized anomaly detection model.

[0024] The input video is fed into the optimized anomaly detection model to obtain the final anomaly probability score of the video. The final anomaly detection result is determined based on the final anomaly probability score of the video.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] 1. The Spectral Residual Generation and Gating Module (MH-SRG) proposed in this invention innovatively introduces a differentiable exponential iterative layer, which dynamically estimates the principal direction of the multimodal background at extremely low cost within the neural network, realizing end-to-end subspace decomposition and overcoming the limitations of traditional PCA offline computation.

[0027] 2. In this invention, the normal background is modeled as a low-rank principal component subspace and the anomaly is modeled as a sparse and high-energy spectral residual in the feature manifold, which effectively achieves semantic decoupling between the background and the anomaly.

[0028] 3. This invention proposes a physically-aware spectral contrast energy loss method. Based on the perspective of energy conservation, it forces the energy of normal frames to be absorbed by the background substrate, while maximizing the "escape rate" of abnormal frames in the residual subspace. This achieves physical-level decoupling of features at the topological level, significantly improving detection accuracy.

[0029] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the weakly supervised video anomaly detection method based on multi-head spectrum residual gating proposed in this invention.

[0031] Figure 2 This is a diagram illustrating the overall architecture of the weakly supervised video anomaly detection method based on multi-head spectrum residual gating proposed in this invention.

[0032] Figure 3 This is a schematic diagram of the residual gating module of the weakly supervised video anomaly detection method based on multi-head spectrum residual gating proposed in this invention.

[0033] Figure 4 This is a structural diagram of the weakly supervised video anomaly detection system based on multi-head spectrum residual gating proposed in this invention. Detailed Implementation

[0034] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0035] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to provide some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0036] Please see Figure 1 This invention proposes a weakly supervised video anomaly detection method based on multi-head spectral residual gating, which includes the following steps:

[0037] Step 1: Construct an anomaly detection model based on the feature extraction module, multi-head manifold background estimation module, spectral residual generation and gating module, and spectral contrast energy optimization module; use the feature extraction module to segment the input video and use a pre-trained feature extractor to extract features to obtain a global feature sequence.

[0038] Please see Figure 2 In step 1, video segmentation: the input raw video stream is uniformly divided into non-overlapping video snippets. For example, each snippet contains 16 consecutive frames. Feature extraction: a deep neural network pre-trained on a large-scale dataset is used as the feature extractor. In this embodiment, an I3D (Inception3D) network or a CLIP (Contrastive Language-Image Pre-training) visual encoder can be selected. Each video snippet is input into the feature extractor to extract feature vectors; for the entire video, a global feature sequence is obtained.

[0039] It should be noted that the features extracted at this time are mixed, containing a large amount of redundant background information and potential sparse anomaly information.

[0040] Step 2: Input the global feature sequence into the multi-head manifold background estimation module, divide it into a predetermined number of subspace heads along the channel dimension to obtain subspace features; for each subspace feature, construct a local autocorrelation matrix, and use a differentiable exponential iterative algorithm to estimate the background principal component direction corresponding to each subspace in the local autocorrelation matrix.

[0041] In step 2, a local autocorrelation matrix is ​​constructed for each subspace feature, and the background principal component direction corresponding to each subspace in the local autocorrelation matrix is ​​estimated using a differentiable exponential iterative algorithm. Specifically, this includes the following steps:

[0042] S201. For each subspace feature, construct a local autocorrelation matrix and calculate the covariance surrogate matrix for each subspace feature;

[0043] S202, Random initialization vector; wherein the initialization vector matches the dimension of the covariance surrogate matrix;

[0044] S203. Perform a preset number of power iterations on the initialization vector and calculate using matrix multiplication to stretch the initialization vector toward the direction of the largest eigenvalue of the covariance proxy matrix to obtain the intermediate vector.

[0045] It should be noted that matrix multiplication refers to performing matrix multiplication between the covariance surrogate matrix and the differentiable matrix of the initial vector. This enables dynamic, end-to-end estimation of the background principal component direction within the neural network, overcoming the limitations of traditional principal component analysis, which is performed offline and cannot dynamically change with the video background.

[0046] S204. Normalize the intermediate vector to obtain the normalized vector in this iteration;

[0047] S205. Using the normalized vector from this iteration as input, repeat steps S203 and S204. When the maximum number of iterations is reached, obtain the result of the last iteration and define the result of the last iteration as the estimated background principal direction.

[0048] The global feature sequence is divided into a predetermined number of subspace heads along the channel dimension. The corresponding relationship in this process is as follows:

[0049] ;

[0050] in, Represents the global feature sequence. This indicates a splitting operation. This represents the feature tensor of the first spatial head segment. The total number of subspace headers is The feature tensor of the segmentation;

[0051] In the steps of constructing a local autocorrelation matrix for each subspace feature and calculating the covariance surrogate matrix for each subspace feature, the corresponding relationship in the process is as follows:

[0052] ;

[0053] in, The covariance surrogate matrix representing the characteristics of the subspace. Indicates the first Characteristics of each subspace Indicates the index of the subspace. This indicates that a transpose operation is being performed. Indicates batch size is The subspace dimension is A set;

[0054] It should be noted that when constructing the local autocorrelation matrix, this invention first performs matrix multiplication. This step has a dual physical significance; first, it achieves the aggregation of temporal information. Because the original features contain… Directly handling timing jitter at multiple time steps is quite difficult. This matrix operation eliminates the time dimension, mapping the global statistical regularities within a video segment to a subspace with a dimension of [missing information]. In the feature relation space. Second, it acts as a descriptor of the background manifold. Since the background usually occupies the majority of the video energy and changes slowly, the covariance surrogate matrix of the subspace features. It can highlight those high-response feature combinations that persist on the time axis, thus providing a "mathematical basis" for the subsequent extraction of the 'background main direction' through exponential iteration.

[0055] In the process of performing a predetermined number of power iterations on the initialization vector, and calculating through matrix multiplication to stretch the initialization vector towards the direction of the largest eigenvalue of the covariance surrogate matrix to obtain the intermediate vector, the corresponding relationship in the process is as follows:

[0056] ;

[0057] in, Indicates the first The unnormalized intermediate vector in the next iteration Indicates the first The normalized vector in the next iteration;

[0058] In the step of normalizing the intermediate vector to obtain the normalized vector in this iteration, the corresponding relationship in the process is as follows:

[0059] ;

[0060] in, Indicates the first The normalized vector in the next iteration Represents the numerically stable term. This represents the L2 norm of a vector.

[0061] It should be noted that the numerical stability term is a minimal constant introduced to prevent the denominator from being zero.

[0062] Step 3: Project the subspace features onto the corresponding background principal component direction using the spectral residual generation and gating module to obtain the reconstructed background features. Subtract the subspace features from the reconstructed background features to calculate the residual features. Then, stitch together the residual features of all subspaces to obtain the spectral residual features.

[0063] Please see Figure 3 In step 3, the subspace features are projected onto the corresponding background principal component directions using the spectral residual generation and gating module to obtain the reconstructed background features. The difference between the subspace features and the reconstructed background features is then calculated to obtain the residual features. Finally, the residual features of all subspaces are concatenated to obtain the spectral residual features. The specific steps include the following:

[0064] The subspace features are projected onto the corresponding background principal component directions using the spectral residual generation and gating module to obtain the projection scalar;

[0065] Background features are reconstructed using projection scalars to obtain the reconstructed background features;

[0066] The subtraction between the subspace features and the reconstructed background features is performed to obtain the residual features;

[0067] The residual features of all subspace heads are concatenated and the original feature dimensions are restored to obtain the spectral residual features.

[0068] The subspace features are projected onto the corresponding background principal component directions to obtain the projection scalar. The relationship in this process is as follows:

[0069] ;

[0070] in, Represents a projected scalar. Indicates the estimated first The background main direction of each subspace Indicates batch size is Timing length is A set with a feature dimension of 1;

[0071] In the step of reconstructing background features using projection scalars, the corresponding relationship in the process is as follows:

[0072] ;

[0073] in, Indicates the background features of the reconstruction. Indicates batch size is Timing length is Feature dimension is A set;

[0074] In the step of subtracting the subspace features from the reconstructed background features to obtain the residual features, the corresponding relationship in the process is as follows:

[0075] ;

[0076] in, Indicates residual characteristics;

[0077] In the steps of concatenating the residual features of all subspace heads and restoring the original feature dimensions to obtain the spectral residual features, the corresponding relationship in the process is as follows:

[0078] ;

[0079] in, Indicates spectral residual characteristics, The total number of subspace headers is The residual characteristics, Indicates batch size is Timing length is Feature dimension is A set of.

[0080] Furthermore, to avoid the high computational complexity of traditional Singular Value Decomposition (SVD) and to achieve end-to-end differentiable training on GPUs, this embodiment introduces a "differentiable exponential iterative layer" to dynamically estimate the principal background direction of each subspace.

[0081] Step 4: Input the spectral residual features into the semantic-aware gating network to generate a mask. Use the mask to enhance the global feature sequence to obtain the enhanced output features.

[0082] In step 4, the spectral residual features are input into the semantic-aware gating network in the spectral residual generation and gating module to generate a mask. The mask is then used to enhance the global feature sequence, resulting in enhanced output features. The specific steps include the following:

[0083] The spectral residual features are input into the first fully connected layer of the semantic awareness gating network in the spectral residual generation and gating module for dimensionality reduction, and then subjected to layer normalization and ReLU activation to obtain the intermediate feature tensor.

[0084] The dimensions of the intermediate feature tensor are recovered through the second fully connected layer in the semantically aware gated network, and a mask is generated using the sigmoid function.

[0085] The global feature sequence is weighted and enhanced using a mask to obtain the enhanced output features.

[0086] The spectral residual features are input into the first fully connected layer of the semantic awareness gating network in the spectral residual generation and gating module for dimensionality reduction, followed by layer normalization and ReLU activation to obtain the intermediate feature tensor. The corresponding relationship in this process is as follows:

[0087] ;

[0088] in, This represents the intermediate feature tensor obtained after processing through the first layer of linear transformation, layer normalization, and ReLU activation function. express Activation function Presentation layer normalization processing, This indicates the first fully connected layer. This represents the first learnable parameter;

[0089] In the steps of restoring the dimension of the intermediate feature tensor through the second fully connected layer in a semantically aware gated network and generating a mask using the Sigmoid function, the corresponding relationship in the process is as follows:

[0090] ;

[0091] in, Indicates the mask. This represents the Sigmoid function. This indicates the second fully connected layer. This represents the second learnable parameter;

[0092] It should be noted that the semantic-aware gating network is implemented through a lightweight neural network with a bottleneck structure: firstly, through a fully connected layer... The nonlinear activation compresses information, focusing on the significant components in the residual features that differ most from common background patterns; subsequently, a second fully connected layer is used to restore the dimension. Along with the sigmoid function, the learned semantic differences are mapped to a soft attention mask. This mask adaptively enhances the feature responses related to anomalies while suppressing background noise caused by changes in lighting, regular motion, etc.

[0093] In the step of using a mask to weight and enhance the global feature sequence to obtain the enhanced output features, the corresponding relationship in the process is as follows:

[0094] ;

[0095] in, This represents the enhanced output features. This represents element-wise multiplication. Represents a constant.

[0096] Step 5: Define the spectral escape rate based on the reconstructed background features and the spectral residual features; input the enhanced output features into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain the anomaly probability score of the video segment; construct the multi-instance learning classification loss based on the anomaly probability score of the video segment; construct the spectral contrast energy loss based on the spectral escape rate; jointly optimize the anomaly detection model using the spectral contrast energy loss and the multi-instance learning classification loss to obtain the optimized anomaly detection model;

[0097] The input video is fed into the optimized anomaly detection model to obtain the final anomaly probability score of the video. The final anomaly detection result is determined based on the final anomaly probability score of the video.

[0098] In step 5, the spectral escape rate is defined based on the reconstructed background features and the spectral residual features. The corresponding relationship in the process is as follows:

[0099] ;

[0100] in, Indicates the spectral escape rate. Indicates in Spectral residual characteristics at time step Indicates in Background features that are constantly being reconstructed.

[0101] A spectral contrast energy loss is constructed based on the spectral escape rate; wherein, the spectral contrast energy loss includes the energy compression loss of normal videos and the energy expansion loss of abnormal videos;

[0102] The energy compression loss of normal video corresponds to the following relationship:

[0103] ;

[0104] in, This indicates the energy compression loss in normal video. This represents a collection of normal videos. Indicates the duration of the video sequence. Indicates the first In the video Spectral escape rate at time step;

[0105] The energy expansion loss of abnormal videos corresponds to the following relationship:

[0106] ;

[0107] in, This indicates the energy expansion loss of abnormal videos. This represents a collection of abnormal videos. This indicates the calculation of the maximum value. Indicates the calculation of the first In the video The maximum value of the escape rate in the time spectrum. This represents the boundary threshold.

[0108] There is also a total loss function, with the corresponding relationship as follows:

[0109] ;

[0110] in, Represents the total loss function. This represents the multi-instance learning classification loss. This represents the balance coefficient (e.g., 0.1), used to control the strength of physical constraints.

[0111] Among them, in obtaining the total loss function Then, the gradient of the total loss function with respect to the network parameters is calculated using the backpropagation algorithm, and the learnable parameters in the model are updated using the optimizer. The specific optimized parameters include: the pre-trained feature extractor (if involved in fine-tuning), the parameters in the multi-head manifold background estimation module, the parameters of the fully connected layers in the semantic-aware gating network, and the network weights of the multi-instance learning classifier, thereby achieving end-to-end joint training of the entire anomaly detection model.

[0112] It's important to note that the spectral contrast energy loss function is based on explicit physical insights and interpretable mathematical optimization objectives. Its core lies in defining and utilizing the "spectral escape rate" metric to quantify the degree to which the visual features of each video frame deviate from the low-rank subspace formed by the principal components of the background. The physical meaning of the spectral escape rate is to simulate an "energy" distribution process: for a video frame, if it can be well reconstructed from the estimated principal orientation of the background, it indicates that its features mainly originate from the scene background, and its "energy" is considered to converge or be "absorbed" within the background subspace. In this case, the calculated spectral escape rate value approaches zero. Conversely, if the frame contains significant information (i.e., anomalies) that cannot be interpreted by the principal components of the background, its features will significantly deviate from the background subspace, manifesting as high-value spectral residuals, thus increasing the spectral escape rate, symbolizing that the feature "energy" "escapes" from the background substrate.

[0113] The spectral contrast energy loss constructed based on this physical concept comprises two components with clearly defined adversarial optimization objectives: First, the normal video energy compression loss. This loss term constrains the model by minimizing the average spectral escape rate of normal video segments over time, forcing the network to compress and absorb the feature energy of all normal, common dynamic changes (such as regular motion illumination changes) into the low-rank subspace of the background as much as possible, thereby purifying the background representation. Second, the anomalous video energy expansion loss. This loss term does not directly optimize all frames, but rather targets anomalous videos, requiring that at least one frame (i.e., the segment most likely to contain an anomalous element) must have a spectral escape rate exceeding a preset boundary threshold. This design encourages the model to strongly "push" the features corresponding to anomalous events away from the background subspace, ensuring that anomalous information is maximized and preserved in the residual space. By jointly optimizing the above two losses, this invention guides the model to learn feature decoupling from a physical perspective: on the one hand, it continuously refines and tightens the modeling of normal backgrounds; on the other hand, it promotes sufficient mismatch (high escape rate) between anomalous features and the background model. This mechanism fundamentally distinguishes between background dynamics and real anomalies, effectively alleviating the problem of feature entanglement, thus achieving accurate frame-level anomaly detection even with only weak video-level supervision signals.

[0114] Please see Figure 4 This invention also proposes a weakly supervised video anomaly detection system based on multi-head spectral residual gating, the system comprising:

[0115] Extraction module, used for:

[0116] An anomaly detection model is constructed based on a feature extraction module, a multi-head manifold background estimation module, a spectral residual generation and gating module, and a spectral contrast energy optimization module. The feature extraction module is used to segment the input video, and a pre-trained feature extractor is used to extract features to obtain a global feature sequence.

[0117] The estimation module is used for:

[0118] The global feature sequence is input into the multi-head manifold background estimation module, and is divided into a predetermined number of subspace heads along the channel dimension to obtain subspace features. For each subspace feature, a local autocorrelation matrix is ​​constructed, and the background principal component direction corresponding to each subspace in the local autocorrelation matrix is ​​estimated using a differentiable exponential iterative algorithm.

[0119] The spectral residual module is used for:

[0120] The subspace features are projected onto the corresponding background principal component direction using the spectral residual generation and gating module to obtain the reconstructed background features. The difference between the subspace features and the reconstructed background features is calculated to obtain the residual features. The residual features of all subspaces are then concatenated to obtain the spectral residual features.

[0121] The spectral residual features are input into the semantic-aware gating network in the spectral residual generation and gating module to generate a mask. The mask is then used to enhance the global feature sequence to obtain the enhanced output features.

[0122] The optimization module is used for:

[0123] The spectral escape rate is defined based on the reconstructed background features and the spectral residual features; the enhanced output features are input into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain the anomaly probability score of the video segment; the multi-instance learning classification loss is constructed based on the anomaly probability score of the video segment; the spectral contrast energy loss is constructed based on the spectral escape rate; the anomaly detection model is jointly optimized using the spectral contrast energy loss and the multi-instance learning classification loss to obtain the optimized anomaly detection model.

[0124] The input video is fed into the optimized anomaly detection model to obtain the final anomaly probability score of the video. The final anomaly detection result is determined based on the final anomaly probability score of the video.

[0125] To verify the effectiveness of this invention, extensive quantitative and qualitative experiments were conducted on several standard video anomaly detection datasets. These datasets include UCF-Crime and ShanghaiTech. UCF-Crime contains a large number of real-world surveillance videos with severe background noise (such as swaying trees and moving traffic), serving as a challenging benchmark for testing the "feature decoupling capability" of this invention. ShanghaiTech was used to verify the model's ability to capture specific motion anomalies in standard scenarios.

[0126] Under weak supervision, this invention uses the industry-standard frame-level AUC (Area Under the ROCCurve) as the primary evaluation metric and compares it with 10 existing (SOTA) methods from recent years (including 2024 and earlier), covering various approaches based on reconstruction, prediction, and multiple instance learning (MIL). Table 1 shows the performance comparison results (AUC / %) of this invention with the current state-of-the-art techniques on the UCF-Crime and ShanghaiTech datasets:

[0127] Table 1. Performance comparison with existing technologies (AUC / %)

[0128]

[0129] Based on the analysis of the experimental data in Table 1, the following conclusions can be drawn:

[0130] (1) On the highly challenging complex background dataset UCF-Crime, this invention achieved an AUC of 88.64%, setting a new best performance record in this field. Compared with traditional MIL methods (such as RTFM), this invention achieved a significant advantage (+4.34%). This proves that relying solely on classification boundary optimization is insufficient to solve the "feature entanglement" problem, while the strategy of explicitly decoupling anomalies from the dynamic background through "multi-head spectral residuals" is more effective.

[0131] (2) Superior to existing feature decoupling methods. Compared to the UR-DMU method, which attempts to separate features through clustering or memory modules, this invention demonstrates superior robustness. This indicates that using "differentiable exponential iteration" to dynamically estimate the background principal components can more accurately adapt to the real-time changing background distribution in the video stream than static cluster centers, avoiding the lag in background modeling.

[0132] (3) Comparable to or even surpassing large-scale multimodal models. Notably, MH-SRG's performance is comparable to methods like VadCLIP that rely on additional textual modal information. This invention achieves higher accuracy without introducing textual descriptions and complex prompt engineering. This demonstrates that optimizing from the underlying physical logic (spectral residuals and energy conservation) is an efficient way to improve the model's understanding of the nature of anomalies, and its inference efficiency is far higher than that of large multimodal models.

[0133] To verify the necessity and contribution of each module in this invention, a decomposition analysis was performed on the UCF-Crime dataset. Table 2 shows the ablation experimental results of the core modules:

[0134] Table 2 Ablation Experiment Results

[0135]

[0136] Table 2 clearly illustrates the independent contribution of each innovative module of this invention:

[0137] (1) The core role of spectral residual generation (SR): Compared with Experiment 2 and Experiment 3, the performance was significantly improved after the introduction of the spectral residual mechanism. It is believed that the model no longer implicitly learns the classification boundary, but infers based on the physical assumption that "anomalies cannot be explained by the principal components of the background", which effectively suppresses false alarms caused by dynamic background (such as cloud drift).

[0138] (2) Physical constraint effect of spectral contrast energy loss (SCE Loss): Experiment 4 shows that the model achieves the best performance after adding energy loss. This is because simple residual calculation may contain high-frequency noise, while the loss function based on energy conservation forces the energy of normal frames to be "absorbed" by the background subspace, while forcing the energy of abnormal frames to "escape", thus ensuring the purity of feature decoupling at the topological level.

[0139] Qualitative analysis and visualization:

[0140] In addition to its quantitative performance advantages, this invention also possesses strong interpretability. To visually demonstrate how this invention solves the "feature entanglement" problem, we conducted a visualization analysis of the model's feature distribution and energy curves in typical scenarios:

[0141] ① Semantic Decoupling Visualization: In the complex dynamic scenario of "traffic-intersection," the features extracted by the baseline model often confuse moving vehicles (normal) with red-light running behavior (abnormal). In contrast, the MH-SRG module of this invention successfully reconstructs the moving traffic flow as "background principal components," while retaining sudden red-light running vehicles in the "spectral residuals," achieving clear separation at the visual semantic level.

[0142] ② Spectral Escape Ratio Curve: At the instant an anomalous event (such as theft) occurs, the spectral escape ratio calculated in this invention exhibits a sharp spike. During normal periods, despite movement of people, the escape ratio remains low. This indicates that the model successfully establishes a "background gravitational field" based on energy conservation; only genuine anomalous events can generate sufficient energy to disrupt the equilibrium and allow escape, rather than simple motion detection.

[0143] In summary, the experimental results fully demonstrate that the weakly supervised video anomaly detection method based on multi-head spectral residual gating proposed in this invention can effectively utilize differentiable exponential iteration and spectral energy theory to break feature entanglement, achieving high-precision and highly interpretable video anomaly detection under weak supervision.

[0144] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0145] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0146] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A weakly supervised video anomaly detection method based on multi-head spectral residual gating, characterized in that, The method includes the following steps: Step 1: Construct an anomaly detection model based on the feature extraction module, multi-head manifold background estimation module, spectral residual generation and gating module, and spectral contrast energy optimization module; use the feature extraction module to segment the input video and use a pre-trained feature extractor to extract features to obtain a global feature sequence; Step 2: Input the global feature sequence into the multi-head manifold background estimation module, divide it into a predetermined number of subspace heads along the channel dimension to obtain subspace features; for each subspace feature, construct a local autocorrelation matrix, and use a differentiable exponential iterative algorithm to estimate the background principal component direction corresponding to each subspace in the local autocorrelation matrix. Specifically, this includes: S201: For each subspace feature, construct a local autocorrelation matrix and calculate the covariance surrogate matrix of each subspace feature; S202, Random initialization vector; wherein the initialization vector matches the dimension of the covariance surrogate matrix; S203. Perform a preset number of power iterations on the initialization vector and calculate using matrix multiplication to stretch the initialization vector toward the direction of the largest eigenvalue of the covariance proxy matrix to obtain the intermediate vector. S204. Normalize the intermediate vector to obtain the normalized vector in this iteration; S205. Using the normalized vector from this iteration as input, repeat steps S203 and S204. When the maximum number of iterations is reached, obtain the result of the last iteration and define the result of the last iteration as the estimated background main direction. Step 3: Project the subspace features onto the corresponding background principal component directions using the spectral residual generation and gating module to obtain the projection scalar; reconstruct the background features using the projection scalar to obtain the reconstructed background features; subtract the subspace features from the reconstructed background features to obtain the residual features; stitch together the residual features of all subspace heads and restore the original feature dimensions to obtain the spectral residual features. Step 4: Input the spectral residual features into the semantic awareness gating network in the spectral residual generation and gating module to generate a mask. Use the mask to enhance the global feature sequence to obtain the enhanced output features. Step 5: Define the spectral escape rate based on the reconstructed background features and the spectral residual features. The spectral escape rate is: ； in, Indicates the spectral escape rate. Indicates in Spectral residual characteristics at time step Indicates in Background features that are constantly being reconstructed. This represents the first numerically stable term; the enhanced output features are input into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain the anomaly probability score of the video segment; a multi-instance learning classification loss is constructed based on the anomaly probability score of the video segment; a spectral contrast energy loss is constructed based on the spectral escape rate, which includes the energy compression loss of normal videos and the energy expansion loss of abnormal videos, wherein the energy compression loss of normal videos is: ； in, This indicates the energy compression loss in normal video. This represents a collection of normal videos. Indicates the duration of the video sequence. Indicates the first In the video Spectral escape rate at time step; The energy expansion loss of the anomalous video is: ； in, This indicates the energy expansion loss of abnormal videos. This represents a collection of abnormal videos. This indicates the calculation of the maximum value. Indicates the calculation of the first In the video The maximum value of the escape rate in the time spectrum. Indicates the boundary threshold; The anomaly detection model is jointly optimized by spectral contrast energy loss and multi-instance learning classification loss to obtain the optimized anomaly detection model. The input video is fed into the optimized anomaly detection model to obtain the final anomaly probability score of the video. The final anomaly detection result is determined based on the final anomaly probability score of the video.

2. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 1, characterized in that, The global feature sequence is divided into a predetermined number of subspace heads along the channel dimension. The corresponding relationship in this process is as follows: ； in, Represents the global feature sequence. This indicates a splitting operation. This represents the feature tensor of the first subspace head segmentation. Indicates the last subspace header The feature tensor of the segmentation; In the steps of constructing a local autocorrelation matrix for each subspace feature and calculating the covariance surrogate matrix for each subspace feature, the corresponding relationship in the process is as follows: ； in, The covariance surrogate matrix representing the characteristics of the subspace. Indicates the first Feature tensors of subspace head segmentation Indicates the index of the subspace. This indicates that a transpose operation is being performed. Indicates batch size is The subspace dimension is A set; In the process of performing a predetermined number of power iterations on the initialization vector, and calculating through matrix multiplication to stretch the initialization vector towards the direction of the largest eigenvalue of the covariance surrogate matrix to obtain the intermediate vector, the corresponding relationship in the process is as follows: ； in, Indicates the first The unnormalized intermediate vector in the next iteration Indicates the first The normalized vector in the next iteration; In the step of normalizing the intermediate vector to obtain the normalized vector in this iteration, the corresponding relationship in the process is as follows: ； in, Indicates the first The normalized vector in the next iteration This represents the second numerically stable term. This represents the L2 norm of a vector.

3. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 2, characterized in that, The subspace features are projected onto the corresponding background principal component directions to obtain the projection scalar. The relationship in this process is as follows: ； in, Represents a projected scalar. Indicates the estimated first The background main direction of each subspace Indicates batch size is Timing length is A set with a feature dimension of 1; In the step of reconstructing background features using projection scalars, the corresponding relationship in the process is as follows: ； in, Indicates the background features of the reconstruction. Indicates batch size is Timing length is Feature dimension is A set; In the step of subtracting the subspace features from the reconstructed background features to obtain the residual features, the corresponding relationship in the process is as follows: ； in, Indicates residual characteristics; In the steps of concatenating the residual features of all subspace heads and restoring the original feature dimensions to obtain the spectral residual features, the corresponding relationship in the process is as follows: ； in, Indicates spectral residual characteristics, Indicates the last subspace header The residual characteristics, Indicates batch size is Timing length is Feature dimension is A set of.

4. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 3, characterized in that, In step 4, the spectral residual features are input into the semantic-aware gating network in the spectral residual generation and gating module to generate a mask. The mask is then used to enhance the global feature sequence to obtain the enhanced output features. Specifically, this includes the following steps: The spectral residual features are input into the first fully connected layer of the semantic awareness gating network in the spectral residual generation and gating module for dimensionality reduction, and then subjected to layer normalization and ReLU activation to obtain the intermediate feature tensor. The dimensions of the intermediate feature tensor are recovered through the second fully connected layer in the semantically aware gated network, and a mask is generated using the sigmoid function. The global feature sequence is weighted and enhanced using a mask to obtain the enhanced output features.

5. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 4, characterized in that, The spectral residual features are input into the first fully connected layer of the semantic awareness gating network in the spectral residual generation and gating module for dimensionality reduction, followed by layer normalization and ReLU activation to obtain the intermediate feature tensor. The corresponding relationship in this process is as follows: ； in, This represents the intermediate feature tensor obtained after processing through the first layer of linear transformation, layer normalization, and ReLU activation function. express Activation function Presentation layer normalization processing, This indicates the first fully connected layer. This represents the first learnable parameter; In the steps of restoring the dimension of the intermediate feature tensor through the second fully connected layer in a semantically aware gated network and generating a mask using the Sigmoid function, the corresponding relationship in the process is as follows: ； in, Indicates the mask. This represents the Sigmoid function. This indicates the second fully connected layer. This represents the second learnable parameter; In the step of using a mask to weight and enhance the global feature sequence to obtain the enhanced output features, the corresponding relationship in the process is as follows: ； in, This represents the enhanced output features. This indicates element-wise multiplication.

6. A weakly supervised video anomaly detection system based on multi-head spectral residual gating, characterized in that, The system employs the weakly supervised video anomaly detection method based on multi-head spectral residual gating as described in any one of claims 1-5, and the system comprises: Extraction module, used for: An anomaly detection model is constructed based on a feature extraction module, a multi-head manifold background estimation module, a spectral residual generation and gating module, and a spectral contrast energy optimization module. The feature extraction module is used to segment the input video, and a pre-trained feature extractor is used to extract features to obtain a global feature sequence. The estimation module is used for: The global feature sequence is input into the multi-head manifold background estimation module, and is divided into a predetermined number of subspace heads along the channel dimension to obtain subspace features. For each subspace feature, a local autocorrelation matrix is ​​constructed, and the background principal component direction corresponding to each subspace in the local autocorrelation matrix is ​​estimated using a differentiable exponential iterative algorithm. The spectral residual module is used for: The subspace features are projected onto the corresponding background principal component direction using the spectral residual generation and gating module to obtain the reconstructed background features. The difference between the subspace features and the reconstructed background features is calculated to obtain the residual features. The residual features of all subspaces are then concatenated to obtain the spectral residual features. The spectral residual features are input into the semantic-aware gating network in the spectral residual generation and gating module to generate a mask. The mask is then used to enhance the global feature sequence to obtain the enhanced output features. The optimization module is used for: The spectral escape rate is defined based on the reconstructed background features and the spectral residual features; the enhanced output features are input into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain the anomaly probability score of the video segment; the multi-instance learning classification loss is constructed based on the anomaly probability score of the video segment; the spectral contrast energy loss is constructed based on the spectral escape rate; the anomaly detection model is jointly optimized using the spectral contrast energy loss and the multi-instance learning classification loss to obtain the optimized anomaly detection model. The input video is fed into the optimized anomaly detection model to obtain the final anomaly probability score of the video. The final anomaly detection result is determined based on the final anomaly probability score of the video.

Images