Weak supervision video anomaly detection method and system based on multi-head spectrum residual gating

Abstract

The invention provides a multi-head spectrum residual gating-based weak supervision video anomaly detection method and system, and the method comprises the steps: carrying out the segment division of an input video, and carrying out the feature extraction through a pre-training feature extractor; segmenting the global feature sequence into a predetermined number of subspace heads along the channel dimension; constructing a local autocorrelation matrix, and estimating a background principal component direction corresponding to each subspace by using a micro-power iterative algorithm; and inputting the spectral residual features into a semantic perception gating network to generate a mask, and enhancing the global feature sequence by using the mask to obtain enhanced output features. According to the method, a normal background is modeled as a low-rank principal component molecular space in a feature manifold, and an anomaly is modeled as a sparse and high-energy spectrum residual, so that semantic decoupling of the background and the anomaly is effectively realized.

Description

A Weakly Supervised Video Anomaly Detection Method and System Based on Multi-head Spectrum Residual Gating Technical Field

[0001] This invention relates to the fields of computer vision and video analysis technology, and in particular to a weakly supervised video anomaly detection method and system based on multi-head spectral residual gating. Background Technology

[0002] With the widespread deployment of intelligent surveillance systems, video anomaly detection (VAD) plays an increasingly important role in public safety maintenance. Due to the extreme scarcity and infinite diversity of real-world anomaly events (such as traffic accidents), traditional fully supervised learning methods struggle to obtain sufficient frame-level annotations. In recent years, weakly supervised video anomaly detection (WSVAD) has gradually become the mainstream research paradigm. The model is trained using only video-level binary labels (normal / abnormal) to infer frame-level anomaly localization.

[0003] Most existing WSVAD methods follow the Multiple Instance Learning (MIL) framework. These methods typically utilize pre-trained models (such as I3D or CLIP) to extract features and optimize the classifier by sparse constraints on Top-K high-scoring segments. Despite progress, existing methods still face key challenges: (1) False positives caused by feature entanglement: Existing feature extractors are often highly sensitive to salient objects in the scene (such as moving vehicles or crowds) rather than focusing on the anomalous behavior itself. MIL classifiers are prone to confusing "dynamic background" with real "abnormal events" and have difficulty decoupling foreground actions from the background manifold.

[0004] (2) The static assumptions of background modeling fail: Early reconstruction methods assumed that normal samples could be well reconstructed by autoencoders, while anomalous samples could not. However, deep neural networks can often perfectly reconstruct anomalous frames (i.e., the "identity mapping" problem). In addition, traditional subspace methods (such as PCA) are usually computed offline and cannot adapt to the real-time changing background distribution in video streams. Summary of the Invention

[0005] In view of the above situation, the main objective of this invention is to propose a weakly supervised video anomaly detection method and system based on multi-head spectrum residual gating, so as to solve the above-mentioned technical problems.

[0006] This invention proposes a weakly supervised video anomaly detection method based on multi-head spectral residual gating. The method includes the following steps: Step 1: Construct an anomaly detection model based on a feature extraction module, a multi-head manifold background estimation module, a spectral residual generation and gating module, and a spectral contrast energy optimization module; use the feature extraction module to segment the input video and use a pre-trained feature extractor to extract features to obtain a global feature sequence; Step 2: Input the global feature sequence into the multi-head manifold background estimation module, and divide it into a predetermined number of subspace heads along the channel dimension to obtain subspace features; for each subspace feature, construct a local autocorrelation matrix, and use a differentiable exponential iterative algorithm to estimate the background principal component direction corresponding to each subspace in the local autocorrelation matrix; Step 3: Use the spectral residual generation and gating module to project the subspace features onto the corresponding background principal component direction to obtain reconstructed background features, and calculate the difference between the subspace features and the reconstructed background features to obtain the residual. Step 4: Input the spectral residual features into the semantic awareness gating network in the spectral residual generation and gating module to generate a mask. Use the mask to enhance the global feature sequence to obtain the enhanced output features. Step 5: Define the spectral escape rate based on the reconstructed background features and the spectral residual features. Input the enhanced output features into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain the anomaly probability score of the video segment. Construct the multi-instance learning classification loss based on the anomaly probability score of the video segment. Construct the spectral contrast energy loss based on the spectral escape rate. Jointly optimize the anomaly detection model using the spectral contrast energy loss and the multi-instance learning classification loss to obtain the optimized anomaly detection model. Input the input video into the optimized anomaly detection model to obtain the final anomaly probability score of the video. Determine the final anomaly detection result based on the final anomaly probability score of the video.

[0007] This invention also proposes a weakly supervised video anomaly detection system based on multi-head spectral residual gating. The system includes: an extraction module, used to: construct an anomaly detection model based on a feature extraction module, a multi-head manifold background estimation module, a spectral residual generation and gating module, and a spectral contrast energy optimization module; segment the input video using the feature extraction module and extract features using a pre-trained feature extractor to obtain a global feature sequence; an estimation module, used to: input the global feature sequence into the multi-head manifold background estimation module, segment it along the channel dimension into a predetermined number of subspace heads to obtain subspace features; construct a local autocorrelation matrix for each subspace feature, and estimate the background principal component direction corresponding to each subspace in the local autocorrelation matrix using a differentiable exponential iterative algorithm; and a spectral residual module, used to: project the subspace features onto the corresponding background principal component direction using the spectral residual generation and gating module to obtain reconstructed background features, and subtract the subspace features from the reconstructed background features. The residual features are calculated and concatenated from the residual features of all subspaces to obtain the spectral residual features. These spectral residual features are then input into the semantic-aware gating network in the spectral residual generation and gating module to generate a mask. The mask is used to enhance the global feature sequence, resulting in enhanced output features. An optimization module is used to: define the spectral escape rate based on the reconstructed background features and the spectral residual features; input the enhanced output features into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain anomaly probability scores for video segments; construct a multi-instance learning classification loss based on the anomaly probability scores of the video segments; construct a spectral contrast energy loss based on the spectral escape rate; jointly optimize the anomaly detection model using the spectral contrast energy loss and the multi-instance learning classification loss to obtain an optimized anomaly detection model; input the input video into the optimized anomaly detection model to obtain the final anomaly probability score for the video; and determine the final anomaly detection result based on the final anomaly probability score.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The spectral residual generation and gating module (MH-SRG) proposed in this invention innovatively introduces a differentiable exponential iteration layer, which dynamically estimates the principal direction of the multimodal background at a very low cost inside the neural network, realizes end-to-end subspace decomposition, and overcomes the limitations of traditional PCA offline calculation.

[0009] 2. In this invention, the normal background is modeled as a low-rank principal component subspace and the anomaly is modeled as a sparse and high-energy spectral residual in the feature manifold, which effectively achieves semantic decoupling between the background and the anomaly.

[0010] 3. This invention proposes a physically-aware spectral contrast energy loss method. Based on the perspective of energy conservation, it forces the energy of normal frames to be absorbed by the background substrate, while maximizing the "escape rate" of abnormal frames in the residual subspace. This achieves physical-level decoupling of features at the topological level, significantly improving detection accuracy.

[0011] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0012] Figure 1 is a flowchart illustrating the weakly supervised video anomaly detection method based on multi-head spectrum residual gating proposed in this invention.

[0013] Figure 2 is an overall architecture diagram of the weakly supervised video anomaly detection method based on multi-head spectrum residual gating proposed in this invention.

[0014] Figure 3 is a schematic diagram of the residual gating module of the weakly supervised video anomaly detection method based on multi-head spectrum residual gating proposed in this invention.

[0015] Figure 4 is an overall structural diagram of the weakly supervised video anomaly detection system based on multi-head spectrum residual gating proposed in this invention. Detailed Implementation

[0016] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0017] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to provide some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0018] Please refer to Figure 1. This embodiment of the invention proposes a weakly supervised video anomaly detection method based on multi-head spectral residual gating. The method includes the following steps: Step 1: An anomaly detection model is constructed based on a feature extraction module, a multi-head manifold background estimation module, a spectral residual generation and gating module, and a spectral contrast energy optimization module; the input video is segmented using the feature extraction module, and a pre-trained feature extractor is used to extract features to obtain a global feature sequence.

[0019] Referring to Figure 2, in step 1, video segmentation: the input raw video stream is uniformly divided into non-overlapping video snippets. For example, each snippet contains 16 consecutive frames. Feature extraction: a deep neural network pre-trained on a large-scale dataset is used as the feature extractor. In this embodiment, an I3D (Inception3D) network or a CLIP (Contrastive Language-Image Pre-training) visual encoder can be selected. Each video snippet is input into the feature extractor to extract feature vectors; for the entire video, a global feature sequence is obtained.

[0020] It should be noted that the features extracted at this time are mixed, containing a large amount of redundant background information and potential sparse anomaly information.

[0021] Step 2: Input the global feature sequence into the multi-head manifold background estimation module, divide it into a predetermined number of subspace heads along the channel dimension to obtain subspace features; for each subspace feature, construct a local autocorrelation matrix, and use a differentiable exponential iterative algorithm to estimate the background principal component direction corresponding to each subspace in the local autocorrelation matrix.

[0022] In step 2, a local autocorrelation matrix is ​​constructed for each subspace feature, and the background principal component direction corresponding to each subspace in the local autocorrelation matrix is ​​estimated using a differentiable power-law iterative algorithm. Specifically, this includes the following steps: S201, constructing a local autocorrelation matrix for each subspace feature and calculating the covariance surrogate matrix for each subspace feature; S202, randomly initializing a vector; wherein the initialization vector matches the dimension of the covariance surrogate matrix; S203, performing a preset number of power-law iterations on the initialization vector, and calculating using matrix multiplication to stretch the initialization vector towards the direction of the largest eigenvalue of the covariance surrogate matrix, obtaining an intermediate vector. It should be noted that calculating using matrix multiplication refers to performing matrix multiplication between the covariance surrogate matrix and the differentiable matrix of the initial vector, realizing dynamic, end-to-end estimation of the background principal component direction within the neural network, overcoming the limitations of traditional principal component analysis, which is offline calculation and cannot dynamically change with the video background.

[0023] S204. Normalize the intermediate vector to obtain the normalized vector in this iteration; S205. Repeat steps S203 and S204 with the normalized vector in this iteration as input. When the maximum number of iterations is reached, obtain the result of the last iteration and define the result of the last iteration as the estimated background principal direction.

[0024] The global feature sequence is divided into a predetermined number of subspace heads along the channel dimension. The corresponding relationship in this process is as follows: ;in, Represents the global feature sequence. This indicates a splitting operation. This represents the feature tensor of the first spatial head segment. The total number of subspace headers is The feature tensor of the segmentation; in the steps of constructing the local autocorrelation matrix for each subspace feature and calculating the covariance surrogate matrix of each subspace feature, the corresponding relationship in the process is as follows: ;in, The covariance surrogate matrix representing the characteristics of the subspace. Indicates the first Characteristics of each subspace Indicates the index of the subspace. This indicates that a transpose operation is being performed. Indicates batch size is The subspace dimension is The set; it should be noted that when constructing the local autocorrelation matrix, this invention first performs matrix multiplication. This step has a dual physical significance; first, it achieves the aggregation of temporal information. Because the original features contain… Directly handling timing jitter at multiple time steps is quite difficult. This matrix operation eliminates the time dimension, mapping the global statistical regularities within a video segment to a subspace with a dimension of [missing information]. In the feature relation space. Second, it acts as a descriptor of the background manifold. Since the background usually occupies the majority of the video energy and changes slowly, the covariance surrogate matrix of the subspace features. It can highlight those high-response feature combinations that persist on the time axis, thus providing a "mathematical basis" for the subsequent extraction of the 'background main direction' through exponential iteration.

[0025] In the process of performing a predetermined number of power iterations on the initialization vector, and calculating through matrix multiplication to stretch the initialization vector towards the direction of the largest eigenvalue of the covariance surrogate matrix to obtain the intermediate vector, the corresponding relationship is as follows: ;in, Indicates the first The unnormalized intermediate vector in the next iteration Indicates the first The normalized vector in the next iteration; in the step of normalizing the intermediate vector to obtain the normalized vector in this iteration, the corresponding relationship exists as follows: ;in, Indicates the first The normalized vector in the next iteration Represents the numerically stable term. This represents the L2 norm of a vector.

[0026] It should be noted that the numerical stability term is a minimal constant introduced to prevent the denominator from being zero.

[0027] Step 3: Project the subspace features onto the corresponding background principal component direction using the spectral residual generation and gating module to obtain the reconstructed background features. Subtract the subspace features from the reconstructed background features to calculate the residual features. Then, stitch together the residual features of all subspaces to obtain the spectral residual features.

[0028] Please refer to Figure 3. In step 3, the subspace features are projected onto the corresponding background principal component directions using the spectral residual generation and gating module to obtain reconstructed background features. The difference between the subspace features and the reconstructed background features is calculated to obtain residual features. The residual features of all subspaces are then concatenated to obtain spectral residual features. Specifically, the steps include: projecting the subspace features onto the corresponding background principal component directions using the spectral residual generation and gating module to obtain projection scalars; reconstructing background features using projection scalars to obtain reconstructed background features; subtracting the subspace features from the reconstructed background features to obtain residual features; and concatenating the residual features of all subspace heads and restoring the original feature dimensions to obtain spectral residual features.

[0029] The subspace features are projected onto the corresponding background principal component directions to obtain the projection scalar. The relationship in this process is as follows: ;in, Represents a projected scalar. Indicates the estimated first The background main direction of each subspace Indicates batch size is The timing length is A set of features with a feature dimension of 1; in the step of reconstructing background features using projection scalars to obtain the reconstructed background features, the corresponding relationship in the process is as follows: ;in, Indicates the background features of the reconstruction. Indicates batch size is The timing length is Feature dimension is The set; in the step of subtracting the subspace features from the reconstructed background features to obtain the residual features, the corresponding relationship in the process is as follows: ;in, The residual features are represented by the following formula: In the steps of concatenating the residual features of all subspace heads and restoring the original feature dimensions to obtain the spectral residual features, the corresponding process relationships are as follows: ;in, Indicates spectral residual characteristics, The total number of subspace headers is The residual characteristics, Indicates batch size is The timing length is Feature dimension is A set of.

[0030] Furthermore, to avoid the high computational complexity of traditional Singular Value Decomposition (SVD) and to achieve end-to-end differentiable training on GPUs, this embodiment introduces a "differentiable exponential iterative layer" to dynamically estimate the principal background direction of each subspace.

[0031] Step 4: Input the spectral residual features into the semantic-aware gating network to generate a mask. Use the mask to enhance the global feature sequence to obtain the enhanced output features.

[0032] In step 4, the spectral residual features are input into the semantic-aware gating network in the spectral residual generation and gating module to generate a mask. The mask is then used to enhance the global feature sequence to obtain the enhanced output features. Specifically, this includes the following steps: inputting the spectral residual features into the first fully connected layer of the semantic-aware gating network in the spectral residual generation and gating module for dimensionality reduction, followed by layer normalization and ReLU activation to obtain an intermediate feature tensor; restoring the dimensionality of the intermediate feature tensor through the second fully connected layer in the semantic-aware gating network and generating a mask using the Sigmoid function; and using the mask to perform weighted enhancement on the global feature sequence to obtain the enhanced output features.

[0033] The spectral residual features are input into the first fully connected layer of the semantic awareness gating network in the spectral residual generation and gating module for dimensionality reduction, followed by layer normalization and ReLU activation to obtain the intermediate feature tensor. The corresponding relationship in this process is as follows: ;in, This represents the intermediate feature tensor obtained after processing through the first layer of linear transformation, layer normalization, and ReLU activation function. express Activation function Presentation layer normalization processing, This indicates the first fully connected layer. Let represent the first learnable parameter; in the steps of restoring the dimension of the intermediate feature tensor through the second fully connected layer in the semantically aware gating network and generating a mask using the Sigmoid function, the corresponding relationship in the process is as follows: ;in, Indicates the mask. This represents the Sigmoid function. This indicates the second fully connected layer. This represents the second learnable parameter; it should be noted that the semantic-aware gating network is implemented through a lightweight neural network with a bottleneck structure: firstly, through a fully connected layer... With nonlinear activation compression, information is focused on the significant components in the residual features that differ most from common background patterns; subsequently, a second fully connected layer is used to restore the dimension. Along with the sigmoid function, the learned semantic differences are mapped to a soft attention mask. This mask adaptively enhances the feature responses related to anomalies while suppressing background noise caused by changes in lighting, regular motion, etc.

[0034] In the step of using a mask to weight and enhance the global feature sequence to obtain the enhanced output features, the corresponding relationship in the process is as follows: ;in, This represents the enhanced output features. This represents element-wise multiplication. Represents a constant.

[0035] Step 5: Define the spectral escape rate based on the reconstructed background features and the spectral residual features; input the enhanced output features into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain the anomaly probability score of the video segment; construct the multi-instance learning classification loss based on the anomaly probability score of the video segment; construct the spectral contrast energy loss based on the spectral escape rate; jointly optimize the anomaly detection model using the spectral contrast energy loss and the multi-instance learning classification loss to obtain the optimized anomaly detection model; input the input video into the optimized anomaly detection model to obtain the final anomaly probability score of the video, and determine the final anomaly detection result based on the final anomaly probability score of the video.

[0036] In step 5, the spectral escape rate is defined based on the reconstructed background features and the spectral residual features. The corresponding relationship in the process is as follows: ;in, Indicates the spectral escape rate. Indicates in Spectral residual characteristics at time step Indicates in Background features that are constantly being reconstructed.

[0037] A spectral contrast energy loss is constructed based on spectral escape rate; whereby the spectral contrast energy loss includes energy compression loss for normal videos and energy expansion loss for abnormal videos; the energy compression loss for normal videos has the following corresponding relationship: ;in, This indicates the energy compression loss in normal video. This represents a collection of normal videos. Indicates the duration of the video sequence. Indicates the first In the video The spectral escape rate at time step 1; the energy expansion loss of the anomalous video, and the corresponding relationship are as follows: ;in, This indicates the energy expansion loss of abnormal videos. This represents a collection of abnormal videos. This indicates the calculation of the maximum value. Indicates the calculation of the first In the video The maximum value of the escape rate in the time spectrum. Indicates the boundary threshold.

[0038] There is also a total loss function, with the corresponding relationship as follows: ;in, Represents the total loss function. This represents the multi-instance learning classification loss. This represents the balance coefficient (e.g., 0.1), used to control the strength of physical constraints.

[0039] Among them, in obtaining the total loss function Then, the gradient of the total loss function with respect to the network parameters is calculated using the backpropagation algorithm, and the learnable parameters in the model are updated using the optimizer. The specific optimized parameters include: the pre-trained feature extractor (if involved in fine-tuning), the parameters in the multi-head manifold background estimation module, the parameters of the fully connected layers in the semantic-aware gating network, and the network weights of the multi-instance learning classifier, thereby achieving end-to-end joint training of the entire anomaly detection model.

[0040] It's important to note that the spectral contrast energy loss function is based on explicit physical insights and interpretable mathematical optimization objectives. Its core lies in defining and utilizing the "spectral escape rate" metric to quantify the degree to which the visual features of each video frame deviate from the low-rank subspace formed by the principal components of the background. The physical meaning of the spectral escape rate is to simulate an "energy" distribution process: for a video frame, if it can be well reconstructed from the estimated principal orientation of the background, it indicates that its features mainly originate from the scene background, and its "energy" is considered to converge or be "absorbed" within the background subspace. In this case, the calculated spectral escape rate value approaches zero. Conversely, if the frame contains significant information (i.e., anomalies) that cannot be interpreted by the principal components of the background, its features will significantly deviate from the background subspace, manifesting as high-value spectral residuals, thus increasing the spectral escape rate, symbolizing that the feature "energy" "escapes" from the background substrate.

[0041] The spectral contrast energy loss constructed based on this physical concept comprises two components with clearly defined adversarial optimization objectives: First, the normal video energy compression loss. This loss term constrains the model by minimizing the average spectral escape rate of normal video segments over time, forcing the network to compress and absorb the feature energy of all normal, common dynamic changes (such as regular motion illumination changes) into the low-rank subspace of the background as much as possible, thereby purifying the background representation. Second, the anomalous video energy expansion loss. This loss term does not directly optimize all frames, but rather targets anomalous videos, requiring that at least one frame (i.e., the segment most likely to contain an anomalous element) must have a spectral escape rate exceeding a preset boundary threshold. This design encourages the model to strongly "push" the features corresponding to anomalous events away from the background subspace, ensuring that anomalous information is maximized and preserved in the residual space. By jointly optimizing the above two losses, this invention guides the model to learn feature decoupling from a physical perspective: on the one hand, it continuously refines and tightens the modeling of normal backgrounds; on the other hand, it promotes sufficient mismatch (high escape rate) between anomalous features and the background model. This mechanism fundamentally distinguishes between background dynamics and real anomalies, effectively alleviating the problem of feature entanglement, thus achieving accurate frame-level anomaly detection even with only weak video-level supervision signals.

[0042] Referring to Figure 4, this invention also proposes a weakly supervised video anomaly detection system based on multi-head spectral residual gating. The system includes: an extraction module, used to: construct an anomaly detection model based on a feature extraction module, a multi-head manifold background estimation module, a spectral residual generation and gating module, and a spectral contrast energy optimization module; segment the input video using the feature extraction module and extract features using a pre-trained feature extractor to obtain a global feature sequence; an estimation module, used to: input the global feature sequence into the multi-head manifold background estimation module, segment it along the channel dimension into a predetermined number of subspace heads to obtain subspace features; construct a local autocorrelation matrix for each subspace feature, and estimate the background principal component direction corresponding to each subspace in the local autocorrelation matrix using a differentiable exponential iterative algorithm; and a spectral residual module, used to: project the subspace features onto the corresponding background principal component direction using the spectral residual generation and gating module to obtain reconstructed background features, and perform a cross-validation between the subspace features and the reconstructed background features. The residual features are calculated by comparing the differences, and the residual features of all subspaces are concatenated to obtain the spectral residual features. The spectral residual features are then input into the semantic awareness gating network in the spectral residual generation and gating module to generate a mask. The mask is used to enhance the global feature sequence to obtain the enhanced output features. The optimization module is used to: define the spectral escape rate based on the reconstructed background features and the spectral residual features; input the enhanced output features into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain the anomaly probability score of the video segment; construct the multi-instance learning classification loss based on the anomaly probability score of the video segment; construct the spectral contrast energy loss based on the spectral escape rate; jointly optimize the anomaly detection model using the spectral contrast energy loss and the multi-instance learning classification loss to obtain the optimized anomaly detection model; input the input video into the optimized anomaly detection model to obtain the final anomaly probability score of the video, and determine the final anomaly detection result based on the final anomaly probability score of the video.

[0043] To verify the effectiveness of this invention, extensive quantitative and qualitative experiments were conducted on several standard video anomaly detection datasets. These datasets include UCF-Crime and ShanghaiTech. UCF-Crime contains a large number of real-world surveillance videos with severe background noise (such as swaying trees and moving traffic), serving as a challenging benchmark for testing the "feature decoupling capability" of this invention. ShanghaiTech was used to verify the model's ability to capture specific motion anomalies in standard scenarios.

[0044] Under weak supervision, this invention uses the industry-standard frame-level AUC (Area Under the ROCCurve) as the primary evaluation metric and compares it with 10 existing (SOTA) methods from recent years (including 2024 and earlier), covering various approaches based on reconstruction, prediction, and multiple instance learning (MIL). Table 1 shows the performance comparison results (AUC / %) of this invention with the current state-of-the-art techniques on the UCF-Crime and ShanghaiTech datasets: Table 1 Performance Comparison with Existing Techniques (AUC / %)

[0045] Based on the experimental data analysis in Table 1, the following conclusions can be drawn: (1) On the highly challenging complex background dataset UCF-Crime, this invention achieved an AUC of 88.64%, breaking the best performance record in this field. Compared with traditional MIL methods (such as RTFM), this invention achieved a significant advantage (+4.34%). This proves that relying solely on classification boundary optimization is insufficient to solve the "feature entanglement" problem, while the strategy of explicitly decoupling anomalies from the dynamic background through "multi-head spectral residuals" in this invention is more effective.

[0046] (2) Superior to existing feature decoupling methods. Compared to the UR-DMU method, which attempts to separate features through clustering or memory modules, this invention demonstrates superior robustness. This indicates that using "differentiable exponential iteration" to dynamically estimate the background principal components can more accurately adapt to the real-time changing background distribution in the video stream than static cluster centers, avoiding the lag in background modeling.

[0047] (3) Comparable to or even surpassing large-scale multimodal models. Notably, MH-SRG's performance is comparable to methods like VadCLIP that rely on additional textual modal information. This invention achieves higher accuracy without introducing textual descriptions and complex prompt engineering. This demonstrates that optimizing from the underlying physical logic (spectral residuals and energy conservation) is an efficient way to improve the model's understanding of the nature of anomalies, and its inference efficiency is far higher than that of large multimodal models.

[0048] To verify the necessity and contribution of each module in this invention, a decomposition analysis was performed on the UCF-Crime dataset. Table 2 shows the ablation experiment results of the core modules: Table 2 Ablation Experiment Results

[0049] Table 2 clearly shows the independent contribution of each innovative module of the present invention: (1) The core role of spectral residual generation (SR): Compared with Experiment 2 and Experiment 3, the performance is significantly improved after the introduction of the spectral residual mechanism. It is believed that the model no longer implicitly learns the classification boundary, but infers based on the physical assumption that "anomalies cannot be explained by the principal components of the background", which effectively suppresses false alarms caused by dynamic background (such as cloud drift).

[0050] (2) Physical constraint effect of spectral contrast energy loss (SCE Loss): Experiment 4 shows that the model achieves the best performance after adding energy loss. This is because simple residual calculation may contain high-frequency noise, while the loss function based on energy conservation forces the energy of normal frames to be "absorbed" by the background subspace, while forcing the energy of abnormal frames to "escape", thus ensuring the purity of feature decoupling at the topological level.

[0051] Qualitative Analysis and Visualization: In addition to its quantitative performance advantages, this invention also possesses strong interpretability. To visually demonstrate how this invention solves the "feature entanglement" problem, we conducted a visualization analysis of the model's feature distribution and energy curves in typical scenarios: ① Semantic Decoupling: In the complex dynamic scenario of "traffic-intersection," the features extracted by the baseline model often confuse moving vehicles (normal) with red-light running behavior (abnormal). In contrast, the MH-SRG module of this invention successfully reconstructs the moving traffic flow into "background principal components," while retaining sudden red-light running vehicles in the "spectral residuals," achieving clear separation at the visual semantic level.

[0052] ② Spectral Escape Ratio Curve: At the instant an anomalous event (such as theft) occurs, the spectral escape ratio calculated in this invention exhibits a sharp spike. During normal periods, despite movement of people, the escape ratio remains low. This indicates that the model successfully establishes a "background gravitational field" based on energy conservation; only genuine anomalous events can generate sufficient energy to disrupt the equilibrium and allow escape, rather than simple motion detection.

[0053] In summary, the experimental results fully demonstrate that the weakly supervised video anomaly detection method based on multi-head spectral residual gating proposed in this invention can effectively utilize differentiable exponential iteration and spectral energy theory to break feature entanglement, achieving high-precision and highly interpretable video anomaly detection under weak supervision.

[0054] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0055] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0056] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A weakly supervised video anomaly detection method based on multi-head spectral residual gating, characterized in that, The method includes the following steps: Step 1: Construct an anomaly detection model based on a feature extraction module, a multi-head manifold background estimation module, a spectral residual generation and gating module, and a spectral contrast energy optimization module; use the feature extraction module to segment the input video and use a pre-trained feature extractor to extract features to obtain a global feature sequence; Step 2: Input the global feature sequence into the multi-head manifold background estimation module, and divide it into a predetermined number of subspace heads along the channel dimension to obtain subspace features; for each subspace feature, construct a local autocorrelation matrix, and use a differentiable exponential iterative algorithm to estimate the background principal component direction corresponding to each subspace in the local autocorrelation matrix; Step 3: Use the spectral residual generation and gating module to generate and gate an anomaly detection model based on a spectral contrast energy optimization module; Step 4: Calculate the background principal component direction corresponding to each subspace in the local autocorrelation matrix using a differentiable exponential iterative algorithm; Step 5: Calculate the background principal component direction corresponding to each subspace in the local autocorrelation matrix using a spectral residual generation and gating module. The difference generation and gating module projects the subspace features onto the corresponding background principal component directions to obtain reconstructed background features. It then calculates the residual features by subtracting the subspace features from the reconstructed background features, and concatenates the residual features from all subspaces to obtain spectral residual features. Step 4: The spectral residual features are input into the semantic-aware gating network in the spectral residual generation and gating module to generate a mask. The mask is used to enhance the global feature sequence, resulting in enhanced output features. Step 5: The spectral escape rate is defined based on the reconstructed background features and the spectral residual features. The enhanced output features are then input into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain the anomaly probability score of the video segment. A multi-instance learning classification loss is constructed based on the anomaly probability score of video clips; a spectral contrastive energy loss is constructed based on the spectral escape rate. The anomaly detection model is jointly optimized using spectral contrast energy loss and multi-instance learning classification loss to obtain an optimized anomaly detection model. The input video is then fed into the optimized anomaly detection model to obtain the final anomaly probability score of the video. The final anomaly detection result is determined based on the final anomaly probability score of the video.

2. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 1, characterized in that, In step 2, a local autocorrelation matrix is ​​constructed for each subspace feature, and the background principal component direction corresponding to each subspace in the local autocorrelation matrix is ​​estimated using a differentiable power-law iterative algorithm. Specifically, this includes the following steps: S201, constructing a local autocorrelation matrix for each subspace feature and calculating the covariance surrogate matrix for each subspace feature; S202, randomly initializing a vector; wherein the initialization vector matches the dimension of the covariance surrogate matrix; S203, performing a preset number of power-law iterations on the initialization vector, calculating through matrix multiplication to stretch the initialization vector towards the direction of the largest eigenvalue of the covariance surrogate matrix, obtaining an intermediate vector; S204, normalizing the intermediate vector to obtain the normalized vector for this iteration; S205, using the normalized vector for this iteration as input, repeating steps S203 and S204 until the maximum number of iterations is reached, obtaining the final iteration result, which is defined as the estimated background principal direction.

3. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 2, characterized in that, The global feature sequence is divided into a predetermined number of subspace heads along the channel dimension. The corresponding relationship in this process is as follows: ;in, Represents the global feature sequence. This indicates a splitting operation. This represents the feature tensor of the first spatial head segment. The total number of subspace headers is The feature tensor of the subspace head segmentation; in the steps of constructing the local autocorrelation matrix for each subspace feature and calculating the covariance surrogate matrix of each subspace feature, the corresponding relationship in the process is as follows: ;in, The covariance surrogate matrix representing the characteristics of the subspace. Indicates the first Characteristics of each subspace Indicates the index of the subspace. This indicates that a transpose operation is being performed. Indicates batch size is The subspace dimension is The set; in the step of performing a preset number of power iterations on the initialization vector, and calculating through matrix multiplication to stretch the initialization vector towards the direction of the largest eigenvalue of the covariance surrogate matrix to obtain the intermediate vector, the corresponding relationship in the process is as follows: ;in, Indicates the first The unnormalized intermediate vector in the next iteration Indicates the first The normalized vector in the next iteration; in the step of normalizing the intermediate vector to obtain the normalized vector in this iteration, the corresponding relationship exists as follows: ;in, Indicates the first The normalized vector in the next iteration Represents the numerically stable term. This represents the L2 norm of a vector.

4. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 3, characterized in that, In step 3, the subspace features are projected onto the corresponding background principal component directions using the spectral residual generation and gating module to obtain the reconstructed background features. The difference between the subspace features and the reconstructed background features is then calculated to obtain the residual features. Finally, the residual features of all subspaces are concatenated to obtain the spectral residual features. Specifically, the steps include: projecting subspace features onto the corresponding background principal component directions using the spectral residual generation and gating module to obtain projection scalars; reconstructing background features using projection scalars to obtain reconstructed background features; and subtracting the subspace features from the reconstructed background features to obtain residual features. The residual features of all subspace heads are concatenated and the original feature dimensions are restored to obtain the spectral residual features.

5. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 4, characterized in that, The subspace features are projected onto the corresponding background principal component directions to obtain the projection scalar. The relationship in this process is as follows: ;in, Represents a projected scalar. Indicates the estimated first The background main direction of each subspace Indicates batch size is The timing length is A set of features with a feature dimension of 1; in the step of reconstructing background features using projection scalars to obtain the reconstructed background features, the corresponding relationship in the process is as follows: ;in, Indicates the background features of the reconstruction. Indicates batch size is The timing length is Feature dimension is The set; in the step of subtracting the subspace features from the reconstructed background features to obtain the residual features, the corresponding relationship in the process is as follows: ;in, The residual features are represented by the following formula: In the steps of concatenating the residual features of all subspace heads and restoring the original feature dimensions to obtain the spectral residual features, the corresponding process relationships are as follows: ;in, Indicates spectral residual characteristics, The total number of subspace headers is The residual characteristics, Indicates batch size is The timing length is Feature dimension is A set of.

6. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 5, characterized in that, In step 4, the spectral residual features are input into the semantic-aware gating network in the spectral residual generation and gating module to generate a mask. The mask is then used to enhance the global feature sequence to obtain the enhanced output features. Specifically, this includes the following steps: inputting the spectral residual features into the first fully connected layer of the semantic-aware gating network in the spectral residual generation and gating module for dimensionality reduction, followed by layer normalization and ReLU activation to obtain an intermediate feature tensor; restoring the dimensionality of the intermediate feature tensor through the second fully connected layer in the semantic-aware gating network, and generating a mask using the Sigmoid function. The global feature sequence is weighted and enhanced using a mask to obtain the enhanced output features.

7. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 6, characterized in that, The spectral residual features are input into the first fully connected layer of the semantic awareness gating network in the spectral residual generation and gating module for dimensionality reduction, followed by layer normalization and ReLU activation to obtain the intermediate feature tensor. The corresponding relationship in this process is as follows: ;in, This represents the intermediate feature tensor obtained after processing through the first layer of linear transformation, layer normalization, and ReLU activation function. express Activation function Presentation layer normalization processing, This indicates the first fully connected layer. Let represent the first learnable parameter; in the steps of restoring the dimension of the intermediate feature tensor through the second fully connected layer in the semantically aware gating network and generating a mask using the Sigmoid function, the corresponding relationship in the process is as follows: ;in, Indicates the mask. This represents the Sigmoid function. This indicates the second fully connected layer. This represents the second learnable parameter. In the step of weighting and enhancing the global feature sequence using a mask to obtain the enhanced output features, the corresponding relationship in the process is as follows: ;in, This represents the enhanced output features. This indicates element-wise multiplication.

8. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 7, characterized in that, In step 5, the spectral escape rate is defined based on the reconstructed background features and the spectral residual features. The corresponding relationship in the process is as follows: ;in, Indicates the spectral escape rate. Indicates in Spectral residual characteristics at time step Indicates in Background features that are constantly being reconstructed.

9. The weakly supervised video anomaly detection method based on multi-head spectral residual gating according to claim 8, characterized in that, A spectral contrast energy loss is constructed based on spectral escape rate; whereby the spectral contrast energy loss includes energy compression loss for normal videos and energy expansion loss for abnormal videos; the energy compression loss for normal videos has the following corresponding relationship: ;in, This indicates the energy compression loss in normal video. This represents a collection of normal videos. Indicates the duration of the video sequence. Indicates the first In the video The spectral escape rate at time step 1; the energy expansion loss of the anomalous video, and the corresponding relationship are as follows: ;in, This indicates the energy expansion loss of abnormal videos. This represents a collection of abnormal videos. This indicates the calculation of the maximum value. Indicates the calculation of the first In the video The maximum value of the escape rate in the time spectrum. Indicates the boundary threshold.

10. A weakly supervised video anomaly detection system based on multi-head spectral residual gating, characterized in that, The system employs the weakly supervised video anomaly detection method based on multi-head spectral residual gating as described in any one of claims 1-9. The system includes: an extraction module, used to: construct an anomaly detection model based on a feature extraction module, a multi-head manifold background estimation module, a spectral residual generation and gating module, and a spectral contrast energy optimization module; segment the input video using the feature extraction module and extract features using a pre-trained feature extractor to obtain a global feature sequence; an estimation module, used to: input the global feature sequence into the multi-head manifold background estimation module, segment it along the channel dimension into a predetermined number of subspace heads to obtain subspace features; construct a local autocorrelation matrix for each subspace feature, and estimate the background principal component direction corresponding to each subspace in the local autocorrelation matrix using a differentiable exponential iterative algorithm; and a spectral residual module, used to: project the subspace features onto the corresponding background principal component direction using the spectral residual generation and gating module to obtain reconstructed background features, and correlate the subspace features with the reconstructed background features. The residual features are calculated by subtracting the background features, and the residual features of all subspaces are concatenated to obtain the spectral residual features. The spectral residual features are then input into the semantic-aware gating network in the spectral residual generation and gating module to generate a mask. The mask is used to enhance the global feature sequence to obtain the enhanced output features. The optimization module is used to: define the spectral escape rate based on the reconstructed background features and the spectral residual features; input the enhanced output features into the multi-instance learning classifier in the spectral contrast energy optimization module to obtain the anomaly probability score of the video segment; construct the multi-instance learning classification loss based on the anomaly probability score of the video segment; construct the spectral contrast energy loss based on the spectral escape rate; jointly optimize the anomaly detection model using the spectral contrast energy loss and the multi-instance learning classification loss to obtain the optimized anomaly detection model; input the input video into the optimized anomaly detection model to obtain the final anomaly probability score of the video, and determine the final anomaly detection result based on the final anomaly probability score of the video.

Images