A Weakly Supervised Video Anomaly Detection Method and System Based on Wolde Decomposition

Abstract

This invention proposes a weakly supervised video anomaly detection method and system based on Wald decomposition. It utilizes a pre-trained feature extractor to extract visual features from video segments; based on Wald decomposition theory, it explicitly decomposes the video feature sequence into deterministic components, harmonic components, and singular components, corresponding to the long-term trend, periodic regularity, and sudden unpredictable disturbances of the video, respectively; it employs an adaptive reconstruction mechanism to perform weighted fusion of each component; it constructs spectral energy distribution and spectral entropy from a frequency domain perspective to characterize the degree of spectral disorder, and uses a sequence prediction model from a temporal domain perspective to measure temporal predictability; and it generates the final frame-level anomaly score through multi-source anomaly evidence fusion. This invention explicitly models the video temporal structure at the generation mechanism level, effectively mitigating the interference of pseudo-anomaly instances, and achieves accurate weakly supervised video anomaly localization with a lightweight model containing only 7.5M parameters.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and video analysis technology, and in particular to a weakly supervised video anomaly detection method and system based on Wold decomposition. Background Technology

[0002] Video anomaly detection (VAD) aims to automatically identify anomalous events that deviate from normal behavioral patterns in long video sequences, and it has significant value in fields such as public safety monitoring and intelligent transportation. Due to the scarcity of anomalous samples in real-world scenarios and the high cost of annotation, weakly-supervised video anomaly detection (VAD) has become a research hotspot. This method relies solely on video-level labels (normal / abnormal) for training, yet requires precise frame-level localization.

[0003] Most existing weakly supervised methods are based on the multiple instance learning (MIL) framework, treating anomalies as a small number of "high-response" or "difficult-to-match" fragments, and learning them through feature similarity or attention mechanisms. However, these methods have significant limitations:

[0004] (1) It ignores the inherent generation structure of video data as a time series, lacks explicit characterization of the time structure, and makes it difficult to capture the disruption of long-term trends or periodic patterns by anomalies.

[0005] (2) Under video-level supervision only, the model is easily affected by noisy frames (pseudo-anomalies), which leads to overfitting to local noise and reduces the robustness and interpretability of the detection.

[0006] (3) Most methods still remain at the level of feature response and lack in-depth modeling of normal and abnormal time sequence mechanisms from the perspective of generation mechanism. Summary of the Invention

[0007] In view of the above, the main objective of this invention is to propose a weakly supervised video anomaly detection method and system based on Wold decomposition to solve the aforementioned technical problems.

[0008] This invention proposes a weakly supervised video anomaly detection method based on Wold decomposition, the method comprising the following steps:

[0009] Step 1: Divide the input video into segments, and use a feature extraction network to extract the visual features of the video segments to obtain a video feature sequence;

[0010] Step 2: Based on the Wald decomposition theory, the video feature sequence is explicitly decomposed into three types of time components: the three types of time components include deterministic components, harmonic components, and singular components;

[0011] Step 3: Perform adaptive weighted reconstruction on the three types of time components to obtain the reconstructed feature sequence;

[0012] Step 4: From the perspective of the frequency domain, perform spectral analysis on the time component, calculate the spectral energy distribution, and use the spectral entropy of the spectral energy distribution as the frequency domain anomaly score to characterize the spectral disorder caused by the anomaly.

[0013] Step 5: From a temporal perspective, use a sequence prediction model to predict future frames of the time components, obtain future frame features, and calculate the predictability score based on the error between the future frame features and the real sequence to measure the disruption of the time structure by anomalies.

[0014] Step 6: Perform frame-level discrimination on the reconstructed feature sequence to obtain the frame-level discrimination score; define the inverse measure of the predictability score as the temporal anomaly score; fuse the frame-level discrimination score, the frequency domain anomaly score, and the temporal anomaly score to obtain the frame-level anomaly score.

[0015] Based on the frame-level anomaly score, a weakly supervised multi-instance learning framework is used for training. After training, the final frame-level anomaly score is obtained.

[0016] This invention also proposes a weakly supervised video anomaly detection system based on Wolde decomposition, wherein the system applies the weakly supervised video anomaly detection method based on Wolde decomposition as described above, and the system includes a feature extraction module, a Wolde decomposition and reconstruction module, a multi-view measurement module, and an optimization and scoring module.

[0017] The feature extraction module includes a video segmentation unit and a pre-trained feature extractor;

[0018] The video segmentation unit is used to divide the input video into segments;

[0019] A pre-trained feature extractor is used to extract visual features from video clips to obtain a video feature sequence;

[0020] The Wold decomposition and reconstruction module includes deterministic / harmonic / singular decomposition units, adaptive weight generation units, and component reconstruction units;

[0021] The deterministic / harmonic / singular decomposition unit is used to explicitly decompose video feature sequences into three types of time components based on the Wald decomposition theory: deterministic components, harmonic components, and singular components.

[0022] An adaptive weight generation unit is used to generate corresponding adaptive weights based on the three types of time components;

[0023] The component reconstruction unit is used to perform adaptive weighted reconstruction of the three types of time components according to adaptive weights to obtain the reconstructed feature sequence;

[0024] The multi-view measurement module includes a frequency domain spectral entropy calculation unit, a time domain predictability measurement unit, and a frame-level discrimination and scoring unit;

[0025] The frequency domain spectral entropy calculation unit is used to perform spectral analysis on the time component from the perspective of the frequency domain, calculate the spectral energy distribution, and use the spectral entropy of the spectral energy distribution as the frequency domain anomaly score.

[0026] The temporal predictability measurement unit is used to predict future frames of time components from a temporal perspective using a sequence prediction model, obtain future frame features, and calculate the predictability score based on the error between the future frame features and the real sequence.

[0027] The frame-level discrimination and scoring unit is used to perform frame-level discrimination on the reconstructed feature sequence and obtain a frame-level discrimination score.

[0028] The optimization and scoring module includes a multi-source evidence fusion unit and a weakly supervised MIL training unit;

[0029] The multi-source evidence fusion unit is used to define the inverse metric of the predictability score as the temporal anomaly score; and to fuse the frame-level discrimination score, frequency domain anomaly score, and temporal anomaly score to obtain the frame-level anomaly score.

[0030] The weakly supervised MIL training unit is used to train the frame-level anomaly score based on the weakly supervised multi-instance learning framework. After training, the final frame-level anomaly score is obtained.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] 1. This invention introduces Wald decomposition into the weakly supervised video anomaly detection task for the first time. From the perspective of time series generation mechanism, it explicitly decouples complex video features into deterministic, harmonic and singular components, providing a stable and physically interpretable structural prior for anomaly detection.

[0033] 2. This invention proposes a complementary time-frequency dual-view measurement method. By characterizing the energy dispersion in the frequency domain through spectral entropy and measuring structural damage in the time domain through predictability, it effectively solves the problem that a single viewpoint is insufficient to fully capture anomalous features.

[0034] 3. This invention designs a multi-source evidence adaptive fusion mechanism, which effectively suppresses noise interference from pseudo-anomaly instances under weak supervision, achieving stable and accurate anomaly localization. Experiments show that this invention significantly improves detection accuracy while maintaining a lightweight model (only 7.5M parameters and a model size of 28.84M).

[0035] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0036] Figure 1 This is a flowchart of the weakly supervised video anomaly detection method based on Wold decomposition proposed in this invention;

[0037] Figure 2 This is a schematic diagram of the structure of the World decomposition, reconstruction and anomaly measurement module of the present invention;

[0038] Figure 3 This is a schematic diagram of the structure of the weakly supervised video anomaly detection system based on Wold decomposition proposed in this invention. Detailed Implementation

[0039] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0040] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0041] Please see Figure 1 This embodiment provides a weakly supervised video anomaly detection method based on Wold decomposition, the method comprising the following steps:

[0042] Step 1: Divide the input video into segments, and use a feature extraction network to extract the visual features of the video segments to obtain a video feature sequence;

[0043] In step 1, the input video is segmented, and a feature extraction network is used to extract the visual features of the video segments to obtain a video feature sequence. Specifically, this includes the following steps:

[0044] The input raw video stream is divided into T non-overlapping video segments. In this embodiment, each segment is set to contain 16 consecutive frames.

[0045] A feature extraction network pre-trained on a large-scale action recognition dataset (such as Kinetics-400 or Action30) is used as the feature extractor. CLIP, I3D, or ViT can be used to extract features; this embodiment uses CLIP.

[0046] Each video segment is input into the feature extractor, which extracts a C-dimensional feature vector; after the feature extraction of the entire video is completed, a video feature sequence is obtained.

[0047] Step 2: Based on the Wald decomposition theory, the video feature sequence is explicitly decomposed into three types of temporal components: deterministic components, harmonic components, and singular components (see [link to relevant documentation]). Figure 2 );

[0048] In a preferred embodiment of the present invention, the deterministic component is used to model the long-term stable behavior pattern of video features that change slowly over time, characterizing the long-term stable trend of the video that changes slowly over time (such as background gradation). Its calculation process is implemented through an adaptive moving average operator, and the corresponding process has the following relationship:

[0049] ;

[0050] in, Describes the deterministic components of frame t. Indicates the video length. This represents the frame index for the summation traversal. Indicates the target frame currently being calculated. This represents the original features of the i-th frame in the video feature sequence. Indicates weight, when hour, Otherwise, it is 0. This indicates the window size, which is adaptively determined based on the effective length of the video. This indicates that the floor function is being performed.

[0051] In a preferred embodiment of the present invention, harmonic components are used to model periodic structures in video features, characterizing behavioral structures with significant periodicity in the video (such as regular mechanical motion or traffic signal cycles). The corresponding calculation process includes:

[0052] The feature sequence is transformed along the time dimension to the frequency domain (using discrete cosine transform, fast Fourier transform, or wavelet transform) to obtain frequency domain coefficients. The first K low-frequency harmonic coefficients are retained and inversely transformed to reconstruct the harmonic components. In this embodiment, the transformation method used is discrete cosine transform, and K is set to 5.

[0053] In a preferred embodiment of the present invention, singular components are used to capture sudden changes that cannot be explained by trends or periods, and to characterize random sudden disturbances that cannot be explained by trends or periods. Singular components are defined as the difference between the video feature sequence and the deterministic and harmonic components, and the corresponding process has the following relationship:

[0054] ;

[0055] in, Indicates exotic components, Indicates harmonic components, Indicates a definite component. This represents the features of the t-th frame in the video feature sequence.

[0056] Step 3: Perform adaptive weighted reconstruction on the three types of time components to obtain the reconstructed feature sequence;

[0057] As a preferred embodiment of the present invention, considering that different types of anomalies (such as sudden falls, fights, or vehicles driving in the wrong direction) have different expressive power on different time components, the present invention introduces learnable weights to reconstruct the components. Adaptive weighted reconstruction is performed on the three types of time components to obtain the reconstructed feature sequence, specifically including the following steps:

[0058] Each time component is mapped using a lightweight network, and then the Softmax function is used to obtain the adaptive weights for the corresponding time components.

[0059] By dynamically adjusting the adaptive weights of the corresponding time components, adaptive reconstructed features are obtained. The corresponding process has the following relationship:

[0060] ;

[0061] in, This represents the adaptive reconstruction features of frame t. This represents the adaptive weights corresponding to the c-th time component, and , This represents the c-th type of time component in the t-th frame. Indicates the component index. , These represent the corresponding deterministic component, harmonic component, and singular component, respectively.

[0062] Step 4: From a frequency domain perspective, perform spectral analysis on the time component, calculate the spectral energy distribution, and use the spectral entropy of the spectral energy distribution as the frequency domain anomaly score to characterize the spectral disturbance caused by the anomaly (see [link to relevant documentation]). Figure 2 );

[0063] In a preferred embodiment of the present invention, spectral analysis of the time component is performed from a frequency domain perspective to calculate the spectral energy distribution, and the spectral entropy of the spectral energy distribution is used as the frequency domain anomaly score. Specifically, the steps include:

[0064] Perform a Fast Fourier Transform (FFT) on each time component to obtain the spectrum of the corresponding time component;

[0065] Calculate the modulus of the spectrum of the time component to obtain the amplitude spectrum of the corresponding time component. The corresponding process has the following relationship:

[0066] ;

[0067] in, Represents the spectrum of the c-th time component. Represents the amplitude spectrum of the c-th type of time component;

[0068] The amplitude spectrum is normalized to obtain the first... l The energy percentage of each frequency component corresponds to the following relationship in the process:

[0069] ;

[0070] in, Indicates the first l The energy percentage of each frequency component Represents the first time component of type c. l The amplitude of each frequency component This represents the amplitude of the j-th frequency component of the c-th time component;

[0071] Based on the energy proportion of the i-th frequency component, the spectral entropy is calculated using the information entropy formula, and the spectral entropy is used as the frequency domain anomaly score. The corresponding process has the following relationship:

[0072] ;

[0073] in, This represents the spectral entropy of the c-th time component. The larger the spectral entropy, the more dispersed the spectral energy distribution, and the higher the corresponding anomaly probability.

[0074] Step 5: From a temporal perspective, use a sequence prediction model to predict future frames of the temporal components, obtain future frame features, and calculate a predictability score based on the error between the future frame features and the true sequence to measure the disruption of the temporal structure by anomalies (e.g., Figure 2 (as shown)

[0075] In a preferred embodiment of the present invention, from a temporal perspective, a sequence prediction model is used to predict future frames of the time component to obtain future frame features. A predictability score is then calculated based on the error between the future frame features and the actual sequence. The specific steps include the following:

[0076] A sequence prediction model is constructed. In this embodiment, a recurrent neural network based on gated recurrent units (GRU) is used to predict the features of the next H time steps using the time components before the current time t, so as to obtain the predicted features of the next H frames.

[0077] The error between the predicted features of future H frames and the corresponding real sequence is calculated to obtain the error of each time component. The corresponding process has the following relationship:

[0078] ;

[0079] in, This represents the prediction error of the c-th time component. Indicates the time step of the prediction. Denotes the square of the L2 norm. This represents the true characteristics of the c-th type of time component in the future t+i frame. This represents the predicted feature of the c-th time component in the future t+i-th frame;

[0080] Mapping the error of each time component to a predictability score yields the predictability score for each time component. The corresponding process follows the following relationship:

[0081] ;

[0082] in, This represents the predictability score of the c-th time component. Indicates the scaling factor. It represents an exponential function with the natural constant e as its base. The lower the predictability, the greater the probability of an anomaly.

[0083] Step 6: Perform frame-level discrimination on the reconstructed feature sequence to obtain the frame-level discrimination score; define the inverse measure of the predictability score as the temporal anomaly score; fuse the frame-level discrimination score, the frequency domain anomaly score, and the temporal anomaly score to obtain the frame-level anomaly score.

[0084] Based on the frame-level anomaly score, a weakly supervised multi-instance learning framework is used for training. After training, the final frame-level anomaly score is obtained.

[0085] In a preferred embodiment of the present invention, to achieve accurate anomaly localization, clues from different perspectives are fused. For each frame t, three types of evidence are collected: frame-level discrimination score, frequency domain anomaly score, and time domain anomaly score. By concatenating the frame-level discrimination score, frequency domain anomaly score, and time domain anomaly score, and then applying a fusion function, the frame-level anomaly score is obtained. The corresponding process follows the following relationship:

[0086] ;

[0087] in, This represents a fusion function composed of fully connected networks. This represents the frame-level discrimination score. Indicates the frequency domain anomaly score. This indicates the score for time-domain anomalies.

[0088] Based on the frame-level anomaly scores, a weakly supervised multi-instance learning framework is used for training. After training, the final frame-level anomaly scores are obtained. The specific steps are as follows:

[0089] Because only video-level tags are available ( (where 0 represents normal videos and 1 represents abnormal videos), and the training uses a multiple instance learning (MIL) strategy.

[0090] 1. Top-k sampling: For anomalous videos, the average of the k segments with the highest anomalous scores is selected as the predicted score of the video; for normal videos, the segment with the highest score is selected as the predicted score of the video.

[0091] 2. Loss Calculation: Construct a classification loss function (such as binary cross-entropy loss) and combine it with spectral entropy constraints and prediction error constraints to perform end-to-end joint optimization of network parameters.

[0092] 3. Parameter description: In this embodiment, the total number of model parameters is controlled at around 7.5M, and the model file size is about 28.84M, achieving lightweight deployment while ensuring high accuracy.

[0093] Please refer to Figure 3 This embodiment also provides a weakly supervised video anomaly detection system based on Wold decomposition, wherein the system applies the weakly supervised video anomaly detection method based on Wold decomposition as described above, and the system includes a feature extraction module, a Wold decomposition and reconstruction module, a multi-view measurement module, and an optimization and scoring module.

[0094] The feature extraction module includes a video segmentation unit and a pre-trained feature extractor;

[0095] The video segmentation unit is used to divide the input video into segments;

[0096] A pre-trained feature extractor is used to extract visual features from video clips to obtain a video feature sequence;

[0097] The Wold decomposition and reconstruction module includes deterministic / harmonic / singular decomposition units, adaptive weight generation units, and component reconstruction units;

[0098] The deterministic / harmonic / singular decomposition unit is used to explicitly decompose video feature sequences into three types of time components based on the Wald decomposition theory: deterministic components, harmonic components, and singular components.

[0099] An adaptive weight generation unit is used to generate corresponding adaptive weights based on the three types of time components;

[0100] The component reconstruction unit is used to perform adaptive weighted reconstruction of the three types of time components according to adaptive weights to obtain the reconstructed feature sequence;

[0101] The multi-view measurement module includes a frequency domain spectral entropy calculation unit, a time domain predictability measurement unit, and a frame-level discrimination and scoring unit;

[0102] The frequency domain spectral entropy calculation unit is used to perform spectral analysis on the time component from the perspective of the frequency domain, calculate the spectral energy distribution, and use the spectral entropy of the spectral energy distribution as the frequency domain anomaly score.

[0103] The temporal predictability measurement unit is used to predict future frames of time components from a temporal perspective using a sequence prediction model, obtain future frame features, and calculate the predictability score based on the error between the future frame features and the real sequence.

[0104] The frame-level discrimination and scoring unit is used to perform frame-level discrimination on the reconstructed feature sequence and obtain a frame-level discrimination score.

[0105] The optimization and scoring module includes a multi-source evidence fusion unit and a weakly supervised MIL training unit;

[0106] The multi-source evidence fusion unit is used to define the inverse metric of the predictability score as the temporal anomaly score; and to fuse the frame-level discrimination score, frequency domain anomaly score, and temporal anomaly score to obtain the frame-level anomaly score.

[0107] The weakly supervised MIL training unit is used to train the frame-level anomaly score based on the weakly supervised multi-instance learning framework. After training, the final frame-level anomaly score is obtained.

[0108] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned weakly supervised video anomaly detection based on Wold decomposition.

[0109] To verify the effectiveness and advancement of this invention, the inventors conducted extensive quantitative and qualitative experiments on multiple standard video anomaly detection datasets. These datasets include UCF-Crime and ShanghaiTech. UCF-Crime contains a large number of real surveillance videos with severe background noise (such as weather changes and lighting fluctuations), serving as a challenging benchmark for testing the invention's "temporal structure anti-interference capability." ShanghaiTech was used to verify the model's ability to capture specific sudden anomalies in standard scenarios.

[0110] Under weak supervision, this invention uses the industry-standard frame-level AUC (Area Under the ROCCurve) as the primary evaluation metric and compares it with mainstream state-of-the-art (SOTA) methods in recent years (covering multi-instance learning, reconstruction, and prediction approaches). Experimental results show that this invention achieves significant advantages in both performance and efficiency.

[0111] 1. Quantitative comparison with existing technologies:

[0112] (1) Significant improvement in detection accuracy: On the highly challenging UCF-Crime dataset, the method of this invention significantly outperforms the traditional multi-instance learning (MIL) method. This proves that relying solely on "feature response amplitude" for classification is insufficient to handle complex scenarios. The strategy of this invention, which introduces Wold decomposition to explicitly separate anomalies from long-term trends and harmonic patterns, is more effective and significantly reduces the false alarm rate caused by environmental noise (such as swaying trees and cloud movement).

[0113] (2) Extreme Lightweight and High Efficiency: Compared with current popular methods based on large vision-language models (such as various variants using CLIP features), this invention achieves extreme lightweighting while maintaining state-of-the-art accuracy. The model of this invention has only 7.5M parameters and a model file size of only 28.84M. This means that this invention does not rely on a large external knowledge base or expensive computing resources. It can achieve efficient inference with extremely low computational cost simply by physically modeling the internal time generation mechanism of the video, making it highly valuable for practical deployment.

[0114] (3) Generalization of different types of anomalies: The invention also performs well on the ShanghaiTech dataset. This is due to the accurate fitting of harmonic components to periodic behaviors (such as walking and cycling), which makes sudden non-periodic anomalies (such as fighting and running) stand out more in the singular components.

[0115] 2. Ablation experiment of core module:

[0116] To verify the necessity and contribution of each core module (Wolde decomposition structure, frequency domain spectral entropy modeling, and time domain predictability modeling) in this invention, a decomposition analysis was performed on a benchmark dataset. The experimental results clearly demonstrate the independent contribution of each innovative module:

[0117] (1) The core role of Wold Decomposition: Compared with the baseline model (which directly performs anomaly detection on the overall features), the detection performance is greatly improved after Wold Decomposition is introduced to explicitly decouple the features into deterministic, harmonic, and singular components. The analysis suggests that this decomposition provides the model with a strong structural prior, forcing the model to learn to distinguish between "long-term background evolution" (deterministic components) and "sudden anomalous perturbations" (singular components), rather than simply overfitting local features.

[0118] (2) Discriminative efficacy of frequency domain spectral entropy: After introducing frequency domain modeling based on spectral energy distribution, the model's sensitivity to unstructured anomalies is significantly increased. Experimental data show that spectral entropy can effectively quantify the degree of spectral disorder when anomalies occur, making up for the shortcomings of time domain analysis in dealing with high-frequency noise.

[0119] (3) Complementarity of time-domain predictability: After adding the time-domain predictability score, the model performs better in capturing abnormal actions such as sudden falls. This proves the validity of the hypothesis that "normal behavior is highly predictable and abnormal behavior disrupts time dependence".

[0120] (4) Robustness of multi-source evidence fusion: The complete model achieves optimal performance by adaptively fusing three types of evidence: frame-level discrimination, frequency domain spectral entropy, and temporal predictability. This indicates that anomaly cues from different perspectives are highly complementary, and joint modeling effectively alleviates the problem of pseudo-label noise in weakly supervised environments.

[0121] 3. Qualitative analysis and visualization:

[0122] In addition to its quantitative performance advantages, this invention also possesses strong interpretability. To visually demonstrate how this invention utilizes time structure to model anomalies, a visualization analysis was conducted on the model's decomposition effects and scoring curves in typical scenarios:

[0123] (1) Structural Decoupling: In a video containing a sudden shoving incident during a regular crowd movement, the visualization results show that the deterministic component of this invention successfully fits the slow changes in the background environment, the harmonic component captures the periodic rhythm of the crowd's movement, and at the moment the shoving occurs, its energy is almost entirely concentrated in the singular component. This physical decoupling allows the anomalous signal to be significantly "purified" in the singular component, achieving a clear separation from the background noise.

[0124] (2) Time-Frequency Scoring Curve:

[0125] ① Frequency domain perspective: When an anomaly occurs, the spectral entropy curve fluctuates violently, indicating that the characteristics at that moment become disordered in the frequency domain and the energy distribution is no longer concentrated.

[0126] ② Temporal perspective: The predictability score drops sharply at anomalous moments, indicating that the behavior at that moment cannot be effectively inferred from historical sequences.

[0127] ③ Comprehensive judgment: The final anomaly score curve forms a peak response with an extremely high signal-to-noise ratio in the anomaly range, while it remains low in the normal time period (even if there are changes in lighting or background movement).

[0128] In summary, the experimental results fully demonstrate that the weakly supervised video anomaly detection method based on Wold decomposition proposed in this invention can effectively utilize the time series generation mechanism to achieve high-precision, high-robustness, and well-physically interpretable video anomaly localization with extremely low parameter count (7.5M).

[0129] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A weakly supervised video anomaly detection method based on Wold decomposition, characterized in that, The method includes the following steps: Step 1: Divide the input video into segments and use a feature extraction network to extract the visual features of the video segments to obtain a video feature sequence. This includes the following steps: Divide the input raw long video stream into T non-overlapping video segments; A feature extraction network pre-trained on a large-scale action recognition dataset is used as a feature extractor; Each video segment is input into a feature extractor, which extracts a C-dimensional feature vector; after the feature extraction of the entire video is completed, a video feature sequence is obtained. Step 2: Based on the Wald decomposition theory, the video feature sequence is explicitly decomposed into three types of time components: the three types of time components include deterministic components, harmonic components, and singular components; Harmonic components are used to model the periodic structure in video features. The corresponding calculation process includes: transforming the time component along the time dimension to the frequency domain to obtain the frequency domain coefficients, retaining the first K low-frequency harmonic coefficients and performing an inverse transformation to reconstruct the harmonic components. The deterministic component is used to model long-term stable behavioral patterns of video features that change slowly over time. Its calculation is achieved through an adaptive moving average operator, and the corresponding process has the following relationship: ； in, Describes the deterministic components of frame t. Indicates the video length. This represents the frame index for the summation traversal. Indicates the target frame currently being calculated. This represents the original features of the i-th frame in the video feature sequence; Indicates weight, when hour, Otherwise, it is 0; This indicates the window size, which is adaptively determined based on the effective length of the video. This indicates that the floor function is being performed; Singular components are used to capture sudden changes that cannot be explained by trends or cycles. Singular components are defined as the difference between the video feature sequence and the deterministic and harmonic components, and the corresponding process has the following relationship: ； in, Indicates exotic components, Indicates harmonic components, Indicates a definite component. This represents the features of the t-th frame in the video feature sequence; Step 3: Perform adaptive weighted reconstruction on the three types of time components to obtain the reconstructed feature sequence, which includes the following steps: Each time component is mapped using a lightweight network, and then the Softmax function is used to obtain the adaptive weights for the corresponding time components. By dynamically adjusting the adaptive weights of the corresponding time components, adaptive reconstructed features are obtained. The corresponding process has the following relationship: ； in, Represents the adaptive reconstruction features of frame t; This represents the adaptive weight corresponding to the c-th time component, and ; This represents the c-th type of time component in the t-th frame. Indicates the component index. , These represent the corresponding deterministic component, harmonic component, and singular component, respectively. Step 4: From the perspective of the frequency domain, perform spectral analysis on the time component, calculate the spectral energy distribution, and use the spectral entropy of the spectral energy distribution as the frequency domain anomaly score. Step 5: From a temporal perspective, use a sequence prediction model to predict future frames of the time components to obtain future frame features. Calculate the predictability score based on the error between the future frame features and the actual sequence. Step 6: Perform frame-level discrimination on the reconstructed feature sequence to obtain the frame-level discrimination score; define the inverse measure of the predictability score as the temporal anomaly score; fuse the frame-level discrimination score, the frequency domain anomaly score, and the temporal anomaly score to obtain the frame-level anomaly score. Based on the frame-level anomaly score, a weakly supervised multi-instance learning framework is used for training. After training, the final frame-level anomaly score is obtained.

2. The weakly supervised video anomaly detection method based on Wold decomposition according to claim 1, characterized in that, In step 4, from a frequency domain perspective, spectral analysis is performed on the time component to calculate the spectral energy distribution, and the spectral entropy of the spectral energy distribution is used as the frequency domain anomaly score. Specifically, this includes the following steps: Perform a fast Fourier transform on each time component to obtain the spectrum of the corresponding time component; Calculate the modulus of the spectrum of the time component to obtain the amplitude spectrum of the corresponding time component. The corresponding process has the following relationship: ； in, Represents the spectrum of the c-th time component. Represents the amplitude spectrum of the c-th type of time component; The amplitude spectrum is normalized to obtain the first... l The energy percentage of each frequency component corresponds to the following relationship in the process: ； in, Indicates the first l The energy percentage of each frequency component Represents the first time component of type c. l The amplitude of each frequency component This represents the amplitude of the j-th frequency component of the c-th time component; According to the l The energy proportion of each frequency component is used to calculate the spectral entropy using the information entropy formula, and the spectral entropy is used as the frequency domain anomaly score. The corresponding process has the following relationship: ； in, This represents the spectral entropy of the c-th time component.

3. The weakly supervised video anomaly detection method based on Wold decomposition according to claim 2, characterized in that, In step 5, from a temporal perspective, a sequence prediction model is used to predict future frames of the time component to obtain future frame features. A predictability score is then calculated based on the error between the future frame features and the actual sequence. Specifically, this includes the following steps: Construct a sequence prediction model, use the time components before the current time t to predict the features of the next H time steps, and obtain the predicted features of the next H frames; The error between the predicted features of future H frames and the corresponding real sequence is calculated to obtain the error of each time component. The corresponding process has the following relationship: ； in, This represents the prediction error of the c-th time component. Indicates the time step of the prediction. Denotes the square of the L2 norm. This represents the true characteristics of the c-th type of time component in the future t+i frame. This represents the predicted feature of the c-th time component in the future t+i-th frame; Mapping the error of each time component to a predictability score yields the predictability score for each time component. The corresponding process follows the following relationship: ； in, This represents the predictability score of the c-th time component. Indicates the scaling factor. This represents an exponential function with the natural constant e as its base.

4. The weakly supervised video anomaly detection method based on Wold decomposition according to claim 3, characterized in that, In step 6, the frame-level discrimination score, frequency domain anomaly score, and time domain anomaly score are fused to obtain the frame-level anomaly score. The corresponding process has the following relationship: ； in, This represents a fusion function composed of fully connected networks. This represents the frame-level discrimination score. Indicates the frequency domain anomaly score. This indicates the score for time-domain anomalies.

5. A weakly supervised video anomaly detection system based on Wold decomposition, characterized in that, The system applies the weakly supervised video anomaly detection method based on Wold decomposition as described in any one of claims 1 to 4. The system includes a feature extraction module, a Wold decomposition and reconstruction module, a multi-view measurement module, and an optimization and scoring module. The feature extraction module includes a video segmentation unit and a pre-trained feature extractor; The video segmentation unit is used to divide the input video into segments; A pre-trained feature extractor is used to extract visual features from video clips to obtain a video feature sequence; The Wold decomposition and reconstruction module includes deterministic / harmonic / singular decomposition units, adaptive weight generation units, and component reconstruction units; The deterministic / harmonic / singular decomposition unit is used to explicitly decompose video feature sequences into three types of time components based on the Wald decomposition theory: deterministic components, harmonic components, and singular components. An adaptive weight generation unit is used to generate corresponding adaptive weights based on the three types of time components; The component reconstruction unit is used to perform adaptive weighted reconstruction of the three types of time components according to adaptive weights to obtain the reconstructed feature sequence; The multi-view measurement module includes a frequency domain spectral entropy calculation unit, a time domain predictability measurement unit, and a frame-level discrimination and scoring unit; The frequency domain spectral entropy calculation unit is used to perform spectral analysis on the time component from the perspective of the frequency domain, calculate the spectral energy distribution, and use the spectral entropy of the spectral energy distribution as the frequency domain anomaly score. The temporal predictability measurement unit is used to predict future frames of time components from a temporal perspective using a sequence prediction model, obtain future frame features, and calculate the predictability score based on the error between the future frame features and the real sequence. The frame-level discrimination and scoring unit is used to perform frame-level discrimination on the reconstructed feature sequence and obtain a frame-level discrimination score. The optimization and scoring module includes a multi-source evidence fusion unit and a weakly supervised MIL training unit; A multi-source evidence fusion unit is used to define the inverse metric of predictability score as a temporal anomaly score; The frame-level discrimination score, frequency domain anomaly score, and time domain anomaly score are fused to obtain the frame-level anomaly score; The weakly supervised MIL training unit is used to train the frame-level anomaly score based on the weakly supervised multi-instance learning framework. After training, the final frame-level anomaly score is obtained.

Images