CLIP-assisted time sequence adaptive filtering weak supervision video anomaly detection method

Abstract

The invention belongs to the technical field of computer vision and video analysis, and particularly relates to a time sequence adaptive filtering weak supervision video anomaly detection method based on CLIP assistance. According to the method, a time sequence modeling and filtering framework fusing vision-language cross-modal information is designed, and the method comprises the steps that firstly, video local and global time sequence dependence is decomposed and modeled through a local-global time sequence adapter, and the capturing capacity of a model for the dynamic evolution process of abnormal events is enhanced; secondly, a double-branch architecture is adopted, pure visual features are used for completing coarse-grained anomaly detection, fine-grained anomaly recognition is achieved in combination with visual-language alignment features, and detection requirements of different complexity scenes are met; and finally, introducing a Fourier transform time sequence filter, adaptively suppressing noise in a frequency domain, strengthening abnormal feature representation, and improving the anti-interference capability of the model. The method is remarkably superior to an existing weak supervision method, and the precision and robustness of anomaly detection can be effectively improved through the synergistic effect of the modules.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and video analysis technology, specifically relating to a CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method, which can be applied to scenarios such as intelligent monitoring, public safety early warning, and traffic violation identification, to achieve efficient and accurate detection of abnormal events in videos. Background Technology

[0002] Video anomaly detection, a core function of intelligent surveillance systems, aims to identify events that deviate from normal behavioral patterns (such as violent attacks, theft, wrong-way traffic, and facility malfunctions) from continuous video sequences. It is a crucial technological support for fields such as public safety, traffic management, and industrial monitoring. With the widespread adoption of surveillance equipment and the explosive growth of video data, the demand for automated, high-precision video anomaly detection methods is becoming increasingly urgent.

[0003] Traditional video anomaly detection methods are mainly divided into two categories: unsupervised and fully supervised. 1) Unsupervised methods: These rely on generative models (such as autoencoders and generative adversarial networks) to learn the feature distribution of normal behavior and classify samples with high reconstruction errors as anomalies. However, these methods rely too heavily on the assumption that "normal data covers all patterns," making them prone to misjudgment when faced with unseen normal behavior and failing to capture the dynamic features of complex anomalies. 2) Fully supervised methods: These require labeling the spatiotemporal location of anomalies frame by frame, resulting in extremely high labeling costs. Furthermore, in real-world scenarios, anomalies are sparse (e.g., violence, theft, etc. have low occurrence rates), making it difficult to construct large-scale labeled datasets and limiting generalization capabilities.

[0004] To balance annotation costs and detection performance, weakly supervised video anomaly detection (WSVAD) has become a research hotspot. It requires only video-level binary labels ("normal" or "abnormal") to train the model, significantly reducing annotation costs. In recent years, vision-language pre-trained models (such as CLIP) have provided a new direction for WSVAD due to their cross-modal semantic alignment capabilities. CLIP's general visual features learned from massive amounts of image and text data can be transferred to anomaly detection tasks, improving the model's adaptability to complex scenes.

[0005] However, existing CLIP-based weakly supervised video anomaly detection methods still suffer from three major shortcomings: 1) Insufficient temporal modeling: Existing methods often directly reuse CLIP's image encoder to extract static frame features, ignoring the temporal correlation between video frames and failing to capture the dynamic evolution of abnormal events (such as the continuous actions of "conflict escalation" in violent incidents); 2) Poor adaptability to abnormal scenes: Abnormal events have strong scene dependence (such as "driving in the wrong direction" is abnormal on ordinary roads but normal in an ambulance emergency mission), and the types of anomalies are diverse (violence, theft, accidents, etc.). Existing methods rely only on visual features and do not combine textual semantic information to achieve fine-grained recognition, making it difficult to cope with complex scenes; 3) Weak noise interference suppression capability: Video data contains high-frequency noise such as changes in lighting, occlusion, and background redundancy. Existing methods often process features in the temporal domain and cannot effectively separate noise from abnormal signals, leading to decreased detection accuracy and increased false alarm rate.

[0006] Mutual information, as an important indicator in information theory for measuring the correlation of variables, has proven its effectiveness in fields such as multi-task learning and feature fusion. Fourier transform, on the other hand, can separate signal from noise through frequency domain decomposition. Based on this, the inventors aim to design a weakly supervised video anomaly detection method (CLIP-TAF) that integrates CLIP cross-modal information, temporal modeling, and frequency domain filtering. This method captures dynamic dependencies through a local-global temporal adapter, suppresses noise through a Fourier temporal filter, and adapts to complex scenarios through dual-branch detection, thus addressing the core shortcomings of existing technologies. Summary of the Invention

[0007] The purpose of this invention is to overcome the problems of difficulty in capturing dynamic events, poor scene adaptability, and strong noise interference in weakly supervised video anomaly detection, and to provide a CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method.

[0008] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:

[0009] This invention provides a CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method, comprising the following steps:

[0010] Step 1: Obtain a weakly supervised video dataset and preprocess the video data to build a training sample library;

[0011] Step 2: Construct a two-branch weakly supervised video anomaly detection network structure based on Fourier transform;

[0012] Step 3: Input the preprocessed video samples into the network, and train the model jointly through multiple loss functions to finally realize the detection and recognition of abnormal events in the video.

[0013] Furthermore, in step one, the video data preprocessing includes the following steps:

[0014] 1) Video frame sampling: Extract key frames from the input video at a fixed frame rate of 5 to 10 frames per second, and remove redundant frames to reduce computational costs;

[0015] 2) Feature standardization: The extracted video frames are normalized to a set size (224×224, compatible with CLIP-ViT-B / 16 encoder) and pixel values ​​are standardized (mean=0.5, standard deviation=0.5).

[0016] 3) Sample library construction: The training set and test set are divided according to video-level labels (normal / abnormal). The training set retains only video-level binary labels, while the test set is additionally labeled with frame-level abnormal intervals for performance evaluation. The weakly supervised video dataset includes the UCF-Crime, XD-Violence public dataset or a custom monitoring scene dataset.

[0017] Furthermore, step two, building the network structure, includes the following steps:

[0018] 1) In the feature extraction stage of frozen weights, the network adopts a context information serial perception structure, and obtains context features at different scales through a local Transformer encoder (LT) and an improved global temporal adapter (GTA-S);

[0019] 2) Use Fourier transform timing filters (TAFs) for frequency domain noise suppression and anomaly feature enhancement;

[0020] 3) Use a dual-branch design to improve the simultaneous completion of coarse and fine particle size detection tasks;

[0021] 4) Use multi-loss function joint training to improve information interaction between the two branch tasks.

[0022] Furthermore, the CLIP feature extraction module is implemented by using a pre-trained CLIP-ViT-B / 16 model as the backbone network and freezing the weights of its image encoder and text encoder.

[0023] Visual feature extraction: The preprocessed video frames are input into the CLIP image encoder, which outputs a frame-level visual feature tensor with dimension d=768. n is the number of video frames;

[0024] Text feature extraction: Construct a text description library of abnormal events (such as "violent attack", "theft", "explosion", etc.), input it into CLIP text encoder to generate text feature vectors for subsequent vision-language alignment tasks.

[0025] Furthermore, the Local-Global Timing Adapter (LGT-Adapter) includes a Local Transformer Encoder (LT) and an Improved Global Timing Adapter (GTA-S), and the specific implementation steps are as follows:

[0026] 1) Local Transformer Encoder (LT): Transforms frame-level visual features The time-series data is segmented into equal-length windows with overlapping regions (overlap rate 50%) (window length is adapted to the dataset: 8 for UCF-Crime and 64 for XD-Violence); window-based multi-head self-attention (W-MSA) computation is performed independently within each window to constrain the scope of information interaction to focus on local temporal dependencies and avoid redundancy in global computation; LayerNorm and Swish activation functions are concatenated after each attention layer to enhance the non-linear expressive power of features;

[0027] 2) Improved Global Temporal Adapter (GTA-S): Based on Graph Convolutional Network (GCN), it models global temporal dependencies and replaces the traditional GeLU activation function with the Swish function to improve gradient propagation efficiency; it constructs two types of adjacency matrices:

[0028] Similarity adjacency matrix : Calculate cosine similarity using frame-level features, combined with a step size function (Retain if similarity ≥ 0.7, otherwise set to 0), the formula is:

[0029] ;

[0030] in Used to avoid a denominator of 0; distance adjacency matrix Manhattan distance based on frame index The calculation formula is as follows:

[0031] ;

[0032] Where e is the learnable scale parameter; Global feature fusion: The local features \(X_l\) output by LT are mapped through a linear transformation matrix W and then normalized by Softmax. , Matrix multiplication is performed, and the final output, after Swish activation, is the global temporal feature. The formula is:

[0033] .

[0034] Furthermore, the Fourier transform time-series filter (TAF) is used for frequency domain noise suppression and anomaly feature enhancement. The specific implementation steps are as follows:

[0035] 1) Fast Fourier Transform (FFT): Transforms the time-series characteristics output by the LGT-Adapter. Transform from the time domain to the frequency domain to obtain the frequency domain signal. ;

[0036] 2) Adaptive energy threshold calculation: Calculate the energy of each sample at each frequency position. , where i is the sample index and j is the frequency index; Where k corresponds to the complex amplitude dimension; calculate the median of the sample energy. , Let i be the frequency domain energy vector of the i-th sample; normalized energy And generate an adaptive mask. :

[0037] ;

[0038] in This is an empirical threshold that can be fine-tuned based on the dataset.

[0039] 3) Frequency domain weighting and inverse transform: transforming frequency domain signals... With adaptive mask Learnable weights Multiply to obtain the filtered frequency domain signal. : Using Inverse Fast Fourier Transform (IFFT) Transform back into the time domain to obtain the denoised temporal features. ;

[0040] 4) Interactive Convolutional Blocks (ICBs): For Parallel processing using dual convolutional layers (3×1 small convolutional kernels to capture local details, and 7×1 large convolutional kernels to capture global context), the outputs are element-wise multiplied and fused, then compressed through a 1×1 convolution to obtain the enhanced features. The formula is:

[0041] ;

[0042] in The output is a 3×1 convolution. The output is a 7×1 convolution. This indicates element-wise multiplication.

[0043] Furthermore, a dual-branch detection module is established, including a visual classification branch and a visual-language alignment branch, to achieve coarse-grained and fine-grained anomaly detection respectively:

[0044] 1) Visual classification branch: This branch utilizes the enhanced features output by the TAF. Input a fully connected layer, output video frame-level anomaly confidence (0-1), calculated using binary cross-entropy loss. Optimize to achieve binary classification of "normal / abnormal" (coarse-grained detection);

[0045] 2) Visual-Language Alignment Branch: Calculating Enhanced Features Cosine similarity to CLIP text features, loss estimated through noise contrast. By maximizing the similarity between abnormal frame features and corresponding abnormal text features, and minimizing the similarity with normal text features, abnormal event type identification (fine-grained detection) can be achieved.

[0046] Furthermore, the loss function for joint training using multiple loss functions includes binary cross-entropy loss. Noise comparison estimation loss Compared with loss The total loss function formula is: ;in:

[0047] : Used for optimizing binary classification tasks in visual classification branches, the formula is:

[0048] ;

[0049] For frame-level real labels (0=normal, 1=abnormal). Predict confidence levels for the model;

[0050] : Used to optimize the vision-language alignment branch, the formula is:

[0051] ;

[0052] The similarity is between positive samples (abnormal frame - abnormal text). The similarity is between negative samples (abnormal frames and normal text).

[0053] : Used to increase the feature distance between normal and abnormal classes, the formula is:

[0054] ;

[0055] Features of the normal class This represents the j-th type of abnormal feature;

[0056] The balancing factor is set to 1e-1 for the UCF-Crime dataset and 1e-4 for the XD-Violence dataset.

[0057] Furthermore, in step three, the model training and inference parameters are set as follows:

[0058] 1) Optimizer: AdamW optimizer is used, with a weight decay rate of 1e-4;

[0059] 2) Learning rate: The learning rate was set to 1e-5 for the UCF-Crime dataset and 2e-5 for the XD-Violence dataset. Cosine annealing was used to adjust the learning rate.

[0060] 3) Batch size: Set to 64 based on a single NVIDIA RTX 3060 GPU;

[0061] 4) Training rounds: 10 rounds for the UCF-Crime dataset and 20 rounds for the XD-Violence dataset;

[0062] 5) Reasoning process: Input the test video into the trained model and output the frame-level anomaly confidence. When the confidence is ≥0.5, it is judged as an abnormal frame. Continuous abnormal frames constitute an abnormal event interval.

[0063] The beneficial effects of this invention are:

[0064] This invention designs a CLIP-assisted temporal adaptive filtering weakly supervised video anomaly detection network (CLIP-TAF). This network utilizes the CLIP cross-modal feature extraction module to acquire general semantic features and captures multi-scale temporal dependencies through a local-global temporal adapter (LGT-Adapter), thereby meeting the modeling requirements of dynamic evolution of anomalous events. Simultaneously, the Fourier transform temporal filter (TAF) achieves noise suppression and anomaly feature enhancement in the frequency domain, the dual-branch detection module adapts to scenes of varying complexity, and multiple loss functions enhance information interaction between tasks. Ultimately, this significantly improves the dynamic capture capability, scene adaptability, and noise resistance of weakly supervised video anomaly detection.

[0065] Of course, any product implementing this invention does not necessarily need to achieve all of the above advantages at the same time. Attached Figure Description

[0066] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 A schematic diagram of a weakly supervised video anomaly detection network based on CLIP-TAF provided in an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of the Local-Global Timing Adapter (LGT-Adapter) in the method provided by the embodiments of the present invention;

[0069] Figure 3 A schematic diagram of the Fourier transform time-series filter (TAF) in the method provided in the embodiments of the present invention;

[0070] Figure 4 This is a schematic diagram of the dual-branch detection module in the method provided in the embodiment of the present invention;

[0071] Figure 5 The image shows the coarse-grained anomaly detection results of the method implemented in this embodiment of the invention on the UCF-Crime public dataset;

[0072] Figure 6 The image shows the coarse-grained anomaly detection results of the method implemented in this embodiment of the invention on the XD-Violence public dataset. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] This embodiment provides a CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method, including the following steps:

[0075] Step 1: Obtain weakly supervised video data to obtain sample information;

[0076] Step 2: Construct a weakly supervised video anomaly detection network structure based on CLIP-TAF using sample information;

[0077] Step 3: Extract video temporal and cross-modal features through network modules, and then detect and identify abnormal events in the video.

[0078] In step one, the acquisition of sample information includes: video data preprocessing, analysis of abnormal event video interpretation markers, and construction of a sample library.

[0079] Video data preprocessing: Video frame sampling: Extract keyframes from the input video at a fixed frame rate of 5-10 frames / second, removing redundant frames to reduce computational costs. For example, in a surveillance video at 25 frames / second, extract one frame every three frames, balancing temporal continuity and data volume optimization; Feature standardization: Normalize the extracted keyframes to a size of 224×224 (compatible with CLIP-ViT-B / 16 encoder), and then apply the formula... Perform pixel value standardization so that the pixel mean is 0 and the standard deviation is 0.5;

[0080] Analysis of video interpretation markers for abnormal events: sort out the visual characteristics (such as "physical conflict" and "rapid action" in violent events, and "concealed action" and "transfer of items" in theft events) and scene characteristics (such as "vehicles driving in the wrong direction" in road scenes, and "crowd gathering and conflict" in shopping mall scenes) and establish a database of interpretation markers;

[0081] Sample library construction: The training set and test set are divided by video-level labels (0=normal, 1=abnormal) in a ratio of 4:1. The training set retains only video-level binary labels, while the test set is additionally labeled with frame-level abnormal intervals (such as "frames 100-200 are violent events"). It supports public datasets such as UCF-Crime and XD-Violence as well as custom monitoring datasets.

[0082] In step two, the steps for building the network structure include: the network adopts a collaborative structure of "cross-modal feature extraction + temporal dependency modeling + frequency domain filtering" in the feature extraction and processing stage, and obtains the temporally processed and denoised cross-modal features in stages through CLIP feature extraction module, local-global temporal adapter (LGT-Adapter) and Fourier transform temporal filter (TAF);

[0083] In the detection phase, a dual-branch detection module is used to aggregate temporal features and cross-modal features to achieve coarse-grained anomaly classification and fine-grained anomaly recognition, respectively. Fourier transform temporal filter (TAF) is used to enhance the noise suppression capability and anomaly feature representation within the model. Multiple loss functions (including contrastive loss with mutual information) are used to improve the information interaction between cross-modal tasks and classification tasks.

[0084] In this embodiment, the CLIP feature extraction module first uses the pre-trained CLIP-ViT-B / 16 model as the backbone network and freezes the weights of the image encoder and text encoder to reuse general cross-modal knowledge.

[0085] Visual feature extraction: The normalized video frames are input into the CLIP image encoder (containing 12 Transformer layers) to extract frame-level visual feature tensors with a dimension of 768. (n is the number of video frames, d=768);

[0086] Text feature extraction: Construct a text description library of abnormal events (such as "violent attack", "theft", "explosion", "car accident"). Each text description is input into the CLIP text encoder to generate a 768-dimensional text feature vector for subsequent visual-language alignment. This module provides basic features for anomaly detection that are both general and semantically relevant through cross-modal feature extraction, avoiding the sample dependency problem caused by training the feature extraction network from scratch.

[0087] In this embodiment, the Local-Global Timing Adapter (LGT-Adapter) includes a Local Transformer Encoder (LT) and an Improved Global Timing Adapter (GTA-S), focusing on capturing local and global temporal dependencies between video frames:

[0088] Local Transformer Encoder (LT): First, it converts frame-level visual features... The dataset is segmented along the temporal dimension into equal-length windows with 50% overlap (window length adapted to datasets: UCF-Crime set to 8, XD-Violence set to 64). Attention is independently computed within each window using window-based multi-head self-attention (W-MSA) to constrain the information interaction range and focus on short-range local temporal dependencies (such as the continuous action of "punching-hitting"). Each attention layer is followed by a LayerNorm layer and a Swish activation function, and residual connections are used to maintain information flow and mitigate the gradient vanishing problem, ultimately outputting local temporal features. Improved Global Temporal Adapter (GTA-S): Comprising two parts: adjacency matrix construction and global feature fusion. Adjacency matrix construction: Constructing a similarity adjacency matrix using frame-level feature cosine similarity. (Retaining connections with similarity ≥ 0.7), construct a distance adjacency matrix using frame index Manhattan distance. The formulas are as follows:

[0089]

[0090] in This is the step size function (1 when z≥0, 0 otherwise). To avoid a denominator of 0, The frame index is the Manhattan distance, and e is the learnable scale parameter; Global feature fusion: combining local features After mapping via the linear transformation matrix W (768×384), it is normalized to Softmax. , Perform matrix multiplication, then activate with Swish to output global temporal features. The formula is: This design enables the LGT-Adapter to extract rich temporal features across local and global levels, providing a comprehensive understanding of the dynamic evolution of anomalous events.

[0091] In this embodiment, the Fourier Transform Temporal Filter (TAF) corresponds to the output characteristics of the LGT-Adapter. It employs a frequency domain adaptive filtering strategy to effectively separate noise and anomalous signals and enhance the characterization of anomalous features. This module first converts the global temporal characteristics... Transform to the frequency domain using Fast Fourier Transform (FFT) ,in and These represent the row and column indices in the time and frequency domains, respectively.

[0092]

[0093] Global timing information in the frequency domain Processing with an adaptive high-pass and low-pass combined filter: High-frequency components (noise-dominant) and an adaptive high-pass filter Dot product, filtering redundant high-frequency noise, the formula is:

[0094]

[0095] Low-frequency components (dominated by anomalous signals) and adaptive low-pass filters Dot product, preserving core semantic information, has the following formula:

[0096]

[0097] in It is the cutoff frequency parameter (learnable by the network), obtained through the step function. Controlled frequency preservation: When the normalized distance of a frequency component to the center of the frequency domain is greater than 100%. Time Retention Otherwise filter The processed frequency domain signal is converted back to the time domain using the inverse fast Fourier transform (IFFT) to obtain the denoised time-series characteristics. :

[0098]

[0099] Finally, interactive convolutional blocks (ICBs) were used to... Multi-scale feature fusion is performed: a 3×1 small convolutional kernel is used to extract local details, and a 7×1 large convolutional kernel is used to capture global context. The output is then element-wise multiplied and compressed by a 1×1 convolution to generate the final enhanced features. The formula is:

[0100]

[0101] in The output is a 3×1 convolution. The output is a 7×1 convolution. This is element-wise multiplication. Furthermore, the dual-branch detection module corresponds to the enhanced features of TAF and employs a "coarse-fine" granular collaborative detection strategy to effectively adapt to anomaly scenarios of varying complexity: Coarse-grained visual classification branch: using 1×1 convolution to combine the enhanced features... The channel is uniformly set to 1, the input to the fully connected layer is the frame-level anomaly confidence (0-1), and the result is obtained through binary cross-entropy loss. The formula for optimizing the "normal / abnormal" binary classification task is:

[0102]

[0103] in For frame-level real labels, For predicting confidence, N is the number of frames; fine-grained visual-language alignment branch: computes augmenting features. The cosine similarity to CLIP text features is used to estimate the loss through noise contrast. The formulas for maximizing the similarity between abnormal frames and abnormal text, and minimizing the similarity between abnormal frames and normal text, are as follows:

[0104]

[0105] in For positive sample similarity, For negative sample similarity, K is the number of samples; this module, through a dual-branch design, achieves both rapid anomaly localization and supports fine-grained anomaly type identification, improving adaptability to complex scenarios. Furthermore, the multiple loss functions include binary cross-entropy loss (…). ), noise contrast estimation loss ( ) and contrast loss ( ), explicitly modeling and optimizing the correlation between classification tasks and cross-modal tasks, thereby more closely aligning tasks: contrastive loss ( Drawing inspiration from mutual information, this approach enhances feature discriminative power by maximizing the distance between features of the normal and abnormal classes. This is defined as:

[0106]

[0107] in The mean of the normal class features. Let be the mean of the j-th class of abnormal features. Minimizing this loss increases the distance between the two classes of features. The total loss function is a weighted sum of the three values, as shown in the formula:

[0108]

[0109] in The balance coefficient (UCF-Crime is set to 1e) -1 XD-Violence is set to 1e -4 To ensure numerical stability, a minimal constant is added to the denominator and logarithmic parameter in each loss calculation. .

[0110] This embodiment also provides a CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection system, including:

[0111] The acquisition module is configured to acquire the raw video data to be detected (such as real-time streams from surveillance cameras or local video files).

[0112] The preprocessing module is configured to perform frame sampling and feature standardization on the original video to obtain anomaly detection samples;

[0113] The CLIP-TAF network module is configured to extract video temporal and cross-modal features through CLIP feature extraction, temporal modeling, frequency domain filtering, and bi-branch detection, and to perform adaptive feature selection and enhancement in the channel and spatial directions, thereby realizing abnormal event detection and recognition.

[0114] This embodiment is based on CLIP-assisted temporal adaptive filtering weakly supervised video anomaly detection method. The core objective is to "utilize CLIP's cross-modal semantic alignment capability, temporal dependency modeling, and frequency domain filtering to promote dynamic feature capture of abnormal events, enhance adaptability to complex scenes, suppress noise interference, and reduce the problems of insufficient dynamic capture, poor scene generalization, and high false alarm rate in weakly supervised video anomaly detection." Targeted deep learning algorithms are designed to improve anomaly detection accuracy and enhance the model's adaptability to different scenes and types of abnormal events.

[0115] A specific application of this embodiment is as follows:

[0116] A weakly supervised video anomaly detection method based on CLIP-TAF includes the following steps:

[0117] Step 1: Selection and Sample Creation Based on Video Data

[0118] The construction of the entire deep learning anomaly detection sample database system includes frame-level annotation of targets, attribute definition, label format conversion, sample scale adaptation and enhancement, and dataset partitioning. The specific process is as follows:

[0119] Frame-level annotation:

[0120] Based on the video interpretation markers for abnormal events (such as "rapid physical conflict" and "person falling to the ground" in violent incidents, "concealed hand movements" and "item transfer" in theft incidents, and "vehicles driving against traffic" and "running red lights" in traffic anomalies), the LabelStudio annotation tool is used to manually visually interpret and label the frame-level intervals of abnormal events in the videos (e.g., "frames 150-220 are a violent attack"). Annotation accuracy directly affects sample quality; therefore, annotation error is strictly controlled: the error between the start and end frames of the abnormal interval should not exceed 2 frames to ensure the label accuracy of subsequent model training.

[0121] (2) Attribute definition

[0122] Using LabelStudio's "Attribute Editing" feature, add attribute labels to the annotated exception event samples, including:

[0123] Anomaly type attributes: such as "violence", "theft", "car accident", "explosion" (adapted for fine-grained detection tasks);

[0124] Scene attributes: such as "indoor shopping mall", "outdoor road", "campus", "industrial zone" (for subsequent scene generalization verification);

[0125] Difficulty attributes: such as "low occlusion", "medium occlusion", "high occlusion" (used to analyze the performance of the model under different disturbances).

[0126] (3) Tag format conversion

[0127] Obtaining anomaly detection labels primarily involves converting frame-level interval annotations into a vector format readable by the model. Using a Python script, the JSON-formatted annotation file exported from LabelStudio is converted into frame-level binary label vectors (length matching the number of video frames, with anomaly frames labeled 1 and normal frames labeled 0). Simultaneously, video-level labels are generated (videos containing anomaly events are labeled 1, and completely normal videos are labeled 0). The converted labels must strictly match the corresponding video frame data (one-to-one frame index correspondence) to avoid training bias due to misalignment.

[0128] (4) Sample scale adaptation and enhancement

[0129] Due to the diverse frame sizes (e.g., 1080P, 720P) and massive data volume of videos, direct input into the network is difficult, requiring scale adaptation and enhancement processing.

[0130] Scale adaptation: Use OpenCV tools to resize all video frames to a uniform size of 224×224 (to adapt to the input requirements of CLIP-ViT-B / 16 encoder), while keeping the aspect ratio of the frames unchanged (filling the edges with black pixels to avoid image stretching and distortion).

[0131] Data augmentation: Offline augmentation strategies are employed to improve sample diversity, including:

[0132] Brightness perturbation: Randomly adjust the brightness of the frame (±15%) to simulate different lighting conditions;

[0133] Horizontal flip: The frame is horizontally flipped with a 50% probability to avoid overfitting the model to the event direction;

[0134] Gaussian noise addition: Gaussian noise with a standard deviation of 0.01 is added with a 30% probability to enhance the model's noise resistance;

[0135] Frame sequence cropping: Randomly crop a subsequence of 32 frames (including 50% overlap) from the video frame sequence to increase the number of training samples.

[0136] (5) Dataset partitioning

[0137] The processed samples were randomly selected according to the ratio of "training set: validation set: test set = 3:1:1" to ensure that the anomaly types and scenario distributions of each set were consistent (stratified sampling was used to avoid excessive concentration of a certain type of anomaly in a single set).

[0138] Training set: Only video-level labels and anomaly type attributes are retained for model parameter training (conforming to the weakly supervised paradigm).

[0139] Validation set: Retains frame-level labels and scene attributes, used to supervise the convergence state during model training (such as early stopping policy triggering and learning rate adjustment).

[0140] Test set: Retains complete frame-level labels, anomaly type attributes, and difficulty attributes to comprehensively evaluate the model's detection accuracy, scene generalization, and anti-interference ability.

[0141] Step 2: Model Network Structure and Module Construction

[0142] (1) Overall network structure

[0143] The network employs a three-stage collaborative structure of "cross-modal feature extraction - temporal dependency modeling - frequency domain filtering" in the feature extraction and processing stage. It acquires temporally processed and denoised cross-modal features in stages through the CLIP feature extraction module, the Local-Global Temporal Adapter (LGT-Adapter), and the Fourier Transform Temporal Filter (TAF). In the detection stage, a dual-branch detection module is used to aggregate multi-source features to achieve "coarse-fine" granular collaborative detection. Simultaneously, multiple loss functions (including contrastive loss based on mutual information) are introduced to enhance information interaction between tasks. Its structure is as follows: Figure 1 As shown.

[0144] The specific process is as follows:

[0145] The input video frames are processed by the CLIP feature extraction module to obtain frame-level visual features and abnormal text features;

[0146] Visual features are input into the LGT-Adapter, which decomposes and models local and global temporal dependencies, and outputs temporal enhancement features.

[0147] The temporal enhancement feature is input into the TAF, which suppresses noise and enhances anomalous features in the frequency domain, and outputs a denoising enhancement feature.

[0148] The noise-reducing and enhanced feature input is used to create a dual-branch detection module, which outputs coarse-grained anomaly confidence and fine-grained anomaly type prediction, respectively.

[0149] The error between the predicted value and the label is calculated using multiple loss functions, and the network parameters are optimized through backpropagation.

[0150] (1) CLIP Feature Extraction Module

[0151] The CLIP feature extraction module, serving as the network's fundamental feature source, is responsible for acquiring cross-modal features that possess both generality and semantic relevance. Specifically, it is implemented as follows: Visual Feature Extraction: A pre-trained CLIP-ViT-B / 16 model is used as the visual encoder, with the weights of its 12 Transformer layers frozen (to avoid sample dependency caused by training from scratch). 224×224 video frames are input into the encoder. After Patch Embedding (dividing the frame into 16×16 pixel patches, mapping the number of channels to 768) and Transformer block attention calculation, a 768-dimensional frame-level visual feature tensor is output. (n is the number of video frames, d=768); Text feature extraction: Construct an abnormal event text description library, with 3-5 diverse text descriptions corresponding to each abnormal type (e.g., "violence" type corresponds to "two-person physical conflict", "beating others", "armed attack"). Input the CLIP text encoder (containing 12 Transformer layers) and output a 768-dimensional text feature vector consistent with the visual feature dimension. (k is the number of anomaly types). Text features need to be in the same embedding space as visual features to provide a basis for subsequent cross-modal alignment.

[0152] (2) Local-Global Timing Adapter (LGT-Adapter)

[0153] The LGT-Adapter is responsible for capturing the temporal dependencies between video frames, solving the problem of lack of dynamic information in CLIP static features. Its structure is as follows: Figure 2 As shown, the specific implementation is as follows:

[0154] Local Transformer Encoder (LT): Frame sequence segmentation, converting visual features The sequence is segmented along the temporal dimension into windows with 50% overlap (window length adapted to the dataset: 8 for UCF-Crime, 16 for XD-Violence). Each window contains w frames of features (e.g., when w=8, a sequence of n=32 frames is segmented into 5 windows). Intra-window attention calculation: Window-based multi-head self-attention (W-MSA) is performed independently within each window, and attention weights are calculated only within the window (to avoid redundancy in global computation). The formula is as follows: in For the query, key, and value matrix of features within the window, (Attention heads are 12); Feature refinement: Each W-MSA layer is followed by a LayerNorm layer (normalizes the feature distribution), a Swish activation function (enhances nonlinear representation), and a residual connection (alleviates gradient vanishing), outputting local temporal features. .

[0155] Improved Global Temporal Adapter (GTA-S): Adjacency Matrix Construction: Modeling global dependencies from two dimensions—feature similarity and temporal distance—to construct two types of adjacency matrices: similarity adjacency matrices. Calculate the cosine similarity of frame-level features, retaining connections with a similarity ≥ 0.7 (filtering irrelevant frames). The formula is: in It is a step function (1 when z≥0, 0 otherwise). Avoid denominators of 0; Distance adjacency matrix Manhattan distance calculation based on frame index captures long-range temporal dependencies. The formula is as follows: in For frame index, Learnable scale parameters (controlling the weight of distance on dependencies); Global feature fusion: combining local features Through linear transformation matrix Mapped to 384 dimensions, and normalized to Softmax. , Matrix multiplication is performed to fuse similarity and distance information, and finally, global temporal features are output after Swish activation. The formula is:

[0156] (3) Fourier Transform Temporal Filter (TAF) To suppress high-frequency noise in video data (such as snowflakes and dynamic shadows in surveillance footage), a TAF module is designed to implement adaptive filtering in the frequency domain. The specific steps are: Frequency domain transformation: global temporal characteristics of GTA-S output. Perform a Fast Fourier Transform (FFT) to convert to the frequency domain and obtain (F is the frequency dimension, taken as half of the number of frames N), the formula is: in For spatial domain indexing, Frequency domain indexing captures the frequency distribution of time-series features. Adaptive frequency domain filtering: Energy calculation: Calculates the energy at each frequency position of the frequency domain features. (where i is the sample index and j is the frequency index), the formula is: (k is the feature channel index), quantize the signal intensity of each frequency component; energy normalization: calculate the median energy of each sample. Normalize the energy ( Eliminate energy scale differences between different samples; Adaptive mask generation: set a threshold (Determined through validation set optimization), generate mask. :

[0157]

[0158] in The high-frequency components correspond to noise, which is suppressed by masking; The low- and mid-frequency components correspond to the effective signal, which are preserved and enhanced. Time-domain transformation and feature enhancement: Inverse Fourier transform: This enhances the filtered frequency domain features. ( Perform an inverse fast Fourier transform (IFFT) on the learnable frequency weights to transform them back to the time domain and obtain the denoised features. Interactive convolutional blocks (ICBs): for Multi-scale convolutional enhancement is performed using parallel 3×1 and 7×1 convolutional kernels (384 output channels). Multi-scale features are fused through element-wise multiplication, as shown in the formula: , , in The output is a 3×1 convolution. The output is a 7×1 convolution, and Conv3 is a 1×1 convolution (768 channels). The final output is the filtered and enhanced temporal features. .

[0159] (4) The dual-branch prediction module adopts a dual-branch structure of visual classification branch and view-text alignment branch to realize coarse-grained and fine-grained anomaly detection respectively: Visual classification branch (coarse-grained detection) input: temporal features output by TAF module Structure: A 2-layer fully connected network (512 hidden layer dimensions, Swish activation function) is used to output video-level anomaly confidence. (Between 0 and 1, ≥0.5 is considered anomaly); Loss function: Binary cross-entropy loss ( The formula is: Where y represents the video-level label (1 = abnormal, 0 = normal), optimizing the model's binary classification ability for abnormal / normal videos. Visual-text alignment branch (fine-grained detection) input: temporal features. CLIP text features Cross-modal alignment: Calculating the cosine similarity between visual features and text features. Loss is estimated using noise comparison ( To optimize alignment, the formula is: Where M is the sample size. Let K represent the true class of sample i, and K be the number of classes, to strengthen the correlation between visual features and text semantics; Fine-grained classification: Based on the aligned features, a fully connected network outputs the class probability distribution. This enables fine-grained differentiation of anomaly categories (such as "attack" and "theft").

[0160] (5) Mutual information constraint loss is introduced to enhance information interaction between two-branch tasks (visual classification and visual-text alignment). ), Explicitly modeling the correlation between the outputs of two tasks: Mutual information definition: mutual information between the outputs of two tasks (Outlier confidence level) and (Class probabilities), calculate their mutual information The formula is: in Entropy measures the uncertainty of a random variable; loss is calculated by minimizing the negative value of mutual information to maximize the correlation between the outputs of the two tasks, as shown in the formula: Among them, joint entropy The joint probability distribution is approximated by the dot product of the two outputs, and then solved. During the calculation process, additional steps are added... Ensure numerical stability; Total loss function: Combining the three loss components, the formula is: Where λ is the weight coefficient, set to 0.1 for the UCF-Crime dataset and 0.05 for the XD-Violence dataset, and determined through validation set tuning to balance the contributions of each loss term.

[0161] Step 3: Model Training and Inference Process

[0162] (1) Model training configuration hardware and software environment Hardware: The training device is a single NVIDIA RTX 3060 GPU (12 GB video memory), the CPU is a 12th Gen Intel(R) core(TM) i7-12700F (2.10 GHz), and the memory is 64GB; Software: The deep learning framework is PyTorch 2.0, CUDA 12.1, Python 3.9, and the dependent libraries include torchvision (0.15.1), clip (1.0), and scikit-learn (1.2.2).

[0163] (2) Training hyperparameter optimizer: AdamW optimizer is used; Batch size: 64 for UCF-Crime dataset and 96 for XD-Violence dataset to adapt to different data volumes; Learning rate: 2e-5 for UCF-Crime dataset and 1e-5 for XD-Violence dataset. Cosine annealing learning rate scheduling is used. The learning rate is reduced to 0.8 times the current value every 5 epochs to avoid training oscillations in the later stages; Training epochs: 10 epochs for UCF-Crime dataset and 10 epochs for XD-Violence dataset. Early stopping strategy is adopted (training is stopped if the validation set AUC does not improve for 3 consecutive epochs) to save the optimal model weights.

[0164] (3) Input processing of inference process: For the video to be detected, extract the frame sequence according to the frame extraction interval of the training stage (1 frame every 8 frames for UCF-Crime, 1 frame every 64 frames for XD-Violence), and perform visual enhancement (brightness and contrast adjustment) consistent with the training stage to ensure that the input feature distribution matches; Feature extraction: Input the frame sequence into CLIP image encoder and output frame-level visual features. Simultaneously, input the anomaly category text (such as "assault" or "normal") to obtain text features. Temporal modeling and filtering: The LGT-Adapter is input, and the data is sequentially passed through a local Transformer encoder and a GTA-S to obtain global temporal features. The input is then fed into the TAF module, where it undergoes frequency domain filtering and ICB enhancement to output the final timing characteristics. ;

[0165] (4) Two-branch prediction: Visual classification branch: Outputting anomaly confidence levels via a fully connected network ,like The video is judged as "abnormal" otherwise; visual-text alignment branch: calculation and Similarity, output class probability The category with the highest probability is selected as the fine-grained anomaly type (only anomaly videos are output); post-processing of results: the frame-level anomaly confidence sequence (each frame corresponds to one confidence level) is smoothed by a moving average (window size of 5 frames) to eliminate local fluctuations; based on the smoothed confidence sequence, the start and end frames of the anomaly event are determined (the first confidence level ≥ 0.5 is the start frame, and the last confidence level ≥ 0.5 is the end frame), and the final detection results are output (video-level judgment, fine-grained category, frame-level anomaly interval).

[0166] Step 4: Model Performance Verification and Analysis

[0167] (1) The accuracy verification of the method in this embodiment is to comprehensively evaluate the performance of the CLIP-TAF model. On the UCF-Crime and XD-Violence datasets, comparative experiments are carried out with advanced methods in the field of weakly supervised video anomaly detection. The comparison methods include: single-modal methods: RTFM (temporal modeling based on Transformer), DMU (dual memory unit), AVVD (audio-visual fusion); CLIP derivative methods: UMIL (CLIP + multi-instance learning), CLIP-TSA (CLIP + temporal self-attention), VadCLIP (CLIP + cross-modal alignment); multi-task method: TPWNG (text prompts + normality guidance).

[0168] (2) Frame-level AUC (Area Under ROC Curve), frame-level AP (Average Precision), and video-level Ano-AUC (abnormal video AUC) were used as core evaluation indicators. The quantitative comparison results are shown in Table 1 and Table 2:

[0169] Table 1 Quantitative Comparison Results of UCF-Crime Datasets

[0170]

[0171] Table 2 Quantitative Comparison Results of XD-Violence Dataset

[0172]

[0173] As shown in Tables 1 and 2, the CLIP-TAF model in this embodiment outperforms the comparison method on all metrics of the two datasets:

[0174] Comparative experiments reveal that the CLIP-based action recognition method proposed by Ju et al. performs poorly on both tasks, achieving an AUC of 84.72% for UCF-Crime and an AP of 76.57% for XD-Violence. This highlights the inherent challenges of weakly supervised video anomaly detection (WSVAD) and also indirectly confirms the effectiveness of our method. VAD-TAF achieves breakthroughs on both datasets: a record-breaking AUC of 88.49% on the UCF-Crime dataset, improving upon the second-best method VadCLIP by 0.47 percentage points; and a state-of-the-art AP of 84.09% on the XD-Violence dataset, improving upon the second-best TPWNG method by 0.49 percentage points.

[0175] In a cross-sectional comparison on the XD-Violence dataset, VAD-TAF achieved a significant performance improvement of 1.68% compared to the DMU method with a hierarchical architecture. Compared to the AVVD model that incorporates external knowledge, this method still maintains a performance improvement of 5.99%. Furthermore, the CLIP-TSA and TPWNG methods, which also utilize CLIP, achieved higher AP improvements of 1.68% and 0.41%, respectively.

[0176] By incorporating frequency domain information, VAD-TAF achieves accurate capture of anomalous behavior patterns, providing a new technical paradigm for video anomaly detection in weakly supervised scenarios. Its advantages are significant. Qualitative comparison results are as follows: Figure 5 , Figure 6 The qualitative results show that the model performs exceptionally well on both the UCF-Crime and XD-Violence datasets, demonstrating strong anomaly detection capabilities.

[0177] Regarding the identification of anomalous events: whether it is Arrest, Arson, Assault, etc. in UCF-Crime, or Fighting, Shooting, Riot, etc. in XD-Violence, the model's anomalous score curve can form obvious peaks or high responses in the interval where the corresponding event occurs (red area), accurately capturing the time period of the anomalous behavior, and is very sensitive to the location and identification of anomalous events.

[0178] Regarding the distinction between normal events: In the Normal scenario, the model's abnormal score curve is generally at a low level, which contrasts sharply with the high scores of abnormal events. It can clearly distinguish between normal and abnormal behavior, demonstrating excellent discrimination ability.

[0179] Cross-dataset generalization: The model maintains stable and excellent performance on both UCF-Crime and XD-Violence datasets, indicating that it has good generalization ability and can adapt to anomaly detection tasks in different scenarios.

[0180] In summary, this model performs quite professionally and efficiently in detecting and locating abnormal events and distinguishing normal events, and its performance on these two datasets is commendable.

[0181] In summary, the CLIP-TAF model in this embodiment effectively solves the core problems of "difficulty in capturing dynamic evolution, poor scene adaptability, and sparse supervision signals" in weakly supervised video anomaly detection by enhancing temporal modeling with LGT-Adapter, suppressing noise with TAF module, and adapting to complex scenes with dual-branch structure. Validation results on public datasets and custom scenes show that this method outperforms existing state-of-the-art methods in both detection accuracy and generalization ability, and can be widely applied to weakly supervised video anomaly detection tasks in fields such as intelligent surveillance, public safety, and industrial inspection.

[0182] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A CLIP-assisted temporal adaptive filtering weakly supervised video anomaly detection method, characterized in that, Includes the following steps: Step 1: Obtain a weakly supervised video dataset and preprocess the video data to build a training sample library; Step 2: Construct a two-branch weakly supervised video anomaly detection network structure based on Fourier transform; Step 3: Input the preprocessed video samples into the network, and train the model jointly through multiple loss functions to finally realize the detection and recognition of abnormal events in the video.

2. The CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method according to claim 1, characterized in that, In step one, video data preprocessing includes the following steps: 1) Video frame sampling: Extract key frames from the input video at a fixed frame rate of 5 to 10 frames per second, and remove redundant frames to reduce computational costs; 2) Feature standardization: Normalize the extracted video frames to a set size and standardize the pixel values; 3) Sample library construction: The training set and test set are divided according to video-level labels. The training set retains only video-level binary labels, while the test set is additionally labeled with frame-level anomaly intervals for performance evaluation. The weakly supervised video dataset includes the UCF-Crime, XD-Violence public dataset or a custom monitoring scene dataset.

3. The CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method according to claim 1, characterized in that, Step two involves building the network structure, including the following steps: 1) In the feature extraction stage of frozen weights, the network adopts a context information serial perception structure, and obtains context features at different scales through a local Transformer encoder and an improved global temporal adapter. 2) Use Fourier transform timing filters for frequency domain noise suppression and anomaly feature enhancement; 3) Use a dual-branch design to improve the simultaneous completion of coarse-grained and fine-grained particle detection tasks; 4) Use multi-loss function joint training to improve information interaction between the two branch tasks.

4. The CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method according to claim 3, characterized in that, The CLIP feature extraction module is implemented by using a pre-trained CLIP-ViT-B / 16 model as the backbone network and freezing the weights of its image encoder and text encoder. Visual feature extraction: The preprocessed video frames are input into the CLIP image encoder, which outputs frame-level visual feature tensors. n is the number of video frames; Text feature extraction: Construct an abnormal event text description library, input it into the CLIP text encoder to generate text feature vectors, which are used for subsequent vision-language alignment tasks.

5. The CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method according to claim 4, characterized in that, The local-to-global timing adapter includes a local Transformer encoder and an improved global timing adapter. The specific implementation steps are as follows: 1) Local Transformer Encoder: Converts frame-level visual features The process is divided into equal-length windows with overlapping regions along the temporal dimension; multi-head self-attention computation based on windows is performed independently within each window to constrain the scope of information interaction to focus on local temporal dependencies and avoid redundancy in global computation; LayerNorm and Swish activation functions are concatenated after each attention layer to enhance the nonlinear expressive power of features; 2) Improved Global Temporal Adapter: Based on graph convolutional networks, it models global temporal dependencies and replaces the traditional GeLU activation function with the Swish function to improve gradient propagation efficiency; it constructs two types of adjacency matrices: Similarity adjacency matrix : Calculate cosine similarity using frame-level features, combined with a step size function The formula is: ； in Used to avoid a denominator of 0; distance adjacency matrix Manhattan distance based on frame index The calculation formula is as follows: ； Where e is the learnable scale parameter; Global feature fusion: The local features \(X_l\) output by LT are mapped through a linear transformation matrix W and then normalized by Softmax. , Matrix multiplication is performed, and the final output, after Swish activation, is the global temporal feature. The formula is: 。 6. The CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method according to claim 5, characterized in that, Fourier transform time-series filters are used for frequency domain noise suppression and anomaly feature enhancement. The specific implementation steps are as follows: 1) Fast Fourier Transform: Transforms the timing characteristics output by the LGT-Adapter. Transform from the time domain to the frequency domain to obtain the frequency domain signal. ; 2) Adaptive energy threshold calculation: Calculate the energy of each sample at each frequency position. , where i is the sample index and j is the frequency index; Where k corresponds to the complex amplitude dimension; calculate the median of the sample energy. , Let i be the frequency domain energy vector of the i-th sample; normalized energy And generate an adaptive mask. : ； in This is an empirical threshold that can be fine-tuned based on the dataset. 3) Frequency domain weighting and inverse transform: transforming frequency domain signals... With adaptive mask Learnable weights Multiply to obtain the filtered frequency domain signal. : By using the inverse fast Fourier transform Transform back into the time domain to obtain the denoised temporal features. ; 4) Interactive convolutional blocks: for The system employs parallel processing with two convolutional layers. After element-wise multiplication and fusion of the outputs, the channel dimensions are compressed using a 1×1 convolution to ultimately obtain the enhanced features. The formula is: ； in The output is a 3×1 convolution. The output is a 7×1 convolution. This indicates element-wise multiplication.

7. The CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method according to claim 6, characterized in that, A dual-branch detection module is established, including a visual classification branch and a visual-language alignment branch, which respectively implement coarse-grained and fine-grained anomaly detection: 1) Visual classification branch: This branch utilizes the enhanced features output by the TAF. Input a fully connected layer, output video frame-level anomaly confidence, using binary cross-entropy loss. Optimize to achieve binary classification of "normal / abnormal"; 2) Visual-Language Alignment Branch: Calculating Enhanced Features Cosine similarity to CLIP text features, loss estimated through noise contrast. By maximizing the similarity between abnormal frame features and corresponding abnormal text features, and minimizing the similarity with normal text features, abnormal event type identification can be achieved.

8. The CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method according to claim 7, characterized in that, The loss function for joint training with multiple loss functions includes binary cross-entropy loss. Noise comparison estimation loss Compared with loss The total loss function formula is: ;in: : Used for optimizing binary classification tasks in visual classification branches, the formula is: ； For frame-level real labels, Predict confidence levels for the model; : Used to optimize the vision-language alignment branch, the formula is: ； For positive sample similarity, Negative sample similarity; : Used to increase the feature distance between normal and abnormal classes, the formula is: ； Features of the normal class This represents the j-th type of abnormal feature; The balancing factor is set to 1e-1 for the UCF-Crime dataset and 1e-4 for the XD-Violence dataset.

9. The CLIP-assisted time-adaptive filtering weakly supervised video anomaly detection method according to claim 1, characterized in that, The model training and inference parameters are set as follows in step three: 1) Optimizer: AdamW optimizer is used, with a weight decay rate of 1e-4; 2) Learning rate: The learning rate was set to 1e-5 for the UCF-Crime dataset and 2e-5 for the XD-Violence dataset. Cosine annealing was used to adjust the learning rate. 3) Batch size: Set to 64 based on a single NVIDIA RTX 3060 GPU; 4) Training rounds: 10 rounds for the UCF-Crime dataset and 20 rounds for the XD-Violence dataset; 5) Reasoning process: Input the test video into the trained model and output the frame-level anomaly confidence. When the confidence is ≥0.5, it is judged as an abnormal frame. Continuous abnormal frames constitute an abnormal event interval.

Images