Video anomaly event detection method based on prompt learning and multi-scale temporal fusion

Abstract

The present application relates to a video anomaly event detection method based on prompt learning and multi-scale temporal fusion, belonging to the field of video processing. The method mainly includes a double-branch structure of visual language prompt learning and multi-scale temporal feature fusion. The visual language prompt learning branch uses the strong prior knowledge of the CLIP model to identify potential abnormal behaviors in the data set and convert them into pseudo-labels, which serve as additional supervision signals to guide the training of the anomaly event detection model. The multi-scale temporal fusion branch uses a multi-scale temporal relationship modeling module to capture fine-grained temporal information at different granularities in the video, improving the recognition effect of the model in handling abnormal events of different time spans. Through the collaborative optimization of the double-branch structure, the robustness and generalization of the anomaly event detection model can be further improved, and it is suitable for various complex scenarios.

Description

Technical Field

[0001] This invention relates to the fields of video anomaly detection, visual language models, and multi-instance learning, specifically to a video anomaly event detection method based on cue learning and multi-scale temporal fusion. Background Technology

[0002] Video anomaly detection, a core technology in intelligent surveillance, primarily relies on computer vision to perform real-time analysis of surveillance videos. It plays a decisive role in improving early warning efficiency, strengthening pre-event prevention, and enhancing post-event tracking. However, due to the complexity and diversity of video anomalies, its research still faces multiple challenges. The primary challenge lies in the fact that current mainstream methods face numerous difficulties in deeply mining and utilizing the temporal information of videos, making it difficult for models to achieve comprehensive effectiveness when dealing with the identification of long-term or short-term anomaly segments. Especially in complex background environments, the boundary between anomalies and normal events becomes increasingly blurred, increasing the difficulty of detection. Furthermore, over-reliance on visual semantic features may limit the model's recognition to simple scenes; for anomaly videos containing complex contextual information, the model may exhibit a high false positive rate, thus affecting the accuracy of anomaly detection. Summary of the Invention

[0003] (a) Technical problems to be solved

[0004] To address the shortcomings of existing technologies, this invention proposes a video anomaly event detection method based on cue learning and multi-scale temporal fusion. It focuses on exploring how to fully utilize the visual language pre-trained model CLIP to enhance the accuracy and robustness of existing models, and studies how to process video temporal information to strengthen the ability to distinguish between abnormal and normal patterns.

[0005] (II) Technical Solution

[0006] To address the aforementioned issues, this invention transforms the anomaly detection task into a zero-shot learning problem. It leverages the CLIP (Visual Language Processing) large-scale model to generate pseudo-labels to guide model training, enhances pseudo-label quality through a pseudo-label ensemble strategy, and introduces a multi-scale temporal relationship modeling module to finely model the coarse-grained information of video temporal sequence. This enables high-accuracy detection of both long and short temporal anomaly segments. The specific technical solution adopted is as follows:

[0007] A video anomaly event detection method based on cue learning and multi-scale temporal fusion includes the following steps:

[0008] S1: Video data preprocessing. For a video v in the dataset... i First, divide the video into t iA video segment consisting of 16 consecutive, non-overlapping frames is fed into the I3D network for visual feature extraction. The video segment features obtained after global average pooling in the Mixed_5c layer of the I3D network are denoted as follows: Where d = 1024. The above operation is performed on each video in the dataset to obtain the visual features of all videos in the dataset, which are then used as input for subsequent processing.

[0009] S2: Video Language Cue Learning and Pseudo-Label Generation. A plug-and-play modular visual language cue learning module was designed, which can be easily integrated into existing video anomaly detection model structures. It utilizes the powerful zero-shot learning capability of visual language pre-trained models (such as CLIP) to generate pseudo-labels to guide model training, thereby transforming the originally weakly supervised learning environment with scarce labeled data into a supervised task environment with rich supervisory signals.

[0010] S2-1: Text and Visual Feature Construction. The constructed text prompt template is input into the CLIP model's text encoder to obtain the corresponding text features. Simultaneously, video frames are input into the CLIP model's image encoder to obtain the corresponding visual features. For text feature construction, we use a method commonly used in natural language processing based on text prompt templates. Assuming a target dataset containing C categories, we construct text for each of the C categories using a pre-designed prompt template (e.g., "a video of [CLS]"). For example, for "Explosion," we construct "a video of explosion," and then input this template into the text encoder to generate the text embedding corresponding to the "Explosion" category. c represents the corresponding category. For visual feature construction, we extract the corresponding visual features f from each video frame using an image encoder. img .

[0011] S2-2: Text-Visual Feature Similarity Calculation. To calculate the probability of each frame of a video belonging to each category, it is necessary to calculate the similarity between the visual features of the video frame and the text features of each category. This can be achieved by calculating cosine similarity:

[0012]

[0013] Here, τ is a temperature coefficient, and sim(·) represents the cosine similarity. Based on this similarity value, a probability distribution can be assigned to each frame, representing the likelihood that it belongs to each category.

[0014] S2-3: Fragment-level pseudo-label generation and ensemble. For each video v in the dataset... i All contain t i The video consists of several non-overlapping video segments, each labeled with a video-level tag indicating its category as c. Taking a single video segment as an example, the probability of each frame within that segment belonging to category c is calculated using S2-2. Next, the highest anomalous segment score among these frames is taken as the probability that the segment belongs to category c. This process is repeated sequentially for each segment in the video, ultimately yielding video v. i The corresponding fragment-level pseudo-label P′.

[0015] The quality of pseudo-labels directly affects the training effect of the model. To further improve the accuracy and stability of pseudo-labels, we propose a simple and efficient pseudo-label ensemble strategy, which integrates pseudo-labels generated by multiple different visual encoders. The core idea of ​​this strategy is to improve the accuracy and stability of pseudo-labels by combining pseudo-labels generated by multiple visual encoders. Specifically, N different visual encoders are used, which independently generate their own pseudo-labels using a unified simple cue template ("a video of a [CLS]"), i.e., {P′1, P′2, ..., P′...}. N During the integration process, an averaging strategy was employed to average the probabilities output by each visual encoder, thereby reducing the bias and noise that might be introduced by a single model. This process can be mathematically expressed as follows:

[0016]

[0017] Where, P∈R T N represents the number of visual encoders, P′ i This represents the pseudo-label generated by the i-th visual encoder. After obtaining the ensembled fragment-level pseudo-label P, it is binarized using a threshold (e.g., 0.5) and directly used to supervise the subsequent training process.

[0018] S3: Multi-scale temporal feature modeling. For x i Multi-scale modeling from coarse-grained to fine-grained is performed to obtain a multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Zn}. L Z = {Z1, Z2, ..., Z} L The lightweight convolutional decoder obtains the final features at various scales, and the final features are then passed through a classifier to obtain the outlier segment scores {S1, S2, ..., S} at each granularity. L Then, an adaptive weight fusion mechanism is used to obtain v. i The final abnormal segment score S is obtained through the following steps:

[0019] S3-1: Feature Mapping Layer. First, x...i The length of the segment t i The length is fixed at T. Specifically, if the video contains fewer than T segments, it is padded with zeros to a length of T. Otherwise, it is downsampled to a predefined length T. A feature mapping layer f is then introduced. proj , used to convert the input features x i It maps to a feature dimension suitable for multi-scale Transformer encoders. It consists of a shallow convolutional neural network that uses ReLU as the activation function to map high-dimensional input features to a low-dimensional embedding space. Its mathematical expression is as follows:

[0020] Z0 = f proj (x i )

[0021] Where Z0∈R T×F This represents the video features after dimensionality reduction mapping, where F = 256 is the feature dimension after mapping.

[0022] S3-2: Multi-scale Transformer Encoder. It consists of multiple stacked Transformer encoders, each layer embedding a Layer Normalization (LN), a Local Multi-Head Self-Attention (LMSA) module, and a Multilayer Perceptron (MLP). For the input video feature sequence Z0, depth information is first extracted and interactively processed using LMSA. Residual connections and layer normalization operations are then employed to ensure effective integration and stable propagation of information. Next, the GELU activation function is used to perform a nonlinear transformation on the output. To capture segment features at different time scales, ↓(·) downsampling is used to obtain feature representations at different time scales.

[0023]

[0024] in, T l / T l-1 It's the downsampling rate. α l , It is a learnable scaling factor. Considering computational efficiency, ↓(·) downsampling uses depthwise convolution, with a downsampling rate of 2 between each scale, i.e., T. l / T l-1 =2. Finally, the multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Z} was obtained. L}, where L represents the number of layers in the pyramid.

[0025] S3-3: Lightweight convolutional decoder. After obtaining the multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Z...} LAfterwards, a lightweight convolutional decoder consisting of one-dimensional convolutions is used to decode to obtain the final scores of anomalous segments at various granularities, and a classifier f is appended to them. cls To predict the probability of an anomalous instance occurring at each time step at each granularity, the mathematical expression is as follows:

[0026] S l =σ(f cls (Z l ;φ cls ),l=1…L

[0027] Among them, f cls It is implemented by three 1D convolutions with a stride of 3, layer normalization, and the ReLU activation function; φ cls σ represents the learnable parameters; σ is the sigmoid activation function. The obtained... Let S represent the score of the outlier segment at layer l. Thus, we obtain the outlier segment scores {S1, S2, ..., S...} at each granularity obtained through temporal feature pyramid decoding. L}

[0028] S3-4: Adaptive weight fusion mechanism. The scores of outlier segments at various granularities {S1, S2, ..., S...} are obtained through a lightweight convolutional decoder and classifier. L As input, in the initial stage of training, a learnable weight matrix β = {β1, β2, ..., β} is first initialized. L}, where β L Let represent the fusion weights of the Lth layer. These weights are uniformly initialized to a unit value of 1, indicating that in the initial stage of training, outlier scores at each granularity are equally important. The model is then iteratively trained and optimized until convergence is achieved. Its mathematical expression is as follows:

[0029]

[0030] Where S∈R T This represents the final score for the abnormal segment, where T represents the number of video segments.

[0031] S4: Classifier Optimization. Under weak supervision, video anomaly detection is treated as a multi-instance learning (MIL) task. This means that during training, the classifier can only rely on limited label information (video-level labels) to learn to recognize anomalies. To better distinguish between anomalous and non-anomalous action patterns, we employ a pseudo-label classification loss function, a dynamic ranking loss function for representative segments, and a triplet loss function to optimize the classifier.

[0032] S4-1: Pseudo-label classification loss function L pseWe address the data imbalance problem using the Focal Loss function, which is designed to solve the issue of insufficient model recognition of minority class samples caused by class imbalance in classification problems. In video anomaly detection tasks, introducing Focal Loss allows the classifier to focus more on those difficult-to-identify abnormal behavior samples, thereby optimizing the learning of minority class samples.

[0033] Specifically, in S3-4, the anomaly score S for each video segment is obtained; in S2, the segment-level pseudo-label is obtained through video language cue learning and pseudo-label generation; and the predicted anomaly segment score is directly optimized using Focal Loss. Its mathematical expression is as follows:

[0034] L pse =Focal(S,P)

[0035] in, T represents the number of video clips.

[0036] S4-2: Represents the fragment dynamic loss function L cls Suppose that for an input video v i It contains t i There are 1 segment, and the corresponding video-level tag is y. i Select the top-k examples (i.e., representative segments) with the highest predicted anomaly scores, among which... And use their mean as the video-level prediction score s i The binary cross-entropy (BCE) loss function is used to optimize the classifier, and its mathematical expression is as follows:

[0037] L cls =BCE(s) i ,y i )

[0038] Among them, y i ∈{0,1}, α is a hyperparameter used to control the number of segments selected.

[0039] S4-3: Triple Loss Function L trip For each video v i First, based on the predicted outlier segment score S, the top-k examples (k values ​​are consistent with S4-2) with the highest outlier segment scores are determined. The feature representations of these examples will be used as anchor points f. a Then, the feature centers of these anomalous examples are calculated and used as positive samples f. p Then, the feature centers of the normal examples are selected as negative samples f. n Its mathematical expression is as follows:

[0040] Ltrip =Triplet(f a ,f p ,f n )

[0041] By minimizing the distance between the anchor point and positive samples, while maximizing the distance between the anchor point and negative samples, better inter-class separation and intra-class compactness can be achieved in the feature space. This ultimately results in a more refined feature representation and a more discriminative feature distribution.

[0042] S4-4: Overall Loss Function L total During training, a combined loss function is constructed by integrating the pseudo-label classification loss function, the dynamic ranking loss function for representative segments, and the triplet loss function. To balance these three loss functions, this section treats them as independent sub-tasks. We employ a multi-task learning approach, enabling the model to adaptively determine the weight relationships of each loss function, thus achieving joint optimization and learning. Its mathematical expression is as follows:

[0043]

[0044] Where, λ i ,i∈{1,2,3} are the learnable loss function weights.

[0045] S5: For a new video, use the method in S1 to obtain x′. i , x′ i The input is a pre-trained multi-scale temporal feature network, which is then passed through a pre-trained lightweight convolutional decoder and classifier to output the predicted category.

[0046] Compared with the prior art, the present invention has at least the following advantages:

[0047] This invention designs an anomaly detection method based on visual language cue learning. By utilizing the CLIP model to generate high-quality pseudo-labels through visual language cue learning, the originally weakly supervised task with scarce labels is transformed into a supervised task with abundant supervisory signals, significantly improving detection accuracy. Simultaneously, this invention also designs a video anomaly detection method based on multi-scale real-series fusion. It utilizes a multi-scale temporal feature pyramid to model video temporal features from coarse to fine granular levels, introduces a triplet loss function from metric learning to improve the distinguishability between anomaly and normal pattern features, and uses an adaptive learning weight mechanism to fuse the detection results from different levels of fine granularity, achieving effective detection of both long-term and short-term anomaly segments with high accuracy. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention and the prior art, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0049] Figure 1 This is a simplified flowchart of the method of the present invention.

[0050] Figure 2 This is a schematic diagram of an anomaly detection framework based on multi-scale temporal fusion.

[0051] Figure 3 This is a flowchart of visual feature extraction.

[0052] Figure 4 This is a schematic diagram of a multi-scale temporal feature pyramid network.

[0053] Figure 5 Schematic diagram of a multi-scale Transformer encoder.

[0054] Figure 6 This is a schematic diagram of the adaptive weight fusion mechanism.

[0055] Figure 7 This is a schematic diagram of an anomaly detection method based on visual language cue learning. Detailed Implementation

[0056] The present invention will now be described in further detail.

[0057] To address the limitations of relying solely on visual features to process complex scenes, the visual-language pre-trained model CLIP effectively solves this problem. By transforming the anomaly detection task into a zero-shot learning problem, it leverages a visual-language cue learning mechanism to generate pseudo-labels to guide model training, and uses a pseudo-label ensemble strategy to enhance pseudo-label quality, thereby improving the accuracy and robustness of anomaly detection. Furthermore, to address the shortcomings in mining and utilizing video temporal information, the multi-scale temporal relationship modeling module solves this problem by finely modeling both coarse and fine-grained information of video temporal sequence. Coarse-grained temporal features capture persistent abnormal events, while fine-grained features focus on accurately locating transient abnormal segments. To enhance the ability to distinguish between abnormal and normal patterns in complex backgrounds, a triplet loss function is used to improve feature discriminability, and an adaptive learning weight mechanism is used to fuse fine-grained detection results from different levels, achieving high-accuracy detection of both long and short temporal anomaly segments.

[0058] This invention proposes an anomaly detection method based on visual language cue learning. This method leverages visual language cue learning and utilizes the CLIP model to generate high-quality pseudo-labels, successfully transforming a weakly supervised task with scarce labels into a supervised task with abundant supervisory signals, significantly improving detection accuracy. Simultaneously, this invention also constructs an anomaly detection method based on multi-scale temporal fusion. This method employs multi-scale temporal feature pyramid technology to achieve layer-by-layer modeling of video temporal features from coarse to fine granular. By introducing the triplet loss function from metric learning, this method significantly improves the ability to distinguish between abnormal and normal pattern features. Furthermore, an adaptive learning weight mechanism is used to effectively fuse the fine-grained detection results from different levels, achieving high-accuracy detection of both long-term and short-term temporal anomaly segments.

[0059] Figure 2 This invention demonstrates an anomaly detection framework based on multi-scale temporal fusion. For the original surveillance video, video feature extraction is first performed. The extracted features are then used for multi-scale temporal feature modeling to further enrich the feature information. The detection results from each scale are then adaptively fused with coarse and fine granularity to obtain the final anomaly segment score.

[0060] Figure 3 This demonstrates the workflow for extracting visual features from a single video. For a single video v in the dataset... i First, divide the video into t i A video segment consisting of 16 consecutive, non-overlapping frames is fed into the I3D network for visual feature extraction. The video segment features obtained after global average pooling in the Mixed_5c layer of the I3D network are denoted as follows: Where d = 1024. The above operation is performed on each video in the dataset to obtain the visual features of all videos in the dataset, which are then used as input for subsequent processing.

[0061] Figure 4 This paper demonstrates the multi-scale temporal feature pyramid network of the present invention. Each video v is obtained through a video feature extraction step. i Corresponding fragment features As input, a multi-scale temporal feature network is constructed. For feature x... i This network performs multi-scale modeling from coarse-grained to fine-grained. It mainly consists of: a feature mapping layer (projection), a multi-scale Transformer encoder, and a lightweight convolutional decoder.

[0062] Figure 5The multi-scale Transformer encoder structure of the present invention is shown. It is composed of multiple stacked Transformer encoders, each layer of which embeds layer normalization, local multi-head self-attention (LMSA) module and multilayer perceptron (MLP).

[0063] Figure 6 The adaptive weight fusion mechanism of this invention is illustrated. Anomaly segment scores {S1, S2, ..., S} at various granularities are obtained through a lightweight convolutional decoder and classifier. L As input, in the initial stage of training, a learnable weight matrix β = {β1, β2, ..., β} is first initialized. L}, where β l This represents the fusion weights of the l-th layer, and these weight coefficients are uniformly initialized to a unit value of 1. This means that in the initial stage of training, the outlier scores at each granularity are equally important. Subsequently, the model is trained iteratively to optimize until it reaches a convergent state.

[0064] Figure 7 This invention demonstrates an anomaly detection method based on visual language cue learning. The method is a plug-and-play modular visual language cue learning module. First, it constructs text and visual features. Preprocessed text and video are used as input, and the CLIP model's image encoder and text encoder are used respectively to construct corresponding text and visual features. To calculate the probability of each frame belonging to each category, the similarity between the visual features of the video frame and the text embedding for each category needs to be calculated. This can typically be achieved by calculating cosine similarity or other similarity measures. Based on these similarity values, a probability distribution can be assigned to each frame, representing its likelihood of belonging to each category.

[0065] For each video v in the dataset i All contain t i The video is divided into several non-overlapping video segments, each with a video-level tag indicating its category as c. Taking a single video segment as an example, the probability of each frame within that segment belonging to category c is calculated using the aforementioned probability formula. Next, the highest anomalous segment score among these frames is taken as the probability that the segment belongs to category c. This process is repeated sequentially for each segment in the video, ultimately yielding video v. i The corresponding fragment-level pseudo-label P′.

[0066] This paper proposes a simple yet efficient pseudo-tag ensemble strategy, which integrates pseudo-tags generated by multiple different visual encoders. The core idea of ​​this strategy is to improve the accuracy and stability of pseudo-tags by combining pseudo-tags generated by multiple visual encoders. Specifically, N different visual encoders are used, each independently generating its own pseudo-tags using a unified simple cue template ("a video of a [CLS]"), i.e., {P1′, P2′, ..., P′}. N During the integration process, an averaging strategy was employed to average the probabilities output by each visual encoder, thereby reducing the bias and noise that might be introduced by a single model.

[0067] The present invention will now be further described in conjunction with the accompanying drawings and embodiments, such as... Figure 1 As shown, a video anomaly event detection method based on cue learning and multi-scale temporal fusion includes the following steps.

[0068] S1: Video data preprocessing. For a video v in the dataset... i First, divide the video into t i A video segment consisting of 16 consecutive, non-overlapping frames is fed into the I3D network for visual feature extraction. The video segment features obtained after global average pooling in the Mixed_5c layer of the I3D network are denoted as follows: Where d = 1024. The above operation is performed on each video in the dataset to obtain the visual features of all videos in the dataset, which are then used as input for subsequent processing.

[0069] S2: Generating pseudo-labels for each video in the dataset, i.e., video language cue learning pseudo-label generation. We designed a plug-and-play modular visual language cue learning module that can be easily integrated into existing video anomaly detection model structures. It utilizes the powerful zero-shot learning capability of visual language pre-trained models (such as CLIP) to generate pseudo-labels to guide model training, thereby transforming the originally weakly supervised learning environment with scarce labeled data into a supervised task environment with rich supervisory signals.

[0070] S2-1: Text and Visual Feature Construction. Using preprocessed text and video as input, the CLIP model's text encoder and image encoder are used to construct corresponding text and visual features, respectively. For text feature construction, we use a method commonly used in natural language processing based on text prompt templates. Assuming a target dataset containing C categories, we construct text for each of the C categories using a pre-designed prompt template (e.g., "a video of [CLS]"). For example, for "Explosion," we construct "a video of explosion," and then feed this into the text encoder to generate the text embedding corresponding to the "Explosion" category. c represents the corresponding category. For visual feature construction, we extract the corresponding visual features f from each video frame using an image encoder. img .

[0071] S2-2: Text-Visual Feature Similarity Calculation. To calculate the probability of each frame of a video belonging to each category, it is necessary to calculate the similarity between the visual features of the video frame and the text features of each category. This can be achieved by calculating cosine similarity:

[0072]

[0073] Here, τ is a temperature coefficient, and sim(·) represents the cosine similarity. Based on this similarity value, a probability distribution can be assigned to each frame, representing the likelihood that it belongs to each category.

[0074] S2-3: Fragment-level pseudo-label generation and ensemble. For each video v in the dataset... i All contain t i The video consists of several non-overlapping video segments, each labeled with a video-level tag indicating its category as c. Taking a single video segment as an example, the probability of each frame within that segment belonging to category c is calculated using S2-2. Next, the highest anomalous segment score among these frames is taken as the probability that the segment belongs to category c. This process is repeated sequentially for each segment in the video, ultimately yielding video v. i The corresponding fragment-level pseudo-label P′.

[0075] The quality of pseudo-labels directly affects the training effect of the model. To further improve the accuracy and stability of pseudo-labels, we propose a simple and efficient pseudo-label ensemble strategy, which integrates pseudo-labels generated by multiple different visual encoders. The core idea of ​​this strategy is to improve the accuracy and stability of pseudo-labels by combining pseudo-labels generated by multiple visual encoders. Specifically, N different visual encoders are used, which independently generate their own pseudo-labels using a unified simple cue template ("a video of a [CLS]"), i.e., {P′1, P′2, ..., P′...}. N During the integration process, an averaging strategy was employed to average the probabilities output by each visual encoder, thereby reducing the bias and noise that might be introduced by a single model. This process can be mathematically expressed as follows:

[0076]

[0077] Where, P∈R T N represents the number of visual encoders, P′ i This represents the pseudo-label generated by the i-th visual encoder. After obtaining the ensembled fragment-level pseudo-label P, it is binarized using a threshold (e.g., 0.5) and directly used to supervise the subsequent training process.

[0078] S3: Multi-scale temporal feature modeling. First, S1 preprocessing is used to obtain the v of each video. i Corresponding fragment features As input, a multi-scale temporal feature network is constructed to process feature x. i Perform multi-scale modeling from coarse-grained to fine-grained. The steps include:

[0079] S3-1: Feature Mapping Layer. First, x... i The length of the segment t i The length is fixed at T. Specifically, if the video contains fewer than T segments, it is padded with zeros to a length of T. Otherwise, it is downsampled to a predefined length T. A feature mapping layer f is then introduced. proj , used to convert the input features x i It maps to a feature dimension suitable for multi-scale Transformer encoders. It consists of a shallow convolutional neural network that uses ReLU as the activation function to map high-dimensional input features to a low-dimensional embedding space. Its mathematical expression is as follows:

[0080] Z0 = f proj (x i )

[0081] Where Z0∈R T×FThis represents the video features after dimensionality reduction mapping, where F = 256 is the feature dimension after mapping.

[0082] S3-2: Multi-scale Transformer Encoder. It consists of multiple stacked Transformer encoders, each layer embedding a Layer Normalization (LN), a Local Multi-Head Self-Attention (LMSA) module, and a Multilayer Perceptron (MLP). For the input video feature sequence Z0, depth information is first extracted and interactively processed using LMSA. Then, residual connections and layer normalization operations are employed to ensure effective integration and stable propagation of information. Next, the GELU activation function is used to perform a nonlinear transformation on the output. To capture segment features at different time scales, ↓(·) downsampling is used to obtain feature representations at different time scales.

[0083]

[0084] in, T l / T l-1 It's the downsampling rate. α l , It is a learnable scaling factor. Considering computational efficiency, ↓(·) downsampling uses depthwise convolution, with a downsampling rate of 2 between each scale, i.e., T. l / T l-1 =2. Finally, the multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Z} was obtained. L}, where L represents the number of layers in the pyramid.

[0085] S3-3: Lightweight convolutional decoder. After obtaining the multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Z...} L Afterwards, a lightweight convolutional decoder consisting of one-dimensional convolutions is used to decode to obtain the final scores of anomalous segments at various granularities, and a classifier f is appended to them. cls To predict the probability of an anomalous instance occurring at each time step at each granularity, the mathematical expression is as follows:

[0086] S l =σ(f cls (Z l ;φ cls ),l=1…L

[0087] Among them, f cls It is implemented by three 1D convolutions with a stride of 3, layer normalization, and the ReLU activation function; φ cls σ represents the learnable parameters; σ is the sigmoid activation function. The obtained... Let S represent the score of the outlier segment at layer l. Thus, we obtain the outlier segment scores {S1, S2, ..., S...} at each granularity obtained through temporal feature pyramid decoding. L}

[0088] S3-4: Adaptive weight fusion mechanism. The scores of outlier segments at various granularities {S1, S2, ..., S...} are obtained through a lightweight convolutional decoder and classifier. L As input, in the initial stage of training, a learnable weight matrix β = {β1, β2, ..., β} is first initialized. L}, where β L Let represent the fusion weights of the Lth layer. These weights are uniformly initialized to a unit value of 1, indicating that in the initial stage of training, outlier scores at each granularity are equally important. The model is then iteratively trained and optimized until convergence is achieved. Its mathematical expression is as follows:

[0089]

[0090] Where S∈R T R represents the final score for the abnormal segments, where T represents the number of video segments, and R represents the score for the abnormal segments. T It is a data representation method that indicates that the fragment anomaly score S is a one-dimensional data of length T, without any practical meaning.

[0091] S4: Classifier Optimization. Under weak supervision, video anomaly detection is treated as a Multiple Instance Learning (MIL) task. This means that during training, the classifier can only rely on limited label information (video-level labels) to learn to recognize anomalies. To better distinguish between anomalous and non-anomalous action patterns, we employ a pseudo-label classification loss function, a dynamic ranking loss function for representative segments, and a triplet loss function to optimize the classifier.

[0092] S4-1: Pseudo-label classification loss function L pse We address the data imbalance problem using the Focal Loss function, which is designed to solve the issue of insufficient model recognition of minority class samples caused by class imbalance in classification problems. In video anomaly detection tasks, introducing Focal Loss allows the classifier to focus more on those difficult-to-identify abnormal behavior samples, thereby optimizing the learning of minority class samples.

[0093] Specifically, in S3-4, the anomaly score S for each video segment is obtained; in S2, the segment-level pseudo-label is obtained through video language cue learning and pseudo-label generation; and the predicted anomaly segment score is directly optimized using Focal Loss. Its mathematical expression is as follows:

[0094] L pse=Focal(S,P)

[0095] in, T represents the number of video clips.

[0096] S4-2: Represents the fragment dynamic loss function L cls Suppose that for an input video v i It contains t i There are 1 segment, and the corresponding video-level tag is y. i Select the top-k examples (i.e., representative segments) with the highest predicted anomaly scores, among which... And use their mean as the video-level prediction score s i The binary cross-entropy (BCE) loss function is used to optimize the classifier, and its mathematical expression is as follows:

[0097] L cls =BCE(s) i ,y i )

[0098] Among them, y i ∈{0,1}, α is a hyperparameter used to control the number of segments selected.

[0099] S4-3: Triple Loss Function L trip For each video v i First, based on the predicted outlier segment score S, the top-k examples (k value is consistent with S4-2) with the highest outlier segment scores are determined. The feature representations of these examples will be used as anchor points f. a Then, the feature centers of these anomalous examples are calculated and used as positive samples f. p Then, the feature centers of the normal examples are selected as negative samples f. n Its mathematical expression is as follows:

[0100] L trip =Triplet(f a ,f p ,f n )

[0101] Among them, f a It is the feature representation of the selected example. Similarly, f p and f n The feature centers of positive and negative samples are selected. By minimizing the distance between the anchor point and the positive sample, while maximizing the distance between the anchor point and the negative sample, better inter-class separation and intra-class compactness can be achieved in the feature space. This ultimately results in a more refined feature representation and a more discriminative feature distribution.

[0102] S4-4: Overall Loss Function L total During training, a combined loss function is constructed by integrating the pseudo-label classification loss function, the dynamic ranking loss function for representative segments, and the triplet loss function. To balance these three loss functions, this section treats them as independent sub-tasks. We employ a multi-task learning approach, enabling the model to adaptively determine the weight relationships of each loss function, thus achieving joint optimization and learning. Its mathematical expression is as follows:

[0103]

[0104] Where, λ i ,i∈{1,2,3} are the learnable loss function weights.

[0105] S5: For a new video, use the method in S1 to obtain x′. i , x′ i The input is a pre-trained multi-scale temporal feature network, which is then passed through a pre-trained lightweight convolutional decoder and classifier to output the predicted category.

[0106] experiment:

[0107] Table 1. Experimental results on the XD-VIOLENCE and UCF-CRIME datasets.

[0108]

[0109] Table 2 shows the effectiveness of the proposed modules. MSIL: coarse-grained and fine-grained fusion method; PL: pseudo-label; √ indicates "contains".

[0110]

[0111] Table 3. Effectiveness of different loss functions. √ indicates "includes".

[0112]

[0113] Table 4. Ablation study of pseudo-label integration strategy. All models use the same prompt template (i.e., "a video of [CLS]").

[0114]

[0115]

[0116] This invention was evaluated on two benchmark datasets for anomaly detection: XD-Violence and UCF-Crime. XD-Violence is a large collection of 4754 unedited videos, totaling 217 hours in length, sourced from various media including movies, sports broadcasts, surveillance footage, and CCTV. This dataset includes a training set of 3954 videos with video-level labels and a test set of 800 videos with frame-level labels. UCF-Crime is a large-scale anomaly detection dataset containing 1900 videos, including 13 types of anomalous events captured by surveillance cameras in both indoor and outdoor daytime and nighttime scenes. The training set with video-level labels contains 1610 videos, while the test set with frame-level labels contains 290 videos.

[0117] This invention uses Average Precision (AP) as the metric to evaluate the detection performance of XD-Violence. Simultaneously, we also utilize the Area Under of Curve (AUC) corresponding to the Receiver Operating Characteristic (ROC) curve to evaluate the performance of UCF-Crime. Higher AP and AUC indicate better and more stable algorithm performance. The sampling segment length T for UCF-Crime and XD-Violence was set to 128 and 256, respectively, with α = 10. Iterative optimization was performed using the AdamW optimizer, with an initial learning rate of 1e-5 for UCF-Crime and 5e-5 for XD-Violence, and a weight decay of 1e-2. Training on both datasets was performed on an RTX 3090 GPU with a batch size of 30.

[0118] Table 1 shows the experimental results of the present invention and other methods in the same field on the XD-VIOLENCE and UCF-CRIME datasets. The bold text indicates the highest detection accuracy. Obviously, the detection accuracy of the present invention is better than other methods, which proves the effectiveness of the present invention.

[0119] Table 2 illustrates the performance improvements of each module and its combination in this invention, and provides a comparative experiment with a single-scale feature map (L=1), i.e., the Baseline model in Table 2. Adding MSIL improved the AP of XD-Violence by 2.7% and the AUC of UCF-Crime by 0.58%, highlighting its effectiveness in capturing both short-term and long-term anomalous instances. Furthermore, adding pseudo-labels generated by visual language cues significantly improved the AP by 4.27% and the AUC by 1.31%, demonstrating its effectiveness in guiding the model to focus on potentially anomalous segments rather than simple segments. The best performance was achieved on both datasets when both MSIL and pseudo-labels were added simultaneously, demonstrating the complementarity between these two branches.

[0120] Table 3 shows the impact of different combinations of loss functions of this invention on the complete model. The baseline model uses only L... cls As a supervisor. By comparing the first two lines, we can find L trip This resulted in a satisfactory performance improvement, highlighting its effectiveness in distinguishing between normal and abnormal distances. The results for the first three cases demonstrate that L... pse This approach is most effective in guiding the network to achieve better performance, thus validating the effectiveness of visual language cues in generating pseudo-labels. Overall, performance is optimal when all three loss functions are used, indicating that they are all effective.

[0121] Table 4 presents the ablation study results of the pseudo-label ensemble strategy. This invention uses CLIP models with different visual architectures for pseudo-label generation and quantitatively evaluates our pseudo-label ensemble strategy. For all models, we used a simple cue template (i.e., "a video of a [CLS]"), and the final ensemble strategy achieved the best results, demonstrating its importance in generating high-quality pseudo-labels.

Claims

1. A video anomaly event detection method based on cue learning and multi-scale temporal fusion, characterized in that: Includes the following steps: S1: Obtain a dataset consisting of multiple videos, and preprocess each video segment. The features of the video segment obtained after preprocessing are denoted as follows: ,in Indicates will The number of video segments that are divided into multiple consecutive frames that do not overlap. Indicates feature dimension; S2: Generate pseudo-labels for each video in the dataset; S3: Construct a multi-scale temporal feature network for... Multi-scale modeling from coarse-grained to fine-grained is performed to obtain a multi-scale temporal feature pyramid. , The lightweight convolutional decoder obtains the final features at various scales, and the final features are then passed through a classifier to obtain the anomaly segment scores at each granularity. Then, an adaptive weight fusion mechanism is used to obtain... Final abnormal segment score A lightweight convolutional decoder and the classifier following it constitute a classification network. ; In step S3, constructing a multi-scale temporal feature network includes: Multi-scale temporal feature modeling first uses S1 preprocessing to obtain each video Corresponding fragment features As input, features Perform multi-scale modeling from coarse-grained to fine-grained; S3-1: Feature mapping layer, fragment length Fixed as If the video Contains fewer than If there are 1, then fill with zeros to the length. Conversely, Downsampling to a predefined length Subsequently, a feature mapping layer was introduced. , used to input features Map to feature dimensions suitable for multi-scale Transformer encoders; A shallow convolutional neural network, using ReLU as the activation function, maps high-dimensional input features to a low-dimensional embedding space. Its mathematical expression is as follows: in, express Video features after dimensionality reduction mapping It is the feature dimension after mapping; S3-2: The multi-scale Transformer encoder consists of multiple stacked Transformer encoders, each layer embedding a layer normalization layer (LN), a local multi-head self-attention module (LMSA), and a multilayer perceptron (MLP); for the input... First, deep information is extracted and processed interactively using LMSA. Then, residual connections and layer normalization are employed to ensure effective information integration and stable propagation. Next, the GELU activation function is used to perform a nonlinear transformation on the output. Downsampling is used to obtain feature representations at different time scales; in, , It is the downsampling rate. This represents the features of the feature pyramid at the next higher level. This represents the features obtained after passing through the first Transformer encoder. This represents the features obtained after passing through the second Transformer encoder. This indicates the final characteristic of this layer of the pyramid. It is a learnable scaling factor. Downsampling is achieved using depthwise convolution, ultimately resulting in a multi-scale temporal feature pyramid. ,in Represents the number of levels in the pyramid; S3-3: A lightweight convolutional decoder consisting of one-dimensional convolutions is used to decode the final features at each scale, and a classifier is appended to it to predict the anomalous segment score of anomalous instances at each time step at each granularity. The mathematical expression is as follows: in, It is implemented by three 1D convolutions with a stride of 3, layer normalization, and ReLU activation function; Indicates learnable parameters; The sigmoid activation function is used to obtain... Indicates the first The score of the abnormal fragments at each layer; thus, the scores of abnormal fragments at each granularity obtained through temporal feature pyramid decoding are obtained. ; S3-4: Adaptive weight fusion mechanism, As input, a learnable weight matrix is ​​defined. ,in Indicates the first Layer fusion weights, final abnormal fragment scores The mathematical expression is as follows: in, This represents the final score for the outlier segment. Indicates the number of video clips. It is a method of representing data, without any actual meaning; S4: Design the overall loss function The loss function is used to update the multi-scale temporal feature network and classification network in S3. The parameters are adjusted until the loss no longer changes, resulting in a well-trained multi-scale temporal feature network, a lightweight convolutional decoder, and a classifier. S5: For a new video, obtain the result using the method in S1. ,Will The input is a pre-trained multi-scale temporal feature network, which is then passed through a pre-trained lightweight convolutional decoder and classifier to output the predicted category.

2. The video anomaly detection method based on cue learning and multi-scale temporal fusion as described in claim 1, characterized in that, In step S1, the video preprocessing includes: For a video in the dataset First, divide the video into A video segment consisting of 16 consecutive, non-overlapping frames is fed into the I3D network for visual feature extraction. The video segment features obtained after global average pooling in the Mixed_5c layer of the I3D network are denoted as follows: ,in The above operation is performed on each video in the dataset to obtain the visual features of all videos in the dataset.

3. The video anomaly detection method based on cue learning and multi-scale temporal fusion as described in claim 1, characterized in that, In step S2, generating pseudo-labels for each video in the dataset includes: S2-1: Using the CLIP model's text encoder and image encoder respectively, construct corresponding text features and visual features for each video in the dataset; for text feature construction, use the text prompt template-based method commonly used in natural language processing to obtain text embeddings. , This represents the corresponding category; for visual features, the corresponding visual features are extracted from each video using an image encoder. ; S2-2: Text-visual feature similarity calculation, using cosine similarity. To achieve: in, It is a temperature coefficient. The cosine similarity is used to assign a probability distribution to each frame based on this similarity value, representing the likelihood that it belongs to each category. S2-3: Fragment-level pseudo-label generation and ensemble: For each video in the dataset All include Each video clip is a unique, non-overlapping video segment, and each video is accompanied by a video-level tag indicating its category. The category of each frame within a segment is calculated using S2-2. The probability is then calculated, and next, the highest score of the anomalous segment in these frames is taken as the category to which that segment belongs. The probability is determined by repeating the above process sequentially for each segment of the video, ultimately yielding the final video. Corresponding fragment-level pseudo-tags ; use A different visual encoder, Each different visual encoder independently generates its own pseudo-label using a unified, simple cue template, denoted as . During the integration process, an averaging strategy is employed to average the probabilities output by each visual encoder, mathematically expressed as follows: in, , Representing the The pseudo-labels generated by the visual encoder are then integrated into fragment-level pseudo-labels. Then, it is binarized by applying a threshold.

4. The video anomaly detection method based on cue learning and multi-scale temporal fusion as described in claim 1, characterized in that: In S4, the overall loss function for: Overall loss function This includes pseudo-label classification loss function, dynamic ranking loss function for representative segments, and triplet loss function; Pseudo-label classification loss function The mathematical expression for optimizing predicted outlier segment scores using Focal Loss is as follows: Representative fragment dynamic loss function The binary cross-entropy loss function is used to optimize the classifier, and its mathematical expression is as follows: in, express middle The tags corresponding to each fragment , It is a hyperparameter used to control the number of segments selected. The segment with the highest predicted anomaly score, among which and will The average of the predicted outlier scores is used as the video-level predicted score. ; S4-3: Triple Loss Function For each video First, based on the predicted final abnormal segment score Determine The segment with the highest score among the abnormal segments; The feature representation of the segment with the highest score among the abnormal segments is used as the anchor point. Then, the feature centers of these abnormal fragments are calculated and used as positive samples. Then, the feature centers of the normal segments are selected as negative samples. , The mathematical expression is as follows: Overall loss function : in, , These are learnable loss function weights.

Images