Video abnormal event detection method based on prompt learning and multi-scale time sequence fusion
By fusion of prompt learning and multi-scale timing, using the CLIP model to generate pseudo-labels and multi-scale timing relationship modeling, the identification problem of video anomaly event detection in the complex background in the prior art is solved, and high accuracy and Robust detection effect.
Patent Information
- Application Number
- CN202410943705.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-15
AI Technical Summary
Existing video anomaly event detection methods are difficult to fully consider the recognition of long-term or short-term abnormal segments in complex processing contexts, and excessive reliance on visual semantic features leads to a high false alarm rate, affecting detection accuracy.
Using a method based on the integration of prompt learning and multi-scale timing, the visual language pre-trained model CLIP generates pseudo-label guidance model training, and finely model video timing information through the multi-scale timing relationship modeling module, combining pseudo-label integration strategy and adaptive Weight fusion mechanism improves the accuracy and robustness of detection.
It significantly improves the accuracy and robustness of video abnormal event detection, and can efficiently identify long and short-term abnormal segments in complex backgrounds, reduce false alarm rates, and enhance the ability to distinguish abnormal and normal modes.
Smart Images

Figure CN118918506A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of video anomaly detection, visual language model and multi-instance learning, and in particular to a video anomaly event detection method based on prompt learning and multi-scale time series fusion. Background Art
[0002] As a core technology in the field of intelligent monitoring, video abnormal event detection mainly relies on computer vision technology to implement real-time analysis of surveillance videos. It plays a decisive role in improving early warning efficiency, strengthening pre-emptive prevention, and strengthening post-event tracking. However, due to the complexity and diversity of video abnormal events, its research process still encounters multiple challenges. The first challenge is that the current mainstream methods face many challenges in deeply mining and utilizing the temporal information of videos, which makes it difficult for the model to achieve a comprehensive and balanced effect when dealing with the recognition task of long or short abnormal clips. Especially in complex background environments, the boundaries between abnormal and normal events become increasingly blurred, increasing the difficulty of detection. In addition, over-reliance on visual semantic features may cause the model to be limited to simple scenes during recognition. For abnormal videos containing complex contextual information, the model may show a high false alarm rate, which in turn affects the accuracy of anomaly detection. Summary of the invention
[0003] 1. Technical issues to be solved
[0004] In view of the shortcomings of the existing technology, the present invention proposes a video abnormal event detection method based on prompt learning and multi-scale time series fusion, focusing on exploring how to make full use of the visual language pre-training model CLIP to enhance the accuracy and robustness of the existing model, and studying how to process video timing information to enhance the ability to distinguish abnormal and normal modes.
[0005] (II) Technical solution
[0006] In order to solve the above problems, the present invention transforms the anomaly detection task into a zero-sample learning problem, generates pseudo labels to guide model training with the help of the visual language large model CLIP, enhances the quality of pseudo labels through pseudo label integration strategy, and introduces a multi-scale temporal relationship modeling module to finely model the coarse and fine granularity information of the video time series, so as to achieve high-accuracy detection of long and short time series abnormal segments. The specific technical solutions adopted are as follows:
[0007] A method for detecting abnormal events in videos based on cue learning and multi-scale temporal fusion, comprising the following steps:
[0008] S1: Video data preprocessing. For a video v in the dataset i , first divide the video into t iA video clip containing 16 consecutive frames that do not overlap is sent to the I3D network for visual feature extraction. The video clip features obtained after global average pooling of the Mixed_5c layer in the I3D network are recorded as Where d = 1024. The above operation is performed on each video in the dataset, and finally the visual features of all videos in the dataset are obtained, which are used as input for subsequent processing.
[0009] S2: Video language cue learning pseudo-label generation. We designed a plug-and-play modular visual language cue learning module that can be easily integrated into the existing video anomaly event detection model structure. We use the powerful zero-shot learning capability of the visual language pre-training model (such as CLIP) to generate pseudo-labels to guide model training, thereby transforming the weakly supervised learning environment with scarce labeled data into a supervised task environment with rich supervisory signals.
[0010] S2-1: Construction of text features and visual features. The constructed text prompt template is input into the text encoder of the CLIP model to obtain the corresponding text features. At the same time, the video frame is input into the image encoder of the CLIP model to obtain the corresponding visual features. For text feature construction, we use the method based on text prompt template (PromptTemplate) commonly used in the field of natural language processing. Assume that for a target dataset containing C categories, the designed prompt template (for example: "a video of [CLS]") is used to construct text for C categories. For example, for Explosion, it is constructed as "a video of explosion" and then sent to the text encoder to generate the text embedding corresponding to the Explosion category. c represents the corresponding category. For visual feature construction, we extract the corresponding visual feature f from each video frame through an image encoder. img .
[0011] S2-2: Text-visual feature similarity calculation. In order to calculate the probability that each frame of the video belongs to each category, it is necessary to calculate the similarity between the visual features of the video frame and the text features of each category. This can be achieved by calculating the cosine similarity:
[0012]
[0013] Among them, τ is a temperature coefficient and sim(·) represents cosine similarity. Based on this similarity value, a probability distribution can be assigned to each frame, indicating the likelihood that it belongs to each category.
[0014] S2-3: Segment-level pseudo-label generation and integration. For each video v in the dataset i All include t i There are non-overlapping video clips, and each video is equipped with a video-level label, and its annotated category is c. Take a clip in the video as an example, and calculate the probability that each frame in the clip belongs to category c through S2-2. Next, take the highest abnormal clip score among these frames as the probability that the clip belongs to category c. Thus, the above process is repeated for each clip in the video in sequence, and finally the video v is obtained. i The corresponding segment-level pseudo-label P′.
[0015] The quality of pseudo-labels directly affects the training effect of the model. In order to further improve the accuracy and stability of pseudo-labels, we propose a simple and efficient pseudo-label integration strategy, which is to integrate pseudo-labels generated by multiple different visual encoders. The core idea of this strategy is to improve the accuracy and stability of pseudo-labels by combining pseudo-labels generated by multiple visual encoders. Specifically, N different visual encoders are used, and these encoders use a unified simple prompt template ("a video of a[CLS]") to independently generate their own pseudo-labels, that is, {P′1,P′2,…,P′ N In the integration process, an averaging strategy is adopted to average the probabilities output by each visual encoder, thereby reducing the deviation and noise that may be brought by a single model. This process can be mathematically expressed as follows:
[0016]
[0017] Where P∈R T , N represents the number of visual encoders, P′ i Represents the pseudo label generated by the i-th visual encoder. After obtaining the integrated segment-level pseudo label P, it is binarized by a threshold (such as 0.5) and used directly to supervise the subsequent training process.
[0018] S3: Multi-scale temporal feature modeling. i Perform multi-scale modeling from coarse-grained to fine-grained to obtain a multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Z L}, Z={Z1,Z2,…,Z L The final features of each scale are obtained through the lightweight convolution decoder, and the final features are passed through the classifier to obtain the abnormal fragment scores of each granularity {S1,S2,…,S L}, and then use the adaptive weight fusion mechanism to get v i The final abnormal fragment score S is obtained by:
[0019] S3-1: Feature mapping layer. First, xi The length of the fragment t i is fixed to T. Specifically, if the video contains less than T segments, it is padded with zeros to a length of T. Otherwise, it is downsampled to a predefined length of T. A feature mapping layer f is then introduced proj , used to transform the input feature x i Mapped to feature dimensions suitable for processing by the multi-scale Transformer encoder. It consists of a shallow convolutional neural network and uses ReLU as the activation function to map high-dimensional input features to a low-dimensional embedding space. Its mathematical expression is as follows:
[0020] Z0=f proj (x i )
[0021] Among them, Z0∈R T×F It represents the video features after dimensionality reduction mapping, and F=256 is the feature dimension after mapping.
[0022] S3-2: Multi-scale transformer encoder. It is composed of multiple stacked transformer encoders, each of which has embedded layer normalization LN, local multi-head self-attention module LMSA and multi-layer perceptron MLP. For the input video feature sequence Z0, deep information extraction and interaction processing are first performed through LMSA, and then residual connection and layer normalization operations are used to ensure effective integration and stable propagation of information. Next, the GELU activation function is used to perform nonlinear transformation on the output. In order to capture the features of fragments at different time scales, feature representations at different time scales are obtained by downsampling ↓(·).
[0023]
[0024] in, T l / T l-1 is the downsampling rate. l , is a learnable scaling factor. Considering computational efficiency, ↓(·) downsampling is implemented using deep convolution, and the downsampling rate between each scale is 2, that is, T l / T l-1 =2. Finally, we get a multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Z L}, where L represents the number of layers of the pyramid.
[0025] S3-3: Lightweight convolution decoder. After obtaining the multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Z L}, a lightweight convolution decoder consisting of one-dimensional convolution is used to decode to obtain the final abnormal fragment scores of each granularity, and a classifier f is appended to it. cls To predict the probability of abnormal instances occurring at each moment at each granularity, the mathematical expression is as follows:
[0026] S l =σ(f cls (Z l ; φ cls )),l=1…L
[0027] Among them, f cls It is implemented by 3 1D convolutions with a step size of 3, layer normalization, and ReLU activation function; φ cls represents the learnable parameter; σ is the sigmoid activation function. represents the abnormal fragment score of the lth layer. Thus, the abnormal fragment scores of each granularity {S1,S2,…,S L}.
[0028] S3-4: Adaptive weight fusion mechanism. The scores of abnormal segments of each granularity {S1, S2, …, S L} as input, in the initial stage of training, first initialize a learnable weight matrix β = {β1,β2,…,β L}, where β L represents the fusion weight of the Lth layer, and these weight coefficients are uniformly initialized to the unit value 1, which means that the anomaly scores of each granularity are equally important in the initial stage of training, and then the training iterative optimization model is trained until it reaches a convergence state. Its mathematical expression is as follows:
[0029]
[0030] Among them, S∈R T represents the final abnormal segment score, and T represents the number of video segments.
[0031] S4: Classifier optimization. In the weakly supervised setting, video abnormal event detection is considered as a multi-instance learning (MIL) task. This means that during training, the classifier can only rely on limited label information (video-level labels) to learn to identify abnormal events. In order to better distinguish abnormal and non-abnormal action patterns, we use pseudo-label classification loss function, representative segment dynamic ranking loss function and triple loss function to optimize the classifier.
[0032] S4-1: Pseudo-label classification loss function L pseWe use the Focal Loss loss function to solve the data imbalance problem. Its original design intention is to solve the problem of insufficient model recognition of minority class samples caused by class imbalance in classification problems. In the task of video abnormal event detection, the introduction of Focal Loss can make the classifier pay more attention to those abnormal behavior samples that are difficult to identify, thereby optimizing the learning of minority class samples.
[0033] Specifically, the anomaly score S of each video segment is obtained in S3-4, and the segment-level pseudo-label P is obtained in S2 video language cue learning pseudo-label generation. The predicted abnormal segment score is directly optimized by Focal Loss, which is mathematically expressed as follows:
[0034] L pse =Focal(S,P)
[0035] in, T represents the number of video clips.
[0036] S4-2: represents the segment dynamic loss function L cls . Assume that for an input video v i , which contains t i clips, the corresponding video-level label is y i . Select the top-k examples (i.e., representative fragments) with the highest predicted anomaly scores, where And take its mean as the video-level prediction score s i The binary cross-entropy (BCE) loss function is used to optimize the classifier, and its mathematical expression is as follows:
[0037] L cls =BCE(s i ,y i )
[0038] Among them, y i ∈{0,1}, α is a hyperparameter that controls the number of fragment selections.
[0039] S4-3: Triplet loss function L trip For each video v i First, we determine the top-k examples (the k value is consistent with S4-2) with the highest abnormal fragment scores based on the predicted abnormal fragment score S. The feature expressions of these examples will be used as anchor points f a , and then calculate the feature centers of these abnormal examples and use them as positive samples f p , and then select the feature center of the normal example as the negative sample f n . Its mathematical expression is as follows:
[0040] Ltrip =Triplet(f a ,f p ,f n )
[0041] By minimizing the distance between the anchor point and the positive sample and maximizing the distance between the anchor point and the negative sample, better inter-class separation and intra-class compactness can be achieved in the feature space, ultimately obtaining a more refined feature expression and a more discriminative feature distribution.
[0042] S4-4: Overall loss function L total During the training process, the overall loss function is constructed by combining the pseudo-label classification loss function, the representative fragment dynamic ranking loss function, and the triple loss function. In order to balance the three loss functions, this section regards the three loss functions as independent subtasks. We use a multi-task learning method to enable the model to adaptively determine the weight relationship between each loss function, thereby performing joint optimization and learning. Its mathematical expression is as follows:
[0043]
[0044] Among them, λ i ,i∈{1,2,3} is the learnable loss function weight.
[0045] S5: For a new video, use the method of S1 to obtain x′ i , x′ i The trained multi-scale temporal feature network is input, and then after the trained lightweight convolutional decoder and classifier, the predicted category is output.
[0046] Compared with the prior art, the present invention has at least the following advantages:
[0047] The present invention designs an abnormal event detection method based on visual language cue learning, which uses the CLIP model to generate high-quality pseudo labels through visual language cue learning, so that the weakly supervised task with insufficient labels is transformed into a supervised task with a large amount of supervisory signals, which greatly improves the accuracy of detection. At the same time, the present invention also designs a video anomaly detection method based on multi-scale real-sequence fusion, which uses a multi-scale temporal feature pyramid to model the video temporal features from coarse-grained to fine-grained, introduces the triple loss function in metric learning to improve the distinguishability between abnormal and normal pattern features, and uses an adaptive learning weight mechanism to fuse fine-grained detection results at different levels, so as to achieve effective detection of both long-series and short-series abnormal fragments with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present invention and the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0049] Figure 1 It is a simplified flow chart of the method of the present invention.
[0050] Figure 2 Schematic diagram of the abnormal event detection framework based on multi-scale time series fusion.
[0051] Figure 3 It is a flowchart of visual feature extraction.
[0052] Figure 4 It is a schematic diagram of a multi-scale temporal feature pyramid network.
[0053] Figure 5 Schematic diagram of the multi-scale Transformer encoder structure.
[0054] Figure 6 It is a schematic diagram of the adaptive weight fusion mechanism.
[0055] Figure 7 Schematic diagram of an abnormal event detection method based on visual language cue learning. DETAILED DESCRIPTION
[0056] The present invention is described in further detail below.
[0057] In view of the limitation of relying solely on visual features to process complex scenes, the visual language pre-training model CLIP can effectively solve this problem. By converting the anomaly detection task into a zero-shot learning problem, using the visual language cue learning mechanism to generate pseudo labels to guide model training, and using the pseudo label integration strategy to enhance the quality of pseudo labels, the accuracy and robustness of anomaly detection can be enhanced. At the same time, for the deficiencies in mining and utilizing video timing information, the multi-scale temporal relationship modeling module can solve this problem by finely modeling the coarse and fine granularity information of the video timing. The coarse-grained timing features capture persistent abnormal events, while the fine-grained features focus on accurately locating transient abnormal fragments. In order to enhance the ability to distinguish abnormal and normal patterns under complex backgrounds, the triplet loss function is used to improve the distinguishability of features, and the fine-grained detection results at different levels are fused through an adaptive learning weight mechanism to achieve high-accuracy detection of long and short time series abnormal fragments.
[0058] The present invention proposes a method for detecting abnormal events based on visual language cue learning. With the help of visual language cue learning, the method uses the CLIP model to generate high-quality pseudo labels, successfully transforming the weakly supervised task with insufficient labels into a supervised task with rich supervisory signals, greatly improving the accuracy of detection. At the same time, the present invention also constructs an abnormality detection method based on multi-scale time series fusion. The method uses multi-scale time series feature pyramid technology to achieve layer-by-layer modeling of video time series features from coarse-grained to fine-grained. By introducing the triple loss function in metric learning, the method significantly improves the ability to distinguish between abnormal and normal pattern features. And the adaptive learning weight mechanism is used to effectively fuse the fine-grained detection results at different levels, achieving high-accuracy detection of long and short time series abnormal fragments.
[0059] Figure 2 The abnormal event detection framework based on multi-scale time series fusion in the present invention is demonstrated. For the original surveillance video, video feature extraction is first required, and the extracted features are modeled with multi-scale time series features to further enrich the feature information. The detection results of each scale feature are adaptively fused at coarse and fine granularity to obtain the final abnormal segment score.
[0060] Figure 3 The process of extracting visual features for a single video is shown. For a video v in the dataset i , first divide the video into t i A video clip containing 16 consecutive frames that do not overlap is sent to the I3D network for visual feature extraction. The video clip features obtained after global average pooling of the Mixed_5c layer in the I3D network are recorded as Where d = 1024. The above operation is performed on each video in the dataset, and finally the visual features of all videos in the dataset are obtained, which are used as input for subsequent processing.
[0061] Figure 4 The multi-scale temporal feature pyramid network of the present invention is presented. The video feature extraction step is used to obtain each video v i Corresponding fragment features As input, a multi-scale temporal feature network is constructed. i Multi-scale modeling from coarse-grained to fine-grained is performed. The network mainly includes: feature mapping layer (Projection), multi-scale Transformer encoder (Encoder), and lightweight convolution decoder (Decoder).
[0062] Figure 5The multi-scale Transformer encoder structure of the present invention is shown. It is composed of multiple stacked Transformer encoders, each of which has layer normalization (Layer Normalization), local multi-headed self-attention module (Local Multi-Headed Self-Attention, LMSA) and multilayer perceptron (Multilayer Perceptron, MLP) embedded in each layer.
[0063] Figure 6 The adaptive weight fusion mechanism process of the present invention is demonstrated. The abnormal fragment scores of each granularity {S1, S2, ..., S L} as input, in the initial stage of training, first initialize a learnable weight matrix β = {β1,β2,…,β L}, where β l Represents the fusion weight of the lth layer, and these weight coefficients are uniformly initialized to the unit value 1, which means that the anomaly scores of each granularity are equally important in the initial stage of training. Subsequently, the model is iteratively optimized until convergence is reached.
[0064] Figure 7 The present invention presents an abnormal event detection method based on visual language cue learning. The method is a plug-and-play modular visual language cue learning module. First, text features and visual features are constructed. The preprocessed text and video are used as input, and the image encoder and text encoder of the CLIP model are used to construct the corresponding text and visual features. In order to calculate the probability that each frame belongs to each category, the similarity between the visual features of the video frame and the text embedding of each category needs to be calculated. This can usually be achieved by calculating cosine similarity or other similarity metrics. Based on these similarity values, a probability distribution can be assigned to each frame, indicating the possibility that it belongs to each category.
[0065] For each video v in the dataset i All include t i There are non-overlapping video clips, and each video is equipped with a video-level label, and its annotated category is c. Take a clip in the video as an example, and use the above probability formula to calculate the probability that each frame in the clip belongs to category c. Next, take the highest abnormal clip score among these frames as the probability that the clip belongs to category c. Thus, the above process is repeated for each clip in the video in sequence, and finally the video v is obtained. i The corresponding segment-level pseudo-label P′.
[0066] A simple and efficient pseudo-label integration strategy is proposed, which integrates pseudo-labels generated by multiple different visual encoders. The core idea of this strategy is to improve the accuracy and stability of pseudo-labels by combining pseudo-labels generated by multiple visual encoders. Specifically, N different visual encoders are used, and these encoders use a unified simple prompt template ("a video of a[CLS]") to independently generate their own pseudo-labels, namely {P1′,P2′,...,P′ N During the integration process, an averaging strategy is adopted to average the probabilities output by each visual encoder, thereby reducing the bias and noise that may be brought by a single model.
[0067] The present invention will be further described with reference to the accompanying drawings and embodiments. Figure 1 As shown, a video abnormal event detection method based on prompt learning and multi-scale time series fusion includes the following steps.
[0068] S1: Video data preprocessing. For a video v in the dataset i , first divide the video into t i A video clip containing 16 consecutive frames that do not overlap is sent to the I3D network for visual feature extraction. The video clip features obtained after global average pooling of the Mixed_5c layer in the I3D network are recorded as Where d = 1024. The above operation is performed on each video in the dataset, and finally the visual features of all videos in the dataset are obtained, which are used as input for subsequent processing.
[0069] S2: Generate pseudo labels for each video in the dataset, i.e., video language cue learning pseudo label generation. We designed a plug-and-play modular visual language cue learning module, which can be easily integrated into the existing video anomaly event detection model structure, and uses the powerful zero-shot learning capability of the visual language pre-training model (such as CLIP) to generate pseudo labels to guide model training, thereby transforming the weakly supervised learning environment with scarce labeled data into a supervised task environment with rich supervisory signals.
[0070] S2-1: Construction of text features and visual features. The preprocessed text and video are used as input, and the text encoder and image encoder of the CLIP model are used to construct the corresponding text and visual features. For text feature construction, we use the method based on text prompt templates commonly used in the field of natural language processing. Assume that for a target dataset containing C categories, a designed prompt template (for example: "a video of [CLS]") is used to construct text for C categories. For example, for Explosion, it is constructed as "a video of explosion", and then sent to the text encoder to generate the text embedding corresponding to the Explosion category. c represents the corresponding category. For visual feature construction, we extract the corresponding visual feature f from each video frame through an image encoder. img .
[0071] S2-2: Text-visual feature similarity calculation. In order to calculate the probability that each frame of the video belongs to each category, it is necessary to calculate the similarity between the visual features of the video frame and the text features of each category. This can be achieved by calculating the cosine similarity:
[0072]
[0073] Among them, τ is a temperature coefficient and sim(·) represents cosine similarity. Based on this similarity value, a probability distribution can be assigned to each frame, indicating the likelihood that it belongs to each category.
[0074] S2-3: Segment-level pseudo-label generation and integration. For each video v in the dataset i All include t i There are non-overlapping video clips, and each video is equipped with a video-level label, and its annotated category is c. Take a clip in the video as an example, and calculate the probability that each frame in the clip belongs to category c through S2-2. Next, take the highest abnormal clip score among these frames as the probability that the clip belongs to category c. Thus, the above process is repeated for each clip in the video in sequence, and finally the video v is obtained. i The corresponding segment-level pseudo-label P′.
[0075] The quality of pseudo-labels directly affects the training effect of the model. In order to further improve the accuracy and stability of pseudo-labels, we propose a simple and efficient pseudo-label integration strategy, which is to integrate pseudo-labels generated by multiple different visual encoders. The core idea of this strategy is to improve the accuracy and stability of pseudo-labels by combining pseudo-labels generated by multiple visual encoders. Specifically, N different visual encoders are used, and these encoders use a unified simple prompt template ("a video of a[CLS]") to independently generate their own pseudo-labels, that is, {P′1,P′2,…,P′ N In the integration process, an averaging strategy is adopted to average the probabilities output by each visual encoder, thereby reducing the deviation and noise that may be brought by a single model. This process can be mathematically expressed as follows:
[0076]
[0077] Where P∈R T , N represents the number of visual encoders, P′ i Represents the pseudo label generated by the i-th visual encoder. After obtaining the integrated segment-level pseudo label P, it is binarized by a threshold (such as 0.5) and used directly to supervise the subsequent training process.
[0078] S3: Multi-scale temporal feature modeling. First, use S1 preprocessing to obtain each video v i Corresponding fragment features As input, a multi-scale temporal feature network is constructed to perform feature x i Carry out multi-scale modeling from coarse-grained to fine-grained. The steps include:
[0079] S3-1: Feature mapping layer. First, x i The length of the fragment t i is fixed to T. Specifically, if the video contains less than T segments, it is padded with zeros to a length of T. Otherwise, it is downsampled to a predefined length of T. A feature mapping layer f is then introduced proj , used to transform the input feature x i Mapped to feature dimensions suitable for processing by the multi-scale Transformer encoder. It consists of a shallow convolutional neural network and uses ReLU as the activation function to map high-dimensional input features to a low-dimensional embedding space. Its mathematical expression is as follows:
[0080] Z0=f proj (x i )
[0081] Among them, Z0∈R T×FIt represents the video features after dimensionality reduction mapping, and F=256 is the feature dimension after mapping.
[0082] S3-2: Multi-scale transformer encoder. It is composed of multiple stacked transformer encoders, each of which has embedded layer normalization (LN), local multi-head self-attention module (LMSA) and multi-layer perceptron (MLP). For the input video feature sequence Z0, deep information extraction and interaction processing are first performed through LMSA, and then residual connection and layer normalization operations are used to ensure effective integration and stable propagation of information. Next, the GELU activation function is used to perform nonlinear transformation on the output. In order to capture the features of fragments at different time scales, feature representations at different time scales are obtained by downsampling ↓(·).
[0083]
[0084] in, T l / T l-1 is the downsampling rate. l , is a learnable scaling factor. Considering computational efficiency, ↓(·) downsampling is implemented using deep convolution, and the downsampling rate between each scale is 2, that is, T l / T l-1 =2. Finally, we get a multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Z L}, where L represents the number of layers of the pyramid.
[0085] S3-3: Lightweight convolution decoder. After obtaining the multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Z L}, a lightweight convolution decoder consisting of one-dimensional convolution is used to decode to obtain the final abnormal fragment scores of each granularity, and a classifier f is appended to it. cls To predict the probability of abnormal instances occurring at each moment at each granularity, the mathematical expression is as follows:
[0086] S l =σ(f cls (Z l ; φ cls )),l=1…L
[0087] Among them, f cls It is implemented by 3 1D convolutions with a step size of 3, layer normalization, and ReLU activation function; φ cls represents the learnable parameter; σ is the sigmoid activation function. represents the abnormal fragment score of the lth layer. Thus, the abnormal fragment scores of each granularity {S1,S2,…,S L}.
[0088] S3-4: Adaptive weight fusion mechanism. The scores of abnormal segments of each granularity {S1, S2, …, S L} as input, in the initial stage of training, first initialize a learnable weight matrix β = {β1,β2,…,β L}, where β L represents the fusion weight of the Lth layer, and these weight coefficients are uniformly initialized to the unit value 1, which means that the anomaly scores of each granularity are equally important in the initial stage of training, and then the training iterative optimization model is trained until it reaches a convergence state. Its mathematical expression is as follows:
[0089]
[0090] Among them, S∈R T represents the final abnormal segment score, T represents the number of video segments, R T It is a method of expressing data in a form, indicating that the fragment anomaly score S is a one-dimensional data with a length of T and has no actual meaning.
[0091] S4: Classifier optimization. Video abnormal event detection is considered as a multi-instance learning (MIL) task under weak supervision. This means that during training, the classifier can only rely on limited label information (video-level labels) to learn to identify abnormal events. In order to better distinguish abnormal and non-abnormal action patterns, we use pseudo-label classification loss function, representative segment dynamic ranking loss function and triple loss function to optimize the classifier.
[0092] S4-1: Pseudo-label classification loss function L pse We use the Focal Loss loss function to solve the data imbalance problem. Its original design intention is to solve the problem of insufficient model recognition of minority class samples caused by class imbalance in classification problems. In the task of video abnormal event detection, the introduction of Focal Loss can make the classifier pay more attention to those abnormal behavior samples that are difficult to identify, thereby optimizing the learning of minority class samples.
[0093] Specifically, the anomaly score S of each video clip is obtained in S3-4, and the segment-level pseudo-label P is obtained in S2 video language cue learning pseudo-label generation. The predicted abnormal segment score is directly optimized by Focal Loss, which is mathematically expressed as follows:
[0094] L pse=Focal(S,P)
[0095] in, T represents the number of video clips.
[0096] S4-2: represents the segment dynamic loss function L cls . Assume that for an input video v i , which contains t i clips, the corresponding video-level label is y i . Select the top-k examples (i.e., representative fragments) with the highest predicted anomaly scores, where And take its mean as the video-level prediction score s i The binary cross-entropy (BCE) loss function is used to optimize the classifier, and its mathematical expression is as follows:
[0097] L cls =BCE(s i ,y i )
[0098] Among them, y i ∈{0,1}, α is a hyperparameter that controls the number of fragment selections.
[0099] S4-3: Triplet loss function L trip For each video v i First, we determine the top-k examples (the k value is consistent with S4-2) with the highest abnormal fragment scores based on the predicted abnormal fragment score S. The feature expressions of these examples will be used as anchor points f a , and then calculate the feature centers of these abnormal examples and use them as positive samples f p , and then select the feature center of the normal example as the negative sample f n . Its mathematical expression is as follows:
[0100] L trip =Triplet(f a ,f p ,f n )
[0101] Among them, f a is the characteristic expression of the selected example. Similarly, f p and f n The feature centers of positive and negative samples are selected. By minimizing the distance between the anchor point and the positive sample and maximizing the distance between the anchor point and the negative sample, better inter-class separation and intra-class compactness can be achieved in the feature space. Ultimately, a more refined feature expression is obtained and a more discriminative feature distribution is obtained.
[0102] S4-4: Overall loss function L total During the training process, the overall loss function is constructed by combining the pseudo-label classification loss function, the representative fragment dynamic ranking loss function, and the triple loss function. In order to balance the three loss functions, this section regards the three loss functions as independent subtasks. We use a multi-task learning method to enable the model to adaptively determine the weight relationship between each loss function, thereby performing joint optimization and learning. Its mathematical expression is as follows:
[0103]
[0104] Among them, λ i ,i∈{1,2,3} is the learnable loss function weight.
[0105] S5: For a new video, use the method of S1 to obtain x′ i , x′ i The trained multi-scale temporal feature network is input, and then after the trained lightweight convolutional decoder and classifier, the predicted category is output.
[0106] experiment:
[0107] Table 1. Experimental results on XD-VIOLENCE and UCF-CRIME datasets
[0108]
[0109] Table 2. Effectiveness of the proposed module. MSIL: coarse-grained and fine-grained fusion method; PL: pseudo label; √ means “included”
[0110]
[0111] Table 3. Effectiveness of different loss functions. √ means “included”
[0112]
[0113] Table 4. Ablation study of pseudo-label integration strategy. All models use the same prompt template (i.e., “a video of [CLS]”)
[0114]
[0115]
[0116] The present invention is evaluated on two anomaly detection benchmark datasets, XD-Violence and UCF-Crime. XD-Violence is a large collection of 4754 unedited videos with a total duration of 217 hours, which are sourced from various media such as movies, sports broadcasts, surveillance footage, and CCTV. The dataset includes a training set of 3954 videos with video-level labels and a test set of 800 videos with frame-level labels. UCF-Crime is a large-scale anomaly detection dataset containing 1900 videos, including 13 types of abnormal events captured by surveillance cameras in indoor and outdoor daytime and nighttime scenes. The training set with video-level labels contains 1610 videos, while the test set with frame-level labels has 290 videos.
[0117] The present invention uses average precision (AP) as an indicator to evaluate the detection performance of XD-Violence. At the same time, we also use the Area Under Curve (AUC) corresponding to the Receiver Operating Characteristic (ROC) curve to evaluate the performance of UCF-Crime. Higher AP and AUC indicate that the algorithm performs better and is more stable. The sampling segment lengths T of UCF-Crime and XD-Violence are set to 128 and 256, respectively, and α=10. The AdamW optimizer is used for iterative optimization. The initial learning rate of UCF-Crime is 1e-5, the initial learning rate of XD-Violence is 5e-5, and the weight decay is 1e-2. These two datasets are trained on the RTX 3090 GPU with a batch size of 30.
[0118] Table 1 shows the experimental results of the present invention and other methods in the same field on the XD-VIOLENCE and UCF-CRIME datasets. The boldface indicates the highest detection accuracy. Obviously, the detection accuracy of the present invention is better than that of other methods, which proves the effectiveness of the present invention.
[0119] Table 2 shows the performance improvement of each module and their combination of the present invention, and conducts comparative experiments with a single-scale feature map (L=1), i.e., the Baseline model in Table 2. After adding MSIL, the AP of XD-Violence is improved by 2.7%, and the AUC of UCF-Crime is increased by 0.58%, which highlights its effectiveness in capturing short-term and long-term abnormal instances. In addition, after adding pseudo-labels generated by visual language cues, the AP is significantly improved by 4.27%, and the AUC is also significantly increased by 1.31%, indicating its effectiveness in guiding the model to focus on potential abnormal fragments rather than simple fragments. When both MSIL and pseudo-labels are added, the performance achieves the best results on both datasets, which proves the complementarity between the two branches.
[0120] Table 3 shows the results of the effects of different loss function combinations on the complete model. The baseline model only uses L cls As supervision. By comparing the first two rows, we can find that L trip It brings a satisfactory performance improvement, highlighting its effectiveness in distinguishing normal and abnormal distances. The results of the first three cases show that L pse It is most effective in guiding the network to achieve better performance, thus verifying the effectiveness of generating pseudo labels from visual language cues. In general, the performance is optimal when all three loss functions are used, indicating that they are all effective.
[0121] Table 4 shows the ablation study results of the pseudo-label ensemble strategy. We use CLIP models with different visual architectures for pseudo-label generation and quantitatively evaluate our pseudo-label ensemble strategy. For all models, we use a simple prompt template (i.e., “a video of a[CLS]”), and the ensemble strategy achieves the best results, demonstrating its importance in generating high-quality pseudo-labels.
Claims
1. A video abnormal event detection method based on prompt learning and multi-scale temporal fusion, characterized by: The steps include: S1: Get a dataset consisting of multiple videos and preprocess each video. i The video clip features obtained after preprocessing are recorded as where t i Indicates that v i The number of video segments into which the video is divided into multiple consecutive frames that do not overlap each other, and d represents the feature dimension; S2: Generate pseudo labels for each video in the dataset; S3: Construct a multi-scale temporal feature network for x i Perform multi-scale modeling from coarse-grained to fine-grained to obtain a multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Z L }, Z = {Z1, Z2, ..., Z L The final features of each scale are obtained through a lightweight convolution decoder, and the final features are passed through a classifier to obtain the abnormal fragment scores of each granularity {S1, S2, ..., S L }, and then use the adaptive weight fusion mechanism to get v i The final abnormal fragment score S, the lightweight convolution decoder and the subsequent classifier constitute the classification network f cls ; S4: Design the overall loss function L total , and use the loss function to update the multi-scale temporal feature network and classification network f in S3 cls Parameters of until the loss no longer changes, and the trained multi-scale temporal feature network, lightweight convolutional decoder and classifier are obtained; S5: For a new video, use the method of S1 to obtain x′i, input x′i into the trained multi-scale temporal feature network, and then output the predicted category after passing through the trained lightweight convolutional decoder and classifier.
2. The method for detecting abnormal events in video based on prompt learning and multi-scale temporal fusion as claimed in claim 1, characterized in that: In S1, preprocessing the video includes: For a video v in the dataset i , first divide the video into t i A video clip containing 16 consecutive frames that do not overlap is sent to the 13D network for visual feature extraction. The video clip features obtained after global average pooling of the Mixed_5c layer in the 13D network are recorded as Where d = 1024, the above operation is performed on each video in the dataset, and finally the visual features of all videos in the dataset are obtained.
3. The method for detecting abnormal events in video based on prompt learning and multi-scale temporal fusion as claimed in claim 1, characterized in that: In S2, generating a pseudo label for each video in the data set includes: S2-1: Use the text encoder and image encoder of the CLIP model to construct corresponding text features and visual features for each video in the dataset; for text feature construction, use the text prompt template-based method commonly used in the field of natural language processing to obtain text embedding c represents the corresponding category; for visual features, the corresponding visual features f are extracted from each video through an image encoder img ; S2-2: Text-visual feature similarity calculation is achieved by calculating the cosine similarity p: Among them, τ is a temperature coefficient, sim(·) represents the cosine similarity, based on which a probability distribution is assigned to each frame, indicating the likelihood that it belongs to each category; S2-3: Segment-level pseudo-label generation and integration: For each video v in the dataset i All include t i non-overlapping video clips, and each video is equipped with a video-level label, and its annotated category is c; the probability of each frame in the clip belonging to category c is calculated by S2-2, and then the highest abnormal clip score among these frames is taken as the probability that the clip belongs to category c. Thus, the above process is repeated for each clip in the video in sequence, and finally the video v is obtained. i The corresponding segment-level pseudo-label P′; N different visual encoders are used, and N different visual encoders use a unified simple prompt template to independently generate their own pseudo labels, denoted as {P′1, P′2, ..., P′ N }; During the integration process, an averaging strategy is adopted to average the probabilities output by each visual encoder. The mathematical expression is as follows: Where P∈R T , P′ i Represents the pseudo-label generated by the i-th visual encoder. After obtaining the integrated segment-level pseudo-label P, it is binarized by thresholding.
4. The method for detecting abnormal events in video based on prompt learning and multi-scale temporal fusion as claimed in claim 1, characterized in that: In S3, constructing a multi-scale temporal feature network includes: Multi-scale temporal feature modeling first uses S1 preprocessing to obtain each video v i Corresponding fragment features As input, for the feature x i Conduct multi-scale modeling from coarse-grained to fine-grained; S3-1: Feature mapping layer, x i The length of the fragment t i Fixed to T, if the video v i If the number of segments is less than T, padded with zeros to length T; otherwise, i Downsampled to a predefined length T; then a feature mapping layer f is introduced proj , used to transform the input feature x i Mapped to a feature dimension suitable for processing by a multi-scale Transformer encoder; f proj A shallow convolutional neural network uses ReLU as the activation function to map high-dimensional input features to a low-dimensional embedding space. The mathematical expression is as follows: Z0=f proj (x i ) Among them, Z0∈R TxF Represents x i Video features after dimensionality reduction mapping, F is the feature dimension after mapping; S3-2: The multi-scale Transformer encoder is composed of multiple stacked Transformer encoders, each of which has embedded layer normalization LN, local multi-head self-attention module LMSA and multi-layer perceptron MLP. For the input Z0, LMSA is first used to extract deep information and interact with each other, and then residual connection and layer normalization operations are used to ensure effective integration and stable propagation of information. Next, the GELU activation function is used to perform nonlinear transformation on the output. Downsampling ↓(·) is used to obtain feature representations of different time scales. in, T l / T l-1 is the downsampling rate, Z l-1 Represents the features of the previous feature pyramid. Represents the features obtained by the first Transformer encoder, Represents the features obtained by the second Transformer encoder, Z l Represents the final feature of this layer of pyramid, is a learnable scaling factor. Downsampling is achieved by deep convolution, and finally a multi-scale temporal feature pyramid Z = {Z1, Z2, ..., Z L }, where L represents the number of layers of the pyramid; S3-3: Use a lightweight convolution decoder composed of one-dimensional convolution to decode and obtain the final features of each scale, and then append a classifier to predict the abnormal fragment fraction of abnormal instances at each moment at each granularity. The mathematical expression is as follows: S l =σ(f cls (Z l ;f cls )),l=1...L Among them, f cls It is implemented by 3 1D convolutions with a stride of 3, layer normalization and ReLU activation function; φ cls represents the learnable parameter; σ is the sigmoid activation function; the obtained represents the abnormal fragment score of the lth layer; thus, the abnormal fragment scores of each granularity {S1, S2, ..., S L }; S3-4: Adaptive weight fusion mechanism, {S1, S2, ..., S L } as input, define the learnable weight matrix β = {β1, β2, ..., β L }, where β L represents the fusion weight of the Lth layer, and the mathematical expression of the final abnormal fragment score S is as follows: Among them, S∈R T represents the final abnormal segment score, T represents the number of video segments, R T It is a method of expressing data in a certain form and has no actual meaning.
5. The method for detecting abnormal video events based on prompt learning and multi-scale temporal fusion according to claim 1, characterized in that: In S4, the overall loss function L total for: The overall loss function L total Including pseudo-label classification loss function, representative fragment dynamic ranking loss function and triple loss function; Pseudo-label classification loss function L pse : FocalLoss is used to optimize the predicted abnormal fragment score, and its mathematical expression is as follows: L pse =Focal(S,P) Represents the segment dynamic loss function L cls : The binary cross entropy loss function is used to optimize the classifier, and its mathematical expression is as follows: L cls =BCE(s i ,y i ) Among them, y i Indicates v i Medium i The label corresponding to the fragment, y i ∈{0, 1}, α is a hyperparameter used to control the number of fragments selected, and select the top-k fragments with the highest predicted anomaly scores. The average of the top-k predicted anomaly scores is taken as the video-level prediction score s i ; S4-3: Triplet loss function L trip , for each video v i First, the top-k segments with the highest abnormal segment scores are determined based on the predicted final abnormal segment score S; the feature expression of the top-k segments with the highest abnormal segment scores is used as the anchor point f a , and then calculate the feature centers of these abnormal fragments and use them as positive samples f p , and then select the feature center of the normal segment as the negative sample f n , L trip The mathematical expression is as follows: L trip =Triplet(f a ,f p ,f n ) The overall loss function L total : Among them, λ i , i∈{1, 2, 3} is the learnable loss function weight.
Citation Information
Patent Citations
Weak supervision video abnormal behavior detection method and system
CN116580345A
Video scene graph generation method based on multi-scale space-time attention network
CN117115706A
Text detection method and device, text detection model optimization method and device and data annotation method and device
CN117275005A
System and method of bridging the gap between object and image-level representations for open-vocabulary detection
US20240203085A1
Cited By
Audio and video event positioning method based on fine tuning pre-training large model
CN119475244A
An audio and video event localization method based on fine-tuning pre-trained large model
CN119475244B
Cloud edge multi-mode abnormal event detection system based on dynamic decision-making mechanism
CN119494068A
Video highlight detection method based on weak supervision multi-mode large model
CN119785257A
Semi-supervised anomaly detection method based on memory enhancement and pseudo marking
CN119831972A
