Weakly supervised video anomaly detection method based on multiple textual cues

Abstract

This invention discloses a weakly supervised video anomaly detection method based on multi-cue learning, comprising the following steps: 1) extracting raw features from video data using a video feature extractor; 2) inputting the raw features into a global-local temporal dependency network to obtain temporal enhancement features; 3) extracting two different types of prompt information from label text using a knowledge graph and a to-be-learned prompt; 4) cross-modally fusing the two types of prompt information with the temporal enhancement features; 5) passing the temporal enhancement features through a segment classifier to obtain a segment anomaly score, and optimizing the network by designing a loss function to achieve the video anomaly detection task. This invention utilizes only video-level labels and constructs a dual-branch self-attention structure to extract the global and local temporal dependencies of segments. Furthermore, it extracts textual prompt information through two different methods and cross-modally fuses them with segment features, supplementing the video anomaly information and effectively improving the accuracy of video anomaly detection.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular to a video anomaly detection method based on weak supervision of multiple text prompts. Background Art

[0002] The goal of video anomaly detection is to identify unusual events or behaviors in videos that do not conform to normal or expected patterns. With the decreasing cost of deploying surveillance cameras, the application of video surveillance has expanded to a wide range of scenarios, demonstrating significant advantages. Consequently, intelligent content review of surveillance videos has become a growing concern in academia and industry. In recent years, with the rapid development of science and technology, VAD has made significant progress thanks to the efforts of numerous researchers.

[0003] Unsupervised video anomaly detection has attracted widespread attention due to its ability to detect anomalies without the need for additional annotations. However, this approach only trains on normal videos, resulting in a limited understanding of abnormal videos. Consequently, USVAD often misidentifies normal events that were not seen during training as anomalies, resulting in a high false positive rate. To address the problem of misidentification of video anomalies in unsupervised settings, a more practical scenario is considered, where only video-level labels are available. That is, during training, only the video is known to be normal or abnormal, but the specific frame is unknown. Weakly supervised video anomaly detection offers several advantages over unsupervised methods: 1) WSVAD combines both normal and abnormal videos during training, enabling accurate detection and identification of abnormal events. 2) WSVAD achieves superior performance requiring only video-level annotations, achieving a better balance between detection performance and the cost of manual annotation. 3) Introducing a small amount of annotation information helps the model understand various abnormal behaviors, enabling better generalization to new, unseen anomalies.

[0004] Most research methods in weakly supervised video anomaly detection follow a systematic and regular detection process. Initially, they leverage pre-trained visual models, such as VideoSwin, C3D, and I3D, to extract frame-level features from the video. Attention mechanisms or other spatiotemporal frameworks are then used to capture the temporal relationships between these features. These features are then fed into a binary classifier based on multi-instance learning and a Top-K mechanism to distinguish between normal and abnormal events based on the anomaly score for each frame. This approach is simple and effective, but it fails to fully leverage cross-modal information to enhance the model's ability to identify anomalies.

[0005] Recently, visual language models (VLMs) have demonstrated remarkable capabilities in understanding video content and rich semantic knowledge. VLMs, trained on large-scale image-text pairs using contrastive learning, effectively align visual and semantic spaces. For example, VLMs (CLIP, ALIGN, CoCa) have demonstrated impressive transferability across a wide range of vision tasks. Consequently, building task-specific models on CLIP is becoming an emerging research topic and has been applied to a wide range of vision tasks, achieving unprecedented performance.

[0006] However, extending CLIP's image-text contrastive learning to the more complex task of video anomaly detection in a weakly supervised setting presents a new challenge. Most previous methods have solely used CLIP's image encoder to extract visual features from videos, ignoring the text encoder and thus overlooking CLIP's rich semantic relationships between vision and language. Therefore, effectively leveraging CLIP in a weakly supervised setting and ensuring its full performance in WSVAD presents several challenges:

[0007] 1) How to effectively capture the temporal contextual dependencies between video clips and pay special attention to abnormal frames.

[0008] 2) How to extract useful semantic information from extremely limited text information.

[0009] 3) How to fuse and align visual information with label information to improve model performance. Summary of the Invention

[0010] In order to address the shortcomings of the above-mentioned prior art, the present invention proposes a weakly supervised video anomaly detection method based on multiple text prompts, in order to effectively supplement the normal or abnormal information in the video, thereby improving the accuracy of anomaly detection.

[0011] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0012] The invention provides a weakly supervised video anomaly detection method based on multiple text prompts, which comprises the following steps:

[0013] S1. Get the video-level label y and category text label A video V of , when y=1, it means that there is an abnormal event in the video V, and when y=0, it means that there is no abnormal event in the video V;

[0014] Divide the video V into T temporally non-overlapping segments ,in, represents the t-th video clip, T is the number of video clips;

[0015] Obtained using the CLIP model's image encoder CLIP features to obtain the initial feature set of the video clip ,in, Indicates the tth video clip CLIP features, D represents the dimension of the clip features;

[0016] S2. Build an anomaly detection network, including: global-local temporal dependency network, multiple text prompt network, and cross-modal fusion network;

[0017] The global-local temporal dependency network consists of a self-attention branch and a segment attention branch;

[0018] The multi-text prompt network consists of a knowledge graph extraction module, a learnable prompt module and a text encoder of the CLIP model;

[0019] The cross-modal fusion network consists of a cosine similarity calculation module, an average pooling layer and a fragment aggregation module;

[0020] S21, will The input is fed into the anomaly detection network and processed by the self-attention branch in the global-local temporal dependency network to obtain the global temporal dependency feature vector between each video clip. ;

[0021] at the same time, After the adaptive local mask and projection processing of the segment attention branch, the adaptive local time-dependent feature vector is obtained ;

[0022] Will and After adaptive fusion, the time-enhanced segment feature set is obtained ,in, Represents the enhanced segment features of the t-th frame;

[0023] Fragment features enhanced by time The anomaly score and video-level label y construct the classification loss ;

[0024] S22. Label the category text Input into the multiple text prompt network and processed by the knowledge graph extraction module to obtain Similar concepts serve as supplementary prompts;

[0025] The text prompt vector to be learned is prefixed with After being combined, the prompts and supplementary prompts are input into the text encoder of the CLIP model for processing to obtain the prompt feature vectors to be learned. and concept tips vector ;

[0026] S23, will and Input into the cross-modal fusion network and calculate and The cosine similarity between them is used to obtain the visual-text similarity matrix , then After performing Softmax operation and average pooling in sequence, the saliency score matrix of the segment is obtained ;

[0027] Depend on and After aggregation, the video feature vector enhanced with concept hints is obtained ;

[0028] Depend on and Constructing symmetric cross entropy loss ;

[0029] S3. Use formula (1) to construct the overall loss function :

[0030] (1)

[0031] In formula (1), and is a hyperparameter; is the cross-modal contrast loss;

[0032] S4. Use the gradient descent method to train the anomaly detection network and calculate the total loss function To update the network parameters until the total loss function Until convergence or reaching the maximum number of training times, the optimal anomaly detection model after training is obtained;

[0033] S5. Input the test video into the trained optimal anomaly detection model for processing, and output an anomaly score for each segment. When the anomaly score of a segment is higher than a preset threshold, it indicates that an abnormal event exists in the corresponding segment.

[0034] The weakly supervised video anomaly detection method based on multiple text prompts of the present invention is also characterized in that the self-attention branch in step S21 uses three different linear layers to Projecting them into the same latent space, we get the query feature set Q, key feature set K, and value feature set U. Then, we perform a dot product operation on Q and K, and then perform a softmax operation to get the attention map. ;Will After dynamic position encoding, it is multiplied by U to obtain the global time-dependent feature vector ;

[0035] The segment attention branch uses adaptive max pooling and adaptive average pooling to After processing, the two pooled feature sets are fed into a shared multi-layer perceptron (MLP) for learning, and two shallow feature sets are obtained accordingly.

[0036] After the two shallow feature sets are mapped by the Add operation and the Sigmoid activation function, the fragment attention weight matrix is ​​obtained. ;

[0037] Will After multiplying with the set parameters, the adaptive mask matrix of the fragment is obtained and projected into the attention map On top, get the local attention map ;

[0038] Will After multiplying by U, we get the adaptive local time-dependent eigenvector .

[0039] The knowledge graph extraction module in step S22 retrieves the After selecting the related concepts and the concepts with the largest correlation scores, they are input into the text encoder of the CLIP model for feature extraction to obtain the concept feature vector ;

[0040] Will After the conversion of the class tag in the Tokenizer tag converter of the CLIP model, it is embedded into m prompt vectors to be learned , thus forming a complete sentence tag , finally, Input into CLIP's text encoder for feature extraction to obtain the prompt feature vector to be learned ,in, represents the mth hint vector to be learned.

[0041] In step S23, As anchor points, calculate and The two-way similarity score between and , used to construct the symmetric cross entropy loss .

[0042] The cross-modal contrast loss in S3 It is constructed as follows:

[0043] S3.1. Calculating the abnormal video feature vector after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ;

[0044] S3.2. Calculating the abnormal video feature vector after concept hint enhancement Normal video feature vector enhanced with concept hints The cosine similarity between ;

[0045] S3.3. Calculating the normal video feature vector after concept hint enhancement and the normal prompt feature vector to be learned The cosine similarity between ;

[0046] S3.4. Calculate the feature vector of normal video after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ;

[0047] S3.5, by 、 、 and Constructing the cross-modal contrast loss .

[0048] In S3.5, the cross-modal contrast loss is constructed using formula (2) :

[0049] (2)

[0050] In formula (2), B represents the total number of videos, is the cosine similarity, and are the bth video V b The corresponding concept prompt enhanced abnormal video feature vector and the abnormal prompt feature vector to be learned, and The bth video V b The corresponding concept cue enhanced normal video feature vector and the normal cue feature vector to be learned.

[0051] The self-attention branch is used to use formula (3) Perform dynamic position encoding to obtain the encoded attention map :

[0052] (3)

[0053] In formula (6), and is the absolute position between any two video clips in video V, and are the weights and bias terms to be learned.

[0054] The S21 is to use formula (4) to and Perform adaptive fusion:

[0055] (4)

[0056] In formula (5), represents the adaptive fusion fusion feature vector, and is obtained by formula (6), is the normalization operation, is a linear layer, It is a layer normalization layer, + represents a residual connection;

[0057] (5)

[0058] In formula (5), is the factor to be learned.

[0059] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the weakly supervised video anomaly detection method, and the processor is configured to execute the program stored in the memory.

[0060] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the weakly supervised video anomaly detection method when the computer program is executed by a processor.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. This invention addresses the complexity of video data and the impact of multiple samples, aiming to enrich and utilize video information as fully as possible. It uses multiple textual cues to introduce category-related concepts to supplement the abnormal / normal information in the video, enabling the model to uncover deeper hidden information in the video.

[0063] 2. This invention takes into account the temporal differences of video data and the different abnormality levels of segments. Therefore, it accurately models each segment according to its importance, and then obtains the local temporal relationship of the video. This enables the model to capture the fine-grained global-local temporal dependency of the video from different angles, and realizes segment-level temporal relationship modeling.

[0064] 3. This invention takes into account the differences between visual and textual multimodal features and generates score saliency through a video concept enhancement mechanism to assign similar concepts to each clip, thereby generating a compact video representation and performing video-level cross-modal bidirectional alignment with the cues to be learned, enriching the model's semantic differentiation capabilities to enhance the accuracy and efficiency of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flowchart of the weakly supervised video anomaly detection method based on multiple text prompts of the present invention. DETAILED DESCRIPTION

[0066] In this embodiment, a weakly supervised video anomaly detection method based on multiple text prompts first uses the image encoder of the CLIP model to extract segment features from video data, then uses the self-attention mechanism to focus on the global temporal dependencies between segments, and uses the segment attention mechanism to quantify the importance of each segment to generate a local adaptive mask matrix, thereby focusing on the local temporal dependencies between segments, and realizing segment-level temporal relationship modeling. Then, two different methods are used to extract text prompt information, and the concept prompt features are fused with the video features to mine the deep hidden information of the video. The learnable prompts and the fused video features are cross-modally aligned to enrich the semantic distinction ability of the model. Finally, contrastive learning is introduced to train the model to enhance the model's ability to distinguish anomalies. Specifically, if Figure 1 As shown, the method includes the following steps:

[0067] S1. Get the video-level label y and the category text label A video V of , when y=1, it means that there is an abnormal event in the video V, and when y=0, it means that there is no abnormal event in the video V; divide the video V into T segments that do not overlap in time ,in, Represents the t-th video clip, where T is the number of video clips; the image encoder of the CLIP model is used to obtain CLIP features to obtain the initial feature set of the video clip ,in, Indicates the tth video clip CLIP features, D represents the dimension of the clip features;

[0068] In this example, training and testing were performed on two standard video anomaly detection datasets: UCF-Crime and XD-Violence. UCF-Crime contains 1,900 surveillance videos covering 13 abnormal event categories, of which 1,610 videos with video-level labels are used for training, and 290 videos with frame-level labels are used for testing. XD-Violence contains 4,754 videos covering six abnormal event categories, of which 3,954 training videos have video-level labels, and 800 testing videos have frame-level labels.

[0069] S2. Build an anomaly detection network, including a global-local temporal dependency network, a multiple text prompt network, and a cross-modal fusion network;

[0070] The global-local temporal dependency network consists of a self-attention branch and a segment attention branch;

[0071] The multi-text prompt network consists of a knowledge graph extraction module, a to-be-learned prompt module, and a text encoder of the CLIP model;

[0072] The cross-modal fusion network consists of a cosine similarity calculation module, an average pooling layer, and a fragment aggregation module.

[0073] S21, will The input is fed into the anomaly detection network and processed by the self-attention branch in the global-local temporal dependency network to obtain the global temporal dependency feature vector between each video clip. ;at the same time, After the adaptive local mask and projection processing of the segment attention branch, the adaptive local time-dependent feature vector is obtained ;Will and After adaptive fusion, the time-enhanced segment feature set is obtained ,in, Represents the enhanced segment features of the t-th frame;

[0074] The self-attention branch uses three different linear layers to Projecting them into the same latent space, we get the query feature set Q, key feature set K, and value feature set U. Then, we perform a dot product operation on Q and K, and then perform a softmax operation to get the attention map. ;Will After dynamic position encoding, it is multiplied by U to obtain the global time-dependent feature vector .

[0075] The segment attention branch uses adaptive max pooling and adaptive average pooling to After processing, the two pooled feature sets are fed into a shared multi-layer perceptron (MLP) for learning, and two shallow feature sets are obtained accordingly.

[0076] After the two shallow feature sets are mapped by the Add operation and the Sigmoid activation function, the fragment attention weight matrix is ​​obtained. ;

[0077] Will After multiplying with the set parameters, the adaptive mask matrix of the fragment is obtained and projected into the attention map On top, get the local attention map ;Will After multiplying by U, we get the adaptive local time-dependent eigenvector .

[0078] The self-attention branch is to use formula (1) to Perform dynamic position encoding to obtain the encoded attention map :

[0079] (1)

[0080] In formula (1), and is the absolute position between any two video clips in video V, and are the weights and bias terms to be learned.

[0081] In this example, the self-attention branch shows good performance in capturing the global temporal dependencies between video clips. 、 and The initial eigenvector Projected into the same latent space, the attention map is generated using formula (2) Afterwards, Perform dynamic position encoding and use formula (3) to obtain the global time-dependent feature vector :

[0082] (2)

[0083] (3)

[0084] In this embodiment, a projection operation is used to retain the elements in the attention map at the position of element 1 in the mask matrix, and set the remaining positions to 0, as shown in formula (4):

[0085] M ij l = { M ij if j ∈ [ max ( 0 , i − ⌊ W i 2 ⌋ ) ,min ( i + ⌊ W i 2 ⌋ , T ) ] −∞ otherwise (4)

[0086] in:

[0087]

[0088] in, represents a multilayer perceptron, represents the local adaptive mask matrix, represents adaptive max pooling, represents adaptive average pooling, is the sigmoid function, It is a parameter used to adjust the window size. is the number of segments of the input video;

[0089] Using formula (5) and Perform adaptive fusion:

[0090] (5)

[0091] In formula (5), represents the adaptive fusion fusion feature vector, and is obtained by formula (6), is the normalization operation, is a linear layer, It is a layer normalization layer, + represents a residual connection;

[0092] (6)

[0093] In formula (6), is the factor to be learned.

[0094] Finally, the feature vector of the segment enhanced by time Anomaly score With video-level tags The cross entropy loss is calculated as shown in formula (7):

[0095] L ce =− ∑ i = 0 B [ y i log ( p i ) + ( 1 − y i ) log ( 1 − p i ) ] (7)

[0096] In formula (7), B is the number of input videos, is the cross entropy loss of the anomaly detection network.

[0097] S22. Label the category text Input into the multiple text prompt network and processed by the knowledge graph extraction module to obtain Similar concepts are used as supplementary prompts; the text prompt vector to be learned is used as a prefix with After being combined, the prompts and supplementary prompts are input into the text encoder of the CLIP model for processing to obtain the prompt feature vectors to be learned. and concept tips vector ;

[0098] The knowledge graph extraction module retrieves and After selecting the related concepts and the concepts with the largest correlation scores, they are input into the text encoder of the CLIP model for feature extraction to obtain the concept feature vector ;Will After the conversion of the class tag in the Tokenizer tag converter of the CLIP model, it is embedded into m prompt vectors to be learned , thus forming a complete sentence tag , finally, Input into CLIP's text encoder for feature extraction to obtain the prompt feature vector to be learned ,in, represents the mth hint vector to be learned.

[0099] The process of using a knowledge graph to retrieve related concepts is a combination of semantic understanding, information retrieval, and data processing. A knowledge graph is essentially a triple: <start node, retrieval relation, end node>. Therefore, we input a given category c as the start or end node of this triple, and the knowledge graph outputs the corresponding end or start node based on the retrieval relation. The CLIP model first converts text into class tokens using a tokenizer. It then selects the element corresponding to the maximum index position in another tensor, token, to form a new tensor, as shown in Equation (8):

[0100] (8)

[0101] In formula (8), It is a knowledge graph. is the number of selected concepts, It is a language tokenizer that converts words into vectors. It is a text encoder for CLIP.

[0102] S23, will and Input into the cross-modal fusion network and calculate and The cosine similarity between them is used to obtain the visual-text similarity matrix , then After performing Softmax operation and average pooling in sequence, the saliency score matrix of the segment is obtained ;Depend on and After aggregation, the video feature vector enhanced with concept hints is obtained ;

[0103] by As anchor points, calculate and The two-way similarity score between and , used to construct the symmetric cross entropy loss .

[0104] The cross-modal fusion network uses the video-concept enhancement mechanism to use the concept hint feature as a query to obtain more refined word-pair segment saliency. The video-concept enhancement mechanism is the process of fusing related concepts into video segments as supplementary information. Since the related concepts are all abnormal, this may interfere with normal events in abnormal videos. Therefore, during the aggregation process, normal segments will be suppressed and abnormal segments will be enhanced, so that the generated compact video features contain more abnormal information and eliminate the interference of noise.

[0105] Score the similarity and With video-level tags Perform symmetric bidirectional cross entropy loss The calculation of is shown in formula (9):

[0106] (9)

[0107] in:

[0108]

[0109]

[0110] in, represents the video-to-word cross entropy loss, represents the word-to-video cross entropy loss, k ∈ K ( i ) = { k | k ∈ [ 1 , B ], y k = y i } , is the temperature hyperparameter for scaling.

[0111] S25. Use formula (10) to construct the overall loss function :

[0112] (10)

[0113] In formula (10), and is a hyperparameter; is the cross-modal contrast loss and is constructed as follows:

[0114] Computing the abnormal video feature vector after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ; Calculate the abnormal video feature vector after concept hint enhancement Normal video feature vector enhanced with concept hints The cosine similarity between ; Calculate the normal video feature vector after concept hint enhancement and the normal prompt feature vector to be learned The cosine similarity between ; Calculate the normal video feature vector after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ;Depend on 、 、 and Use formula (11) to construct cross-modal contrast loss .

[0115] (11)

[0116] In formula (11), B represents the total number of videos, is the cosine similarity, and are the bth video V b The corresponding concept prompt enhanced abnormal video feature vector and the abnormal prompt feature vector to be learned, and The bth video V b The corresponding concept cue enhanced normal video feature vector and the normal cue feature vector to be learned.

[0117] and It is to identify the feature representation of a video sample (whether normal or abnormal), and on this basis, evaluate its cosine similarity with the feature representation of each opposite type of video in the batch, and finally select the maximum value of these similarities as the optimization target.

[0118] The idea of ​​contrastive loss is to compare normal and abnormal video data samples to learn representations, so that video samples of the same type are closer in the representation space, while samples of different types are farther away. Video features are used as anchors to control the boundary between normal video and category text features and abnormal video and category text features, thereby achieving good separability between normal and abnormal.

[0119] S3. Use the gradient descent method to train the anomaly detection network and calculate the total loss function To update the network parameters until the total loss function Until convergence or reaching the maximum number of training times, the optimal anomaly detection model after training is obtained;

[0120] S4. Input the test video into the trained optimal anomaly detection model for processing, and output an anomaly score for each segment. When the anomaly score of the segment is higher than a preset threshold, it indicates that an abnormal event exists in the segment.

[0121] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0122] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0123] The present invention will Set to 20 and 14 respectively, The training process was set to 0.01. During the training process, the batch size was set to 32, the number of training rounds was 50, the initial learning rate was initialized to 5.e-4, the CLIP model was ViT-B / 16, and the number of training video clips was cropped or expanded to 256. Training and testing were performed in Python 3.9 and torch 2.1.0. The experiments were conducted using an Nvidia GeForce RTX 4060 GPU:

[0124] Table 1 is a graph showing the comparative experimental results of the present invention and the existing anomaly detection method.

[0125]

[0126] Table 1 shows the performance of different methods, comparing AUC and AP on UCF-Crime and XD-Violence, including the proposed method and eight baseline methods. The eight baseline methods are:

[0127] RTFM is Robust Event Feature Magnitude Learning, from the paper:

[0128] "Weakly-supervised video anomaly detection with robust temporal feature magnitude learning; published at ICCV 2021;

[0129] CRFD stands for Causal Temporal Relationship Learning, and comes from the paper: "Learning Causal Temporal Relation and Feature Discrimination for Anomaly Detection", published in the 2021 IEEE Transactions on Image Processing;

[0130] MSL is self-training multi-sequence learning, from the paper:

[0131] "Self-training multi-sequence learning with transformer for weaklysupervised video anomaly detection", published at AAAI in 2022;

[0132] MGFN stands for Magnitude-contrastive glance and focus network, from the paper "Mgfn: Magnitude-contrastive glance-and-focus network for weakly-supervised video anomaly detection.", published at AAAI 2023.

[0133] UR-DMU is an anomaly detection network based on dual memory units with uncertainty regulation, from the paper "Dual memory units with uncertainty regulation for weakly supervised video anomaly detection", published at AAAI 2023;

[0134] CLIP-TSA is a clip-assisted temporal self-attention anomaly detection network, from the paper: "Clip-tsa: Clip-assisted temporal self-attention for weakly-supervised video anomaly detection", published in ICIP in 2023;

[0135] TPWNG is a text prompt anomaly detection network based on normality guidance, from the paper: "Text prompt with normality guidance for weakly supervised video anomaly detection", published in CVPR 2024;

[0136] VadCLIP is a visual language model adapted for weakly supervised video anomaly detection, from the paper: "Vadclip: Adapting vision-language models for weakly supervised video anomaly detection", published in AAAI in 2024;

[0137] The proposed method outperforms the baseline method in every evaluation metric and achieves significant improvements compared to the baseline, demonstrating the superiority of the proposed model.

[0138] In summary, the present invention captures fine-grained global-local temporal dependencies in videos from different angles, and uses multiple text prompts to introduce category-related concepts to supplement the abnormal / normal information in the video, thereby mining deeper hidden information in the video and providing clearer decision boundaries for videos of different categories, thereby improving the accuracy of video anomaly detection.

Claims

1. A weakly supervised video anomaly detection method based on multiple text prompts, characterized by: The following steps are involved: S1. Get the video-level label y and category text label A video V of , when y=1, it means that there is an abnormal event in the video V, and when y=0, it means that there is no abnormal event in the video V; Divide the video V into T temporally non-overlapping segments ,in, represents the t-th video clip, T is the number of video clips; Obtained using the CLIP model's image encoder CLIP features to obtain the initial feature set of the video clip ,in, Indicates the tth video clip CLIP features, D represents the dimension of the clip features; S2. Build an anomaly detection network, including: global-local temporal dependency network, multiple text prompt network, and cross-modal fusion network; The global-local temporal dependency network consists of a self-attention branch and a segment attention branch; The multi-text prompt network consists of a knowledge graph extraction module, a learnable prompt module and a text encoder of the CLIP model; The cross-modal fusion network consists of a cosine similarity calculation module, an average pooling layer and a fragment aggregation module; S21, will The input is fed into the anomaly detection network and processed by the self-attention branch in the global-local temporal dependency network to obtain the global temporal dependency feature vector between each video clip. ; at the same time, After the adaptive local mask and projection processing of the segment attention branch, the adaptive local time-dependent feature vector is obtained. ; Will and After adaptive fusion, the time-enhanced segment feature set is obtained ,in, Represents the enhanced segment features of the t-th frame; Fragment features enhanced by time The anomaly score and video-level label y construct the classification loss ; S22. Label the category text Input into the multiple text prompt network and processed by the knowledge graph extraction module to obtain Similar concepts serve as supplementary prompts; The text prompt vector to be learned is prefixed with After being combined, the prompts and supplementary prompts are input into the text encoder of the CLIP model for processing to obtain the prompt feature vectors to be learned. and concept tips vector ; S23, will and Input into the cross-modal fusion network and calculate and The cosine similarity between them is used to obtain the visual-text similarity matrix , then After performing Softmax operation and average pooling in sequence, the saliency score matrix of the segment is obtained ; Depend on and After aggregation, the video feature vector enhanced with concept hints is obtained ; Depend on and Constructing symmetric cross entropy loss ; S3. Use formula (1) to construct the overall loss function : (1) In formula (1), and is a hyperparameter; is the cross-modal contrast loss; S4. Use the gradient descent method to train the anomaly detection network and calculate the overall loss function To update the network parameters until the total loss function Until convergence or reaching the maximum number of training times, the optimal anomaly detection model after training is obtained; S5. Input the test video into the trained optimal anomaly detection model for processing, and output the anomaly score of each segment; when the anomaly score of a segment is higher than the preset threshold, it indicates that an abnormal event exists in the corresponding segment.

2. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 1 is characterized in that: The self-attention branch in step S21 uses three different linear layers to Projecting them into the same latent space, we get the query feature set Q, key feature set K, and value feature set U. Then, we perform a dot product operation on Q and K, and then perform a softmax operation to get the attention map. ;Will After dynamic position encoding, it is multiplied by U to obtain the global time-dependent feature vector ; The segment attention branch uses adaptive max pooling and adaptive average pooling to After processing, the two pooled feature sets are fed into a shared multi-layer perceptron (MLP) for learning, and two shallow feature sets are obtained accordingly. After the two shallow feature sets are mapped by the Add operation and the Sigmoid activation function, the fragment attention weight matrix is ​​obtained. ; Will After multiplying with the set parameters, the adaptive mask matrix of the fragment is obtained and projected into the attention map On the top, get the local attention map ; Will After multiplying by U, we get the adaptive local time-dependent eigenvector .

3. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 1 is characterized in that: The knowledge graph extraction module in step S22 retrieves the After selecting the related concepts and the concepts with the largest correlation scores, they are input into the text encoder of the CLIP model for feature extraction to obtain the concept feature vector ; Will After the conversion of the class tag in the Tokenizer tag converter of the CLIP model, it is embedded into m prompt vectors to be learned , thus forming a complete sentence tag , finally, Input into CLIP's text encoder for feature extraction to obtain the prompt feature vector to be learned ,in, represents the mth hint vector to be learned.

4. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 1 is characterized in that: In step S23, As anchor points, calculate and The two-way similarity score between and , used to construct the symmetric cross entropy loss .

5. The video anomaly detection method based on weak supervision of multiple text prompts according to claim 1 is characterized in that: The cross-modal contrast loss in S3 It is constructed as follows: S3.

1. Calculating the abnormal video feature vector after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ; S3.

2. Calculating the abnormal video feature vector after concept hint enhancement Normal video feature vector enhanced with concept hints The cosine similarity between ; S3.

3. Calculating the normal video feature vector after concept hint enhancement and the normal prompt feature vector to be learned The cosine similarity between ; S3.

4. Calculate the feature vector of normal video after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ; S3.5, by 、 、 and Constructing the cross-modal contrast loss .

6. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 5 is characterized in that: In S3.5, the cross-modal contrast loss is constructed using formula (2) : (2) In formula (2), B represents the total number of videos, is the cosine similarity, and are respectively the bth video V b The corresponding concept prompt enhanced abnormal video feature vector and the abnormal prompt feature vector to be learned, and The bth video V b The corresponding concept cue enhanced normal video feature vector and the normal cue feature vector to be learned.

7. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 2 is characterized in that: The use of self-attention branch is to use formula (3) to Perform dynamic position encoding to obtain the encoded attention map : (3) In formula (6), and is the absolute position between any two video clips in video V, and are the weights and bias terms to be learned.

8. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 1 is characterized in that: The S21 is to use formula (4) to and Perform adaptive fusion: (4) In formula (5), represents the adaptive fusion feature vector, and is obtained by formula (6), is the normalization operation, is a linear layer, It is a layer normalization layer, + represents a residual connection; (5) In formula (5), is the factor to be learned.

9. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the weakly supervised video anomaly detection method according to any one of claims 1 to 7, and the processor is configured to execute the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the weakly supervised video anomaly detection method according to any one of claims 1 to 7 are executed.

Images