Weak supervision video anomaly detection method based on multiple text prompts

Abstract

The invention discloses a weak supervision video anomaly detection method based on multiple prompt learning. The method comprises the following steps: 1) extracting video data through a video feature extractor to obtain original features; 2) inputting the original features into a global-local time dependence network to obtain time enhancement features; 3) obtaining two kinds of different prompt information from the label text through the knowledge graph and the to-be-learned prompt; 4) performing cross-modal fusion on the two kinds of prompt information and the time enhancement feature; and 5) obtaining a fragment exception score of the time enhancement feature through a fragment classifier, and optimizing the network by designing a loss function to realize an exception detection task of the video. According to the method, the global and local time dependency relationship of the fragment is extracted by constructing a self-attention double-branch structure only by using the video-level label, and the text prompt information and the fragment features are extracted through two different modes to perform cross-modal fusion, so that the abnormal information of the video is supplemented, and the accuracy of video anomaly detection is effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular is a weakly supervised video anomaly detection method based on multiple text prompts. Background Art

[0002] The purpose of video anomaly detection is to identify abnormal events or behaviors in a video that do not conform to normal or expected patterns. With the reduction in the cost of deploying surveillance cameras, the application of video surveillance has been widely expanded to different scenarios and has shown great advantages. Therefore, intelligent surveillance video content review has become an issue of increasing concern in academia and industry. With the vigorous development of science and technology in recent years, VAD has made significant progress with the efforts of many researchers.

[0003] Unsupervised video anomaly detection has attracted extensive attention from researchers due to its ability to detect anomalies without the need for additional annotations. However, this method only involves normal videos during the training phase, resulting in limited understanding of abnormal videos. Therefore, USVAD usually regards normal events that are not seen during the training phase as abnormal, which will show a high false alarm rate. In order to solve the misidentification of video anomalies in an unsupervised environment, a more practical scenario is considered, that is, only video-level labels are available, that is, during the training process, only the video is known to be normal or abnormal, but it is not known which specific frame is abnormal. Weakly supervised video anomaly detection has the following advantages over unsupervised: 1) WSVAD combines normal and abnormal videos during the training phase, so that abnormal events can be accurately detected and identified. 2) WSVAD can achieve better performance with only video-level annotations, achieving a better balance between detection performance and manual annotation costs. 3) Introducing a small amount of annotation information helps the model understand various abnormal behaviors, so as to better generalize to new anomalies that have not been seen.

[0004] In weakly supervised video anomaly detection, most research methods follow a systematic and regular detection process, where, in the initial stage, pre-trained visual models are used to extract frame-level features from the video, such as VideoSwin, C3D, and I3D. Then, attention mechanisms or other spatiotemporal frameworks are used to capture the temporal relationship between features, which are then input into a binary classifier based on multi-instance learning and Top-K mechanisms to distinguish normal and abnormal events based on the anomaly score of each frame. This method is simple and effective, but it does not fully utilize cross-modal information to enhance the model's ability to identify anomalies.

[0005] Recently, visual language models have provided extraordinary video content understanding capabilities and rich semantic knowledge. VLMs are trained on large-scale image-text pairs using contrastive learning, effectively aligning visual and semantic spaces, such as (CLIP, ALIGN, CoCa) and have demonstrated impressive transferability across a wide range of visual tasks. Therefore, building task-specific models on CLIP is becoming an emerging research topic and has been applied to a wide range of visual tasks, achieving unprecedented performance.

[0006] However, in the weakly supervised setting, how to extend CLIP's image-text contrastive learning to more complex video anomaly detection tasks is a new challenge. Most previous methods only used CLIP's image encoder to extract the visual features of the video, turning a blind eye to the text encoder, thus ignoring CLIP's rich semantic relationship between vision and language. Therefore, in the case of weak supervision, in order to effectively utilize CLIP and enable it to fully exert its performance in WSVAD, there are still many challenges:

[0007] 1) How to effectively capture the temporal contextual dependencies between video clips and pay special attention to abnormal frames.

[0008] 2) How to extract useful semantic information from extremely limited text information.

[0009] 3) How to fuse and align visual information with label information to improve the performance of the model. Summary of the invention

[0010] The present invention aims to solve the deficiencies of the above-mentioned prior art and proposes a weakly supervised video anomaly detection method based on multiple text prompts, so as to effectively supplement the normal or abnormal information in the video, thereby improving the accuracy of anomaly detection.

[0011] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0012] The invention discloses a method for weakly supervised video anomaly detection based on multiple text prompts, comprising the following steps:

[0013] S1. Get the video-level label y and category text label A video V of , when y=1, it means that there is an abnormal event in the video V, and when y=0, it means that there is no abnormal event in the video V;

[0014] Divide the video V into T temporally non-overlapping segments ,in, represents the tth video clip, T is the number of video clips;

[0015] Using the CLIP model image encoder to obtain CLIP features to obtain the initial feature set of the video clip ,in, Indicates the tth video clip CLIP features, D represents the dimension of the clip features;

[0016] S2. Construct anomaly detection network, including global-local time dependency network, multiple text prompt network, and cross-modal fusion network;

[0017] The global-local time dependency network consists of a self-attention branch and a fragment attention branch;

[0018] The multiple text prompt network consists of a knowledge graph extraction module, a learnable prompt module and a text encoder of the CLIP model;

[0019] The cross-modal fusion network consists of a cosine similarity calculation module, an average pooling layer and a fragment aggregation module;

[0020] S21. The input is fed into the anomaly detection network and processed by the self-attention branch in the global-local temporal dependency network to obtain the global temporal dependency feature vector between each video clip. ;

[0021] at the same time, After the adaptive local mask and projection processing of the fragment attention branch, the adaptive local time-dependent feature vector is obtained. ;

[0022] Will and After adaptive fusion, the temporally enhanced segment feature set is obtained. ,in, Represents the enhanced fragment features of the tth frame;

[0023] The characteristics of the fragments enhanced by time The anomaly score and video-level label y construct the classification loss ;

[0024] S22. Label the category text Input into the multiple text prompt network and processed by the knowledge graph extraction module to obtain Similar concepts serve as supplementary prompts;

[0025] The text prompt vector to be learned is used as a prefix with After being combined, the supplementary prompts are input into the text encoder of the CLIP model for processing to obtain the prompt feature vectors to be learned. And concept tips vector ;

[0026] S23, and Input into the cross-modal fusion network and calculate and The cosine similarity between them gives the visual-text similarity matrix , then After performing Softmax operation and average pooling in sequence, the saliency score matrix of the clip is obtained. ;

[0027] Depend on and After aggregation, the video feature vector enhanced with concept hints is obtained ;

[0028] Depend on and Constructing symmetric cross entropy loss ;

[0029] S3. Use formula (1) to construct the overall loss function :

[0030] (1)

[0031] In formula (1), and is a hyperparameter; is the cross-modal contrast loss;

[0032] S4. Train the anomaly detection network using the gradient descent method and calculate the total loss function To update the network parameters until the total loss function Until convergence or reaching the maximum number of training times, the optimal anomaly detection model after training is obtained;

[0033] S5, input the test video into the trained optimal anomaly detection model for processing, and output the anomaly score of each segment; when the anomaly score of a segment is higher than a preset threshold, it indicates that an abnormal event exists in the corresponding segment;

[0034] The weakly supervised video anomaly detection method based on multiple text prompts of the present invention is also characterized in that the self-attention branch in step S21 uses three different linear layers to Projected into the same latent space, we get the query feature set Q, key feature set K and value feature set U, then dot product operation on Q and K, and then softmax operation to get the attention map ;Will After dynamic position encoding, it is multiplied by U to obtain the global time-dependent feature vector ;

[0035] The segment attention branch uses adaptive maximum pooling and adaptive average pooling to After processing, the two pooled feature sets are sent to a shared multi-layer perceptron MLP for learning, and two shallow feature sets are obtained accordingly;

[0036] After the two shallow feature sets are mapped by the Add operation and the Sigmoid activation function, the fragment attention weight matrix is ​​obtained. ;

[0037] Will After multiplying with the set parameters, the adaptive mask matrix of the fragment is obtained and projected into the attention map On the top, we get the local attention map ;

[0038] Will After multiplying by U, we get the adaptive local time-dependent eigenvector .

[0039] The knowledge graph extraction module in step S22 retrieves the After selecting the concepts with the largest correlation scores, they are input into the text encoder of the CLIP model for feature extraction to obtain the concept feature vector ;

[0040] Will After the conversion of the class tag in the Tokenizer tag converter of the CLIP model, it is embedded into the m prompt vectors to be learned In order to form a complete sentence tag Finally, Input into the text encoder of CLIP for feature extraction to obtain the prompt feature vector to be learned ,in, represents the mth prompt vector to be learned.

[0041] In step S23, As anchor points, calculate and The two-way similarity score between and , used to construct the symmetric cross entropy loss .

[0042] The cross-modal contrast loss in S3 It is constructed as follows:

[0043] S3.1. Calculation of abnormal video feature vector after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ;

[0044] S3.2. Calculation of abnormal video feature vector after concept hint enhancement Normal video feature vector enhanced with concept hints The cosine similarity between ;

[0045] S3.3. Calculate the normal video feature vector after concept hint enhancement and the normal prompt feature vector to be learned The cosine similarity between ;

[0046] S3.4. Calculate the feature vector of normal video after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ;

[0047] S3.5. , , and Constructing the cross-modal contrast loss .

[0048] In S3.5, formula (2) is used to construct the cross-modal contrast loss :

[0049] (2)

[0050] In formula (2), B represents the total number of videos. is the cosine similarity, and are the bth video V b The corresponding concept prompt enhanced abnormal video feature vector and the abnormal prompt feature vector to be learned, and The bth video V b The corresponding concept cue enhanced normal video feature vector and the normal cue feature vector to be learned.

[0051] The self-attention branch is used to use formula (3) Perform dynamic position encoding to obtain the encoded attention map :

[0052] (3)

[0053] In formula (6), and is the absolute position between any two video clips in video V, and are the weights and bias terms to be learned.

[0054] In S21, formula (4) is used to and Perform adaptive fusion:

[0055] (4)

[0056] In formula (5), represents the adaptive fusion fusion feature vector, and is obtained by formula (6), is a normalization operation, is a linear layer, is a layer normalization layer, + indicates a residual connection;

[0057] (5)

[0058] In formula (5), is the factor to be learned.

[0059] The electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the weakly supervised video anomaly detection method, and the processor is configured to execute the program stored in the memory.

[0060] The present invention provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, which is characterized in that when the computer program is run by a processor, the steps of the weakly supervised video anomaly detection method are executed.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. Aiming at the complexity of video data and the impact of multiple samples, the present invention aims to enrich and utilize video information as fully as possible. Multiple text prompts are used to introduce category-related concepts to supplement abnormal / normal information in the video, so that the model can dig out deeper hidden information in the video.

[0063] 2. The present invention takes into account the temporal differences of video data and the different abnormalities of the segments. Therefore, it accurately models the importance of each segment in the video, and then obtains the local temporal relationship of the video, so that the model can capture the fine-grained global-local temporal dependency of the video from different angles, and realizes the temporal relationship modeling at the segment level.

[0064] 3. The present invention takes into account the differences between visual-textual multimodal features, generates score significance through a video concept enhancement mechanism to assign similar concepts to each clip, and then generates a compact video representation, which is bidirectionally aligned with the cues to be learned at the video level, enriching the semantic distinction ability of the model to enhance the accuracy and efficiency of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 The flowchart of the weakly supervised video anomaly detection method based on multiple text prompts of the present invention. DETAILED DESCRIPTION

[0066] In this embodiment, a weakly supervised video anomaly detection method based on multiple text prompts first uses the image encoder of the CLIP model to extract segment features from video data, then uses the self-attention mechanism to focus on the global temporal dependencies between segments, and uses the segment attention mechanism to quantify the importance of each segment to generate a local adaptive mask matrix, and then focuses on the local temporal dependencies between segments, achieving segment-level temporal relationship modeling, and then uses two different methods to extract text prompt information, fuse concept prompt features with video features, and mine deep hidden information in the video. The learnable prompts and the fused video features are cross-modally aligned to enrich the semantic distinction ability of the model. Finally, contrastive learning is introduced to train the model to enhance the model's ability to distinguish anomalies. Specifically, if Figure 1 As shown, the method comprises the following steps:

[0067] S1. Take the video-level label y and the category text label A video V of , when y=1, it means that there is an abnormal event in the video V, and when y=0, it means that there is no abnormal event in the video V; divide the video V into T segments that do not overlap in time ,in, represents the tth video clip, T is the number of video clips; the image encoder of the CLIP model is used to obtain CLIP features to obtain the initial feature set of the video clip ,in, Indicates the tth video clip CLIP features, D represents the dimension of the clip features;

[0068] In this example, training and testing were performed on two standard video anomaly detection datasets, UCF-Crime and XD-Violence. UCF-Crime contains 1,900 surveillance videos, covering 13 abnormal event categories, of which 1,610 videos with video-level labels are used for training, and 290 videos with frame-level labels are used for testing. XD-Violence contains 4,754 videos, covering 6 abnormal event categories, of which 3,954 training videos have video-level labels, and 800 test videos have frame-level labels.

[0069] S2. Construct anomaly detection network, including global-local temporal dependency network, multiple text prompt network, and cross-modal fusion network;

[0070] The global-local temporal dependency network consists of a self-attention branch and a segment attention branch;

[0071] The multi-text prompt network consists of a knowledge graph extraction module, a prompt module to be learned, and a text encoder of the CLIP model;

[0072] The cross-modal fusion network consists of a cosine similarity calculation module, an average pooling layer, and a fragment aggregation module.

[0073] S21. The input is fed into the anomaly detection network and processed by the self-attention branch in the global-local temporal dependency network to obtain the global temporal dependency feature vector between each video clip. ;at the same time, After the adaptive local mask and projection processing of the fragment attention branch, the adaptive local time-dependent feature vector is obtained. ;Will and After adaptive fusion, the temporally enhanced segment feature set is obtained. ,in, Represents the enhanced fragment features of the tth frame;

[0074] The self-attention branch uses three different linear layers to Projected into the same latent space, we get the query feature set Q, key feature set K and value feature set U, then dot product operation on Q and K, and then softmax operation to get the attention map ;Will After dynamic position encoding, it is multiplied by U to obtain the global time-dependent feature vector .

[0075] The segment attention branch uses adaptive maximum pooling and adaptive average pooling to After processing, the two pooled feature sets are sent to a shared multi-layer perceptron MLP for learning, and two shallow feature sets are obtained accordingly;

[0076] After the two shallow feature sets are mapped by the Add operation and the Sigmoid activation function, the fragment attention weight matrix is ​​obtained. ;

[0077] Will After multiplying with the set parameters, the adaptive mask matrix of the fragment is obtained and projected into the attention map On the top, we get the local attention map ;Will After multiplying by U, we get the adaptive local time-dependent eigenvector .

[0078] The self-attention branch uses formula (1) to Perform dynamic position encoding to obtain the encoded attention map :

[0079] (1)

[0080] In formula (1), and is the absolute position between any two video clips in video V, and are the weights and bias terms to be learned.

[0081] In this example, the self-attention branch shows good performance in capturing the global temporal dependencies between video clips. Three different linear layers are used. , and The initial feature vector Projected into the same latent space, the attention map is generated using formula (2) After that, Perform dynamic position encoding and use formula (3) to obtain the global time-dependent feature vector :

[0082] (2)

[0083] (3)

[0084] In this embodiment, a projection operation is used to retain the elements in the attention map at the position where element 1 in the mask matrix is ​​located, and the remaining positions are set to 0, as shown in formula (4):

[0085] M ij l = { M ij if j ∈ [ max ( 0 , i − ⌊ W i 2 ⌋ ) ,min ( i + ⌊ W i 2 ⌋ , T ) ] −∞ otherwise (4)

[0086] in:

[0087]

[0088] in, represents a multi-layer perceptron, represents the local adaptive mask matrix, represents adaptive maximum pooling, represents adaptive average pooling, is the sigmoid function, is a parameter used to adjust the window size. is the number of segments of the input video;

[0089] Using formula (5) and Perform adaptive fusion:

[0090] (5)

[0091] In formula (5), represents the adaptive fusion fusion feature vector, and is obtained by formula (6), is a normalization operation, is a linear layer, is a layer normalization layer, + indicates a residual connection;

[0092] (6)

[0093] In formula (6), is the factor to be learned.

[0094] Finally, the feature vector of the segment enhanced by time Anomaly score With video-level tags The cross entropy loss is calculated as shown in formula (7):

[0095] L ce =− ∑ i = 0 B [ y i log ( p i ) + ( 1 − y i ) log ( 1 − p i ) ] (7)

[0096] In formula (7), B is the number of input videos, is the cross entropy loss of the anomaly detection network.

[0097] S22. Label the category text Input into the multiple text prompt network and processed by the knowledge graph extraction module to obtain Similar concepts are used as supplementary prompts; the text prompt vector to be learned is used as a prefix with After being combined, the supplementary prompts are input into the text encoder of the CLIP model for processing to obtain the prompt feature vectors to be learned. And concept tips vector ;

[0098] The knowledge graph extraction module retrieves and After selecting the concepts with the largest correlation scores, they are input into the text encoder of the CLIP model for feature extraction to obtain the concept feature vector ;Will After the conversion of the class tag in the Tokenizer tag converter of the CLIP model, it is embedded into the m prompt vectors to be learned In order to form a complete sentence tag Finally, Input into the text encoder of CLIP for feature extraction to obtain the prompt feature vector to be learned ,in, represents the mth prompt vector to be learned.

[0099] The process of using the knowledge graph to retrieve related concepts is a process that combines semantic understanding, information retrieval, and data processing. The knowledge graph is essentially a triple: <start node, retrieval relationship, end node>. Therefore, we input a given category c as the start node or end node of this triple, and the knowledge graph outputs the corresponding end node or start node according to the retrieval relationship. The CLIP model first converts the text into a class tag through a Tokenizer, and then selects the element corresponding to the maximum value index position in another tensor token to form a new tensor, as shown in formula (8):

[0100] (8)

[0101] In formula (8), It is a knowledge graph. is the number of selected concepts, It is a language tokenizer that converts words into vectors. It is the text encoder for CLIP.

[0102] S23, and Input into the cross-modal fusion network and calculate and The cosine similarity between them gives the visual-text similarity matrix , then After performing Softmax operation and average pooling in sequence, the saliency score matrix of the clip is obtained. ;Depend on and After aggregation, the video feature vector enhanced with concept hints is obtained ;

[0103] by As anchor points, calculate and The two-way similarity score between and , used to construct the symmetric cross entropy loss .

[0104] The cross-modal fusion network uses the video-concept enhancement mechanism to use the concept hint feature as a query to obtain more refined word pair segment saliency. The video-concept enhancement mechanism is the process of fusing related concepts into video segments as supplementary information. Since the related concepts are all abnormal, this may interfere with normal events in abnormal videos. Therefore, the normal segments will be suppressed and the abnormal segments will be enhanced during the aggregation process, so that the generated compact video features contain more abnormal information and exclude noise interference.

[0105] Score the similarity and With video-level tags Perform symmetric bidirectional cross entropy loss The calculation of is shown in formula (9):

[0106] (9)

[0107] in:

[0108]

[0109]

[0110] in, represents the video-to-word cross entropy loss, represents the word-to-video cross entropy loss, k ∈ K ( i ) = { k | k ∈ [ 1 , B ], y k = y i } , is the scaled temperature hyperparameter.

[0111] S25. Use formula (10) to construct the overall loss function :

[0112] (10)

[0113] In formula (10), and is a hyperparameter; is the cross-modal contrast loss and is constructed as follows:

[0114] Compute the feature vector of abnormal video after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ; Calculate the abnormal video feature vector after concept hint enhancement Normal video feature vector enhanced with concept hints The cosine similarity between ; Calculate the normal video feature vector after concept hint enhancement and the normal prompt feature vector to be learned The cosine similarity between ; Calculate the normal video feature vector after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ;Depend on , , and Use formula (11) to construct the cross-modal contrast loss .

[0115] (11)

[0116] In formula (11), B represents the total number of videos, is the cosine similarity, and are the bth video V b The corresponding concept prompt enhanced abnormal video feature vector and the abnormal prompt feature vector to be learned, and The bth video V b The corresponding concept cue enhanced normal video feature vector and the normal cue feature vector to be learned.

[0117] and It is to identify the feature representation of a video sample (whether normal or abnormal), and on this basis, evaluate its cosine similarity with the feature representation of each opposite type of video in the batch, and finally select the maximum value of these similarities as the optimization target.

[0118] The idea of ​​contrastive loss is to learn representations by comparing normal and abnormal video data samples, so that video samples of the same type are closer in the representation space, while samples of different types are farther away. Video features are used as anchors to control the boundary between normal video and category text features and abnormal video and category text features, thereby achieving good separability between normal and abnormal.

[0119] S3. Use the gradient descent method to train the anomaly detection network and calculate the total loss function To update the network parameters until the total loss function Until convergence or reaching the maximum number of training times, the optimal anomaly detection model after training is obtained;

[0120] S4, input the test video into the trained optimal anomaly detection model for processing, and output the anomaly score of each segment; when the anomaly score of the segment is higher than the preset threshold, it indicates that an abnormal event exists in the segment;

[0121] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0122] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

[0123] The present invention will Set to 20 and 14 respectively, Set to 0.01. During the training process, the batch size is set to 32, the number of training rounds is 50, the initial learning rate is initialized to 5.e-4, the model of the CLIP model is ViT-B / 16, and the number of clips of the training video is cropped or expanded to 256. Training and testing are carried out in the environment of Python3.9 and torch2.1.0. The experiment is carried out using Nvidia GeForce RTX 4060 GPU:

[0124] Table 1 is a graph showing the experimental results of the present invention and the existing anomaly detection method.

[0125]

[0126] Table 1 shows the performance of different methods, comparing AUC and AP on UCF-Crime and XD-Violence, including the proposed method and eight baseline methods. The eight baseline methods are: RTFM is Robust Event Feature Magnitude Learning, from the paper: "Weakly-supervised video anomaly detection with robust temporal feature magnitude learning; published at ICCV 2021; CRFD stands for Causal Temporal Relation Learning, which comes from the paper: Learning Causal Temporal Relation and Feature Discrimination for Anomaly Detection, published in IEEE Transactions on Image Processing in 2021; MSL is self-training multi-sequence learning, from the paper: "Self-training multi-sequence learning with transformer for weaklysupervised video anomaly detection", published at AAAI in 2022; MGFN is a Magnitude-contrastive glance-and-focus network, from the paper: "Mgfn: Magnitude-contrastive glance-and-focus network for weakly-supervised video anomaly detection.", published in AAAI 2023; UR-DMU is an anomaly detection network based on dual memory units with uncertainty regulation, from the paper: "Dual memory units with uncertainty regulation for weakly supervised video anomaly detection", published in AAAI 2023; CLIP-TSA is a clip-assisted temporal self-attention anomaly detection network, from the paper: "Clip-tsa: Clip-assisted temporal self-attention for weakly-supervised video anomaly detection", published in ICIP in 2023; TPWNG is a text prompt anomaly detection network based on normality guidance, from the paper: "Text prompt with normality guidance for weakly supervised video anomaly detection", published in CVPR in 2024; VadCLIP is a visual language model adapted for weakly supervised video anomaly detection, from the paper: "Vadclip: Adapting vision-language models for weakly supervised video anomaly detection", published in AAAI in 2024;

[0127] The proposed method outperforms the baseline method in every evaluation metric and achieves significant improvements over the baseline, demonstrating the superiority of the proposed model.

[0128] In summary, the present invention captures fine-grained global-local temporal dependencies of videos from different angles, and uses multiple text prompts to introduce category-related concepts to supplement abnormal / normal information in videos, thereby mining deeper hidden information in videos and providing clearer decision boundaries for videos of different categories, thereby improving the accuracy of video anomaly detection.

Claims

1. A weakly supervised video anomaly detection method based on multiple text prompts, characterized in that: The following steps are involved: S1. Get the video-level label y and category text label A video V of , when y=1, it means that there is an abnormal event in the video V, and when y=0, it means that there is no abnormal event in the video V; Divide the video V into T temporally non-overlapping segments ,in, represents the tth video clip, T is the number of video clips; Using the CLIP model image encoder to obtain CLIP features to obtain the initial feature set of the video clip ,in, Indicates the tth video clip CLIP features, D represents the dimension of the clip features; S2. Construct anomaly detection network, including global-local time dependency network, multiple text prompt network, and cross-modal fusion network; The global-local time dependency network consists of a self-attention branch and a fragment attention branch; The multiple text prompt network consists of a knowledge graph extraction module, a learnable prompt module and a text encoder of the CLIP model; The cross-modal fusion network consists of a cosine similarity calculation module, an average pooling layer and a fragment aggregation module; S21. The input is fed into the anomaly detection network and processed by the self-attention branch in the global-local temporal dependency network to obtain the global temporal dependency feature vector between each video clip. ; at the same time, After the adaptive local mask and projection processing of the fragment attention branch, the adaptive local time-dependent feature vector is obtained. ; Will and After adaptive fusion, the temporally enhanced segment feature set is obtained. ,in, Represents the enhanced fragment features of the tth frame; Fragment features enhanced by time The anomaly score and video-level label y construct the classification loss ; S22. Label the category text Input into the multiple text prompt network and processed by the knowledge graph extraction module to obtain Similar concepts serve as supplementary prompts; The text prompt vector to be learned is used as a prefix with After being combined, the supplementary prompts are input into the text encoder of the CLIP model for processing to obtain the prompt feature vectors to be learned. And concept tips vector ; S23, and Input into the cross-modal fusion network and calculate and The cosine similarity between them gives the visual-text similarity matrix , then After performing Softmax operation and average pooling in sequence, the saliency score matrix of the clip is obtained. ; Depend on and After aggregation, the video feature vector enhanced with concept hints is obtained ; Depend on and Constructing symmetric cross entropy loss ; S3. Use formula (1) to construct the overall loss function : (1) In formula (1), and is a hyperparameter; is the cross-modal contrast loss; S4. Train the anomaly detection network using the gradient descent method and calculate the total loss function To update the network parameters until the total loss function Until convergence or reaching the maximum number of training times, the optimal anomaly detection model after training is obtained; S5. Input the test video into the trained optimal anomaly detection model for processing, and output the anomaly score of each segment; when the anomaly score of a segment is higher than a preset threshold, it indicates that an abnormal event exists in the corresponding segment.

2. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 1 is characterized in that: The self-attention branch in step S21 uses three different linear layers to Projected into the same latent space, we get the query feature set Q, key feature set K and value feature set U, then dot product operation on Q and K, and then softmax operation to get the attention map ;Will After dynamic position encoding, it is multiplied by U to obtain the global time-dependent feature vector ; The segment attention branch uses adaptive maximum pooling and adaptive average pooling to After processing, the two pooled feature sets are sent to a shared multi-layer perceptron MLP for learning, and two shallow feature sets are obtained accordingly; After the two shallow feature sets are mapped by the Add operation and the Sigmoid activation function, the fragment attention weight matrix is ​​obtained. ; Will After multiplying with the set parameters, the adaptive mask matrix of the fragment is obtained and projected into the attention map On the top, we get the local attention map ; Will After multiplying by U, we get the adaptive local time-dependent eigenvector .

3. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 1 is characterized in that: The knowledge graph extraction module in step S22 retrieves the After selecting the concepts with the largest correlation scores, they are input into the text encoder of the CLIP model for feature extraction to obtain the concept feature vector ; Will After the conversion of the class tag in the Tokenizer tag converter of the CLIP model, it is embedded into the m prompt vectors to be learned In order to form a complete sentence tag Finally, Input into the text encoder of CLIP for feature extraction to obtain the prompt feature vector to be learned ,in, represents the mth prompt vector to be learned.

4. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 1 is characterized in that: In step S23, As anchor points, calculate and The two-way similarity score between and , used to construct the symmetric cross entropy loss .

5. The video anomaly detection method based on weak supervision of multiple text prompts according to claim 1 is characterized in that: The cross-modal contrast loss in S3 It is constructed as follows: S3.

1. Calculation of abnormal video feature vector after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ; S3.

2. Calculation of abnormal video feature vector after concept hint enhancement Normal video feature vector enhanced with concept hints The cosine similarity between ; S3.

3. Calculate the normal video feature vector after concept hint enhancement and the normal prompt feature vector to be learned The cosine similarity between ; S3.

4. Calculate the feature vector of normal video after concept hint enhancement and the abnormal prompt feature vector to be learned The cosine similarity between ; S3.

5. , , and Constructing the cross-modal contrast loss .

6. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 5 is characterized in that: In S3.5, formula (2) is used to construct the cross-modal contrast loss : (2) In formula (2), B represents the total number of videos. is the cosine similarity, and are the bth video V b The corresponding concept prompt enhanced abnormal video feature vector and the abnormal prompt feature vector to be learned, and The bth video V b The corresponding concept cue enhanced normal video feature vector and the normal cue feature vector to be learned.

7. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 2 is characterized in that: The self-attention branch is used to use formula (3) Perform dynamic position encoding to obtain the encoded attention map : (3) In formula (6), and is the absolute position between any two video clips in video V, and are the weights and bias terms to be learned.

8. The weakly supervised video anomaly detection method based on multiple text prompts according to claim 1 is characterized in that: In S21, formula (4) is used to and Perform adaptive fusion: (4) In formula (5), represents the adaptive fusion fusion feature vector, and is obtained by formula (6), is a normalization operation, is a linear layer, is a layer normalization layer, + represents a residual connection; (5) In formula (5), is the factor to be learned.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute any one of the weakly supervised video anomaly detection methods described in claims 1-7, and the processor is configured to execute the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the weakly supervised video anomaly detection method described in any one of claims 1 to 7 are executed.

Images