Weak supervision video anomaly detection method for text interaction context features

Abstract

The invention relates to a weak supervision video anomaly detection method for text interaction context features. The method comprises a global local context framework, a cross-modal interaction framework and a feature fusion and multi-instance learning framework. Obtaining an abnormal detection video data set and a corresponding text dictionary, and dividing the abnormal detection video data set and the corresponding text dictionary into positive / abnormal video data sets; normal / abnormal context video features are generated in a global local context framework through I3D features obtained by the normal / different video data sets through an I3D feature extraction network; meanwhile, a video data set and a corresponding text dictionary are trained from a CLIP model, video features and text features are generated, and text guide video features are generated in a cross-modal interaction framework; and the normal / abnormal context video features and the text guidance video features are correspondingly fused in a feature fusion and multi-instance learning framework to obtain a weak supervision video anomaly detection result. The performance of weak supervision video anomaly detection can be remarkably improved, and the application scene of the method is widened.

Description

Technical Field

[0001] The technical solution of the present invention relates to the technical field of video anomaly detection, and specifically to a weakly supervised video anomaly detection method based on text interaction context features. Background Art

[0002] In recent years, weakly supervised video anomaly detection (WSVAD) has become a research hotspot due to its great application potential in intelligent video surveillance, video content analysis and other fields. WSVAD aims to infer frame-level anomaly confidence only by relying on video-level labels. Its mainstream research framework usually follows the following steps: first, pre-trained visual models (such as Convolutional 3D Networks, C3D; Inflated 3D ConvNets, I3D; Temporal Segment Networks, TSN; and VideoSwin, etc.) are used to extract the spatiotemporal features of the video and construct a rich video representation; then, a scoring module is designed, which takes deep features as input and calculates anomaly scores for each video frame or segment under weak supervision of video-level labels. The weakly supervised video anomaly detection strategy focuses on using incomplete or precision-limited label information for model training, aiming to reduce the complexity of data annotation while maintaining detection performance. Specifically, weakly supervised methods usually rely on video-level labels rather than detailed frame-by-frame or pixel-level annotations, which significantly reduces the annotation cost.

[0003] There are two types of weakly supervised video anomaly detection methods at present: a one-stage method based on multiple instance learning (MIL) and a two-stage method based on pseudo-label self-training. The method based on multiple instance learning does not need to generate pseudo labels during model training, and directly detects anomalies through the multi-instance features of video clips, which simplifies the training process. Since learning is performed directly in a single stage, this method can converge faster and has high efficiency when processing large amounts of data. In addition, the MIL method can effectively use the implicit information in the data for anomaly detection when the annotation information is limited, and performs well in some application scenarios. However, the MIL method may be limited by insufficient supervision information, resulting in limited room for improving the performance of the model.

[0004] Traditional multi-instance learning methods lack sufficient supervision when dealing with various abnormal situations, resulting in misclassification and high false alarm rates. In video sequences, how to effectively capture and fuse global and local contextual information to improve the accuracy of anomaly detection; existing methods are mainly based on a single modality or a combination of multiple modalities based on word text, and fail to fully utilize the potential of each modality. Although some existing methods have begun to use text-visual models, they often use text features only as an auxiliary to video features and fail to fully utilize the potential of text information.

[0005] For example, CN118196694A discloses a weakly supervised video anomaly detection method and system under text prompt learning. This method only uses abnormal video data and corresponding text encoding features, uses a multi-granularity multi-scale time attention network to encode video features, introduces a memory module to calculate the similarity of the two features to enhance the video features, and finally uses cross attention for fusion. The generalization ability in complex multi-scenes needs to be verified, and the fuzziness of the text and the diversity of expressions affect the accuracy of anomaly detection to some extent. CN109508671B discloses a video abnormal event detection system and method based on weakly supervised learning. A normal / abnormal event classifier is constructed by applying a weakly supervised learning method to a massive weakly annotated video data set. However, weakly annotated data (only annotating whether the video contains an abnormal event) may introduce noise and uncertainty. Multi-level appearance-motion joint expression features are used, but the effectiveness of this feature extraction method may vary in different types of videos and scenes. Summary of the invention

[0006] In view of the shortcomings of the prior art, the technical problem to be solved by the present invention is to provide a weakly supervised video anomaly detection method based on text interaction context features. In this method, text information is independently modeled and feature extracted, and as an input source that is equally important as video features, it fully participates in the training and reasoning process of the model, thereby reflecting its non-auxiliary role in anomaly detection. Through the joint attention mechanism and multimodal cross network, the visual content features and text features of the video frame are effectively combined to achieve accurate multi-instance learning and anomaly detection.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A weakly supervised video anomaly detection method based on text interaction context features, including the global local context framework GLCA, the cross-modal interaction framework CIM, feature fusion and multi-instance learning framework;

[0009] Obtain an anomaly detection video data set and a text dictionary consisting of one-to-one corresponding texts, and divide the data into a normal video data set and an abnormal video data set, wherein the video clips in the normal video data set are all normal behaviors, and the video clips in the abnormal video data set contain normal behaviors and abnormal behaviors;

[0010] In the global-local context framework, it includes I3D feature extraction network, global context module, local context module, adaptive fusion module and multi-layer perceptron.

[0011] The global context attention module adopts the Transformer attention module to capture the global context dependencies of the video clips to model the complete temporal relationship;

[0012] The local context attention module also adopts the Transformer attention module, and adds mask operation and temporal mask operation to it. The output of the local context attention module is expressed by formula (2):

[0013]

[0014] Among them, D k is the dimension of the embedding vector. m (i,j)=-[|ij| / e β ] represents the distance between the i-th position and the j-th position in time, β is a sensitivity factor used to balance the local distance; f v (X I3D ) is the value vector V after linear projection; M ij is the output feature of the masking operation; softmax[T m ·f v (X I3D )] indicates the timing Mask operation; is the local context feature;

[0015] M ij The expression is:

[0016]

[0017] Where T represents the size of the mask window; j and i range from 0 to T-1; w is a hyperparameter;

[0018] The normal video dataset and the abnormal video dataset are respectively subjected to the I3D feature extraction network to obtain I3D features, and then enter the global and local context modules as well as the adaptive fusion module and the multi-layer perceptron MLP to generate normal / abnormal context video features;

[0019] In the cross-modal interaction framework, the video dataset and the corresponding text dictionary are trained from the CLIP model to generate video features and text features respectively, and enter the cross-modal interaction module to calculate the weights between text and frame features and generate text-guided video features. The expression of the cross-modal interaction module is formula (7) and (8):

[0020]

[0021]

[0022] Among them, α k represents the text conditional weight of the kth frame in the video feature v', f' k represents the temporal features of the kth frame in the video feature v'; τ represents a scaling factor; is the text feature; F t It is the text-guided video feature;

[0023] In the feature fusion and multi-instance learning framework, normal context video features and normal text-guided video features, abnormal context video features and abnormal text-guided video features are fused together for multi-instance learning loss ranking to obtain weakly supervised video anomaly detection results.

[0024] Furthermore, the specific process in the global-local context framework is: the output of the I3D feature extraction network enters the global context module and the local context module respectively, the output of the global context module and the local context module are processed by the adaptive fusion module and then residually connected with the output of the I3D feature extraction network through normalization, and finally processed by the multi-layer perceptron MLP and layer normalization LN to obtain the context video features.

[0025] Furthermore, the expression of the adaptive fusion module is: Among them, ,λ and 1-λ represent the global context feature A and the local context feature respectively. is the fusion weight of the context, and X is the global and local features output by the adaptive fusion module.

[0026] Furthermore, in the feature enhancement fusion and multi-instance learning framework, a feature fusion enhancement module is used to transform the normal context video features into and normal text guided video features Perform feature fusion, similarly, abnormal context video features and abnormal text-guided video features Perform feature fusion to obtain the normal fusion feature F a and abnormal fusion feature F n , the formula is:

[0027]

[0028] Among them, ψ(·) is the feature fusion enhancement module, which represents the feature alignment and fusion operation;

[0029] Construct a normal / abnormal behavior classification model, input the normal fusion features and abnormal fusion features into the multi-layer perceptron and causal convolution layer respectively to predict the abnormal score S of the video clip a and S n , the formula of the normal / abnormal behavior classification model is:

[0030]

[0031] Among them, f t(·) is a causal convolutional layer with a kernel size of Δt, σ(·) is a sigmoid activation function, is the anomaly score of the i-th video clip in the anomaly fusion feature; is the abnormality score of the i-th video clip in the normal fusion feature; MLP is a multi-layer perceptron;

[0032] The multi-instance learning loss function uses binary cross entropy loss to obtain the binary cross entropy between the frequency-level prediction and the true label to calculate the classification loss, and determines whether the video contains anomalies based on the predicted anomaly score.

[0033] Compared with the existing known technologies, the technical solution provided by the present invention has the following significant effects:

[0034] (1) The present invention proposes a new global-local context framework, which not only improves efficiency through the similarity matrix, but also captures the features and information of the entire video sequence and over a long period of time; the local context focuses on the details and features of abnormal behaviors within a smaller range in the video sequence. Therefore, global information can help the model understand whether the current behavior conforms to the overall background and scene, while local information can capture specific abnormal actions or changes. At the same time, the present invention makes more comprehensive use of multimodal information, taking text information as an input source that is equally important as video features, and fully participating in the training and reasoning process of the model to improve the effect of weakly supervised video anomaly detection. Through the cross-modal interaction framework, text information is not only used as an auxiliary to video features, but is deeply integrated with video features to enhance the semantic understanding of video features. This method can provide more accurate and more explanatory results in anomaly detection, effectively reducing misclassification and high false alarm rates.

[0035] (2) Compared with the CN118196694A method, the method of the present invention combines the global attention mechanism with the local context calibration. Compared with the simple multi-granularity multi-scale temporal attention network, the present method significantly reduces the computational cost by introducing a masking window to limit the calculation scope. The text encoder and video encoder of the CLIP model are used to extract features, and the weights between the text and frame features are calculated and aggregated so that the video features can better reflect the semantic association between the video and the text. This approach not only enhances the utilization of text information, but also improves the generalization ability of the model in complex scenes through a multi-cue learning strategy.

[0036] (3) Compared with the CN117351392A method, the method of the present invention uses a global local context attention module. Compared with the local temporal modeling and global temporal modeling used in the VadCLIP model, the method of the present invention focuses more on capturing the long-term features of the entire video sequence through the global context, and uses the local context to focus on the details of abnormal behavior within a smaller range in the video sequence. This method can not only help the model understand whether the current behavior conforms to the overall background and scene, but also better capture specific abnormal actions or changes. In addition, the VadCLIP model mainly uses the frozen CLIP image encoder and text encoder to extract image features and text label features, and realizes video abnormal behavior detection through coarse-grained binary classification and fine-grained multi-classification. The present invention proposes a new fusion mechanism, which uses a multimodal network and a cross-attention mechanism to take text information as an input source that is equally important as video features, and fully participates in the training and reasoning process of the model. This method can provide more accurate and more interpretable results in anomaly detection, effectively reducing misclassification and high false alarm rates.

[0037] (4) The multimodal fusion mechanism of the present invention fully utilizes the complementary advantages of multiple modalities, can significantly improve the performance of weakly supervised video anomaly detection, and broaden its application scenarios. It fully utilizes text information to enhance the semantic understanding of video features, and effectively integrates the content features and text features of video frames to achieve multi-instance learning, thereby improving the anomaly detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0039] Figure 1 The figure is a flowchart of the weakly supervised video anomaly detection method based on text interaction context features of the present invention.

[0040] Figure 2 Schematic diagram of the overall structure of the weakly supervised video anomaly detection method based on text interaction context features of the present invention.

[0041] Figure 3 Schematic diagram of the structure of the first-stream global-local context framework in the present invention.

[0042] Figure 4 Schematic diagram of the structure of the global context module (left) and the local context module (right) in the present invention.

[0043] Figure 5 It is a schematic diagram of the structure of the second stream cross-modal interaction framework in the present invention. DETAILED DESCRIPTION

[0044] The invention discloses a weakly supervised video anomaly detection method of text interaction context features, and the overall framework has three major frameworks, namely, a global-local context framework (GLCA), a cross-modal interaction framework (CIM), and a feature fusion and multi-instance learning framework. The first stream, the global-local context framework, is an I3D feature obtained by the I3D feature extraction network of the normal video data set and the abnormal video data set, and then enters the global, local context module and the adaptive fusion module to generate normal / abnormal context video features. Then the second stream, the cross-modal interaction framework, the video data set and the corresponding text dictionary are trained from the CLIP model respectively, and video features and text features are generated respectively, and the cross-modal interaction module is entered to calculate the weight between the text and frame features, and generate text-guided video features. Finally, the feature fusion and multi-instance learning framework fuses the context video features and the text-guided video features into the multi-instance learning loss ranking.

[0045] Based on weakly supervised multi-instance learning: The fused multimodal features are input into the multi-instance learning loss ranking, and the video data is divided into multiple instances containing several frames through instance grouping. Then each instance is scored, the anomaly score is calculated, and the score is optimized using the multi-instance learning loss function. The instances are classified according to their anomaly scores. Instances with higher scores are judged as abnormal, and instances with lower scores are judged as normal. The scores of all instances are combined to generate the overall anomaly score of the video, and the video is judged to be abnormal based on the set threshold to generate the final detection result.

[0046] The specific steps are as follows:

[0047] The first stream builds a global-local context framework, which includes a P3D feature extraction network, a global context module, a local context module, an adaptive fusion module, and a multi-layer perceptron (MLP);

[0048] Use the I3D feature extraction network to extract features from the input normal and abnormal video data to obtain spatiotemporal multi-scale feature I3D features, which serve as the basic input for subsequent processing;

[0049] The global context module adopts the Transformer attention module;

[0050] The local context module includes a Transformer attention module, a mask operation and a temporal Mask operation;

[0051] The outputs of the I3D feature extraction network enter the global context module and the local context module respectively. The outputs of the global context module and the local context module are processed by the adaptive fusion module and then residually connected with the output of the I3D feature extraction network through normalization. Finally, the contextual video features are obtained after processing by the multi-layer perceptron MLP and layer normalization LN.

[0052] The second stream is a cross-modal interaction framework, which includes a CLIP visual encoder, a CLIP text encoder and a cross-modal interaction module;

[0053] The cross-modal interaction module includes a cross-attention mechanism of the Transformer encoder, a similarity calculation and a feature weighting unit;

[0054] Dataset text information is a dictionary of text information for each normal / abnormal video. Take the UCF-Crime dataset as an example. The video number is "Abuse / Abuse002_x264.mp4". The text information of the video is: "Akid fell out of the trunk of a van and was crushed by the car behind, then the nearby crowds came to help.";

[0055] The training normal video and the corresponding text information, the abnormal video and the corresponding text information are respectively fed into the CLIP visual encoder and CLIP text encoder Get normal / abnormal video features and text features;

[0056] Taking the normal video data set input as an example, the normal video and the corresponding text information enter the CLIP visual encoder and CLIP text encoder Get normal video features v' and text features t';

[0057] Next, generate text-guided video features, calculate the weights between normal video features v' and text features t' (frame features and text), aggregate the frame features, and finally output normal text-guided video features

[0058]

[0059] Among them, α k represents the text conditional weight of the k-th frame feature, f' k represents the temporal features of the kth frame. τ represents a scaling factor used to control the smoothness in weight calculation.

[0060] Output normal text guided video features Similarly, the abnormal video data set is input, and finally the abnormal normal text guide video features are output

[0061] The normal / abnormal context video features are output by the first-stream GLCA, respectively. and

[0062] The normal / abnormal text-guided video features are represented by the CIM output of the second stream. and

[0063] The normal context video features and the normal text-guided video features are fused. Similarly, the abnormal context video features and the abnormal text-guided video features are fused. The formula is as follows:

[0064]

[0065] Among them, ψ(·) is the feature fusion enhancement module, which represents the feature alignment and fusion operation. The normal feature fusion F is obtained. a and abnormal feature fusion F n ;

[0066] A multiple instance learning module, including causal convolutional layers and multi-layer perceptrons, to predict segment-level anomaly scores, taking into account current and historical observations;

[0067] Using the multi-instance learning module, the normal features are fused into F a and abnormal feature fusion F n Entering the causal convolutional layer and multi-layer perceptron, the normal and abnormal score formulas are as follows: S a =σ[f t (MLP(F a ))] and S n =σ[f t (MLP(F n ))].

[0068] Binary Cross Entropy Loss: The final loss function is calculated by the binary cross entropy between the predicted video-level labels and the true labels. The average of the anomaly confidences from the normal videos and the highest anomaly confidence value from the anomaly videos are selected as the video-level prediction for the test video. The classification loss is then calculated using the binary cross entropy between this video-level prediction and the true label.

[0069] The mask operation in the present invention uses the similarity matrix of the masking window to calibrate the local context to ensure that only the local area is focused.

[0070] In the feature fusion and multi-instance learning framework:

[0071] Causal convolution layer: performs temporal convolution on the fused features to capture the temporal dependency of the features;

[0072] Multilayer Perceptron: The convolutional features are input into the multilayer perceptron to further extract high-level features and predict scores;

[0073] Sigmoid activation function: After the output of the multi-layer perceptron, the score is normalized using the Sigmoid activation function.

[0074] In the embodiment, through the above-mentioned video abnormal behavior detection method, GLCA is used to capture the temporal dependencies between video frames from both global and local levels, thereby improving the model's ability to understand video temporal features. The use of CIM text information can provide contextual guidance for video features, help the model better understand and identify complex abnormal behaviors, and enhance the richness and discrimination ability of feature representation. The feature fusion and multi-instance learning frameworks enable the above two frameworks to effectively adapt to weakly supervised video abnormal behavior detection tasks, thereby making full use of the knowledge of pre-trained models. Through a cross-modal mechanism based on multi-instance learning, the performance of video anomaly detection under weak supervision is further optimized, ensuring the effective fusion of features of different modalities, thereby improving the accuracy and robustness of anomaly detection.

[0075] Example 1

[0076] Part I: Global-local Contextual Attention Mechanism

[0077] Step 1: Data Preparation

[0078] We regard video anomaly detection as a weakly supervised multi-instance learning task. Specifically,

[0079] The normal video dataset sequence is defined as It includes M a Normal video clips and its corresponding label y i ; Define the abnormal video dataset sequence as It includes M n Abnormal video clips and its corresponding label y i ; The video clips in the normal video dataset are all normal behaviors, and the video clips in the abnormal video dataset contain normal behaviors and abnormal behaviors.

[0080] Each video clip v i Corresponding to a video-level label y i , label y iTaken from the set range {0,1}, where y=0 means that the video is completely normal and does not contain any abnormal behavior segments; and y=1 means that there is at least one abnormal segment in the video;

[0081] Step 2: Preprocess the video

[0082] The I3D (Inflated 3D) model pre-trained on the Kinetics dataset is used as the backbone network to extract the RGB features of each consecutive 16 frames as a video clip and splice them in the time dimension to obtain the spatiotemporal multi-scale features. Where N represents the number of video clips in the normal video dataset or the abnormal video dataset, and F represents the feature dimension of each clip;

[0083] Step 3: Global Contextual Attention Module

[0084] The Transformer attention module is adopted through the global contextual attention module to capture the global contextual dependencies of the fragments to model the complete temporal relationship.

[0085] Spatiotemporal multi-scale features X I3D First, different linear layer projections are used in the linear projection layer to obtain the query vector Q (query), key vector K (key) and value vector V (value), which are expressed as Q = f q (X I3D ), K = f k (X I3D ) and V = f v (X I3D ).

[0086] Then the global context feature A is obtained through the self-attention mechanism, which is the output of the global context attention module, see formula (1)

[0087]

[0088] Among them, D k is the dimension of the embedding vector (same as the dimension of D), T represents the transpose, and W Q , W K and W V is the weight matrix, responsible for converting X I3D Mapped to a feature vector with deeper semantics; Attention represents the self-attention function; The scaled dot product result Input into the softmax function to calculate the attention weight of each K relative to Q. The Softmax function converts the original score into a probability distribution so that the sum of the attention weights of all Keys is 1. The processing result of the Softmax function is then multiplied by V to obtain A.

[0089] Because the attention mechanism in the Transformer model allows the model to consider the information of all positions in the sequence at the same time when processing video sequences, thereby promoting the efficiency of global context modeling;

[0090] Step 4: Local Context Attention Module

[0091] The local context attention module also uses the Transformer attention module, and adds mask operation and temporal mask operation to it. The output of the local context attention module is expressed by formula (2):

[0092]

[0093] Among them, D k is the dimension of the embedding vector (which is the same as the dimension of the linear projection Q). m (i,j)=-[|ij| / e β ] represents the distance between the i-th position and the j-th position in time, β is a sensitivity factor used to balance the local distance; f v (X I3D ) is the value vector V after linear projection, M ij is the output feature of the masking operation; softmax[T m ·f v (X I3D )] indicates the timing Mask operation; is a local context feature.

[0094] Define the query vector Q and key vector K after linear projection and their respective weight matrices W Q , W K The result of the dot product after multiplication is used as the similarity matrix, and the similarity matrix is ​​used to perform a masking operation to calibrate the local context. The specific definition of the masking operation is formula (3):

[0095]

[0096] Among them, M ij Represents the features of the mask output, that is, the output features of the mask operation, which represents the original similarity or correlation between the j-th position and the i-th position in the mask window, and the value range is an integer from 0 to T-1; w is a hyperparameter, and T represents the size of the mask window.

[0097] Step 5: Global and local adaptive fusion

[0098] The global context feature A and the local context feature A are combined using the adaptive fusion module Fusion is performed to achieve local-global fusion, and the formula is:

[0099]

[0100] Among them, λ and 1-λ represent the global context feature A and the local context feature respectively. is the fusion weight of the context, and X is the global and local features output by the adaptive fusion module.

[0101] Step 6: Normalization & MLP

[0102] The global local feature X is normalized by combining power normalization and L2 normalization, and then X I3D Perform residual connection, then pass through multi-layer perceptron and layer normalization LN, and finally get the context video feature F v , the specific formula is as follows:

[0103] F v =LN[MLP(Norm(X)+X I3D )] (5)

[0104] Among them, Norm(·) combines power normalization and L2 normalization to adjust the data scale.

[0105] Input the normal video dataset into the GLCA framework to obtain normal context video features Similarly, the anomaly dataset is input into the GLCA framework, and finally the anomaly context video features are output

[0106] Part II: Cross-modal Interaction Framework CIM

[0107] Step 7: Extract text features and video features

[0108] First, the dataset definition D = {V, T} is a set of anomaly detection video dataset and a one-to-one corresponding text, where V represents a video clip and T is the corresponding sentence of the video clip; the anomaly detection video dataset includes a normal video dataset and an abnormal video dataset; the number of video clips in a normal video dataset or an abnormal video dataset is recorded as N;

[0109] For extracting text features, given a w Words of input text Text encoder using pre-trained CLIP model Extract word features Then [EOS] is embedded into the word feature as the sentence feature Where D represents the feature dimension; text encoder The output is the text feature

[0110] To extract video features, firstly, a t Video sample of frames Split into non-overlapping blocks of fixed size and add a [CLS] token in front as a preprocessed video frame; then, feed the preprocessed video frame into the video encoder of the pre-trained CLIP model Extract frame features; then combine [CLS] token and frame features as input, and use Temporal Transformer to obtain the temporal perception in the frame sequence. The final output formula for extracting video features is:

[0111]

[0112] Represents a video encoder, where v represents

[0113] f' k represents the temporal features of the kth frame; f' represents the temporal features output by Temporal Transformer;

[0114] N t Represents the number of frames uniformly sampled from the video.

[0115] Step 8: Generate text-guided video features

[0116] As a granular interaction mechanism, the cross-modal interaction module aims to capture the global semantics between sentences and videos, calculate the weights between text and frame features, and aggregate frame features to obtain text-guided video features F. t , the specific expression is shown in formulas (7) and (8):

[0117]

[0118] Among them, α k represents the text conditional weight of the kth frame in the video feature v', f' k represents the temporal features of the kth frame in the video feature v'; τ represents a scaling factor used to control the smoothness in the weight calculation; is the text feature; F t It is a text-guided video feature that integrates the semantic information of the text description, so that the text-guided video feature can better reflect the semantic relationship between the video and the text;

[0119] For normal video datasets and corresponding text features, normal text-guided video features are obtained after CIM processing. For abnormal video datasets and corresponding text features, abnormal text-guided video features are obtained after CIM processing. Part III: Feature Enhancement Fusion and Multi-Instance Learning Framework

[0120] Step 9: Feature Enhancement Fusion

[0121] The first part above outputs contextual video features through the global local context framework GLCA and In the second part, the text output guides the video features through the cross-modal interaction framework CIM and

[0122] The feature fusion enhancement module is used to fuse the normal context video features and the normal text-guided video features. Similarly, the abnormal context video features and the abnormal text-guided video features are fused to obtain the normal fusion feature F a and abnormal fusion feature F n , the formula is as follows:

[0123]

[0124]

[0125] Among them, ψ(·) is the feature fusion enhancement module, which represents the feature alignment and fusion operation.

[0126] Step 10: Build a normal / abnormal behavior classification model

[0127] The normal fusion features and abnormal fusion features are input into the multi-layer perceptron and causal convolution layer respectively to predict the abnormal score S of the video clip a and S n , this layer takes into account current and historical observations to obtain more reliable results. The formula for the normal / abnormal behavior classification model is:

[0128]

[0129] Among them, f t (·) is a causal convolutional layer with a kernel size of Δt, σ(·) is a sigmoid activation function, is the anomaly score of the i-th video clip in the anomaly fusion feature; is the abnormality score of the i-th video clip in the normal fusion feature; MLP is a multi-layer perceptron;

[0130] Step 11: Multi-Instance Learning Loss Function

[0131] Next, we adopt a multi-instance learning-based loss as the basic objective function, where labels are provided for bags of instances rather than for individual instances within a bag. In this context, a bag usually contains multiple instances (in this case, segments of a video). Each bag can be labeled as a positive bag or a negative bag. Specifically, the video-level prediction p is determined by taking the average of the top k anomaly scores. i ;

[0132] For the positive bag, select the first The average of the anomaly scores is used to calculate the video-level prediction value p i ;

[0133] For the negative bag, select the first k=1 anomaly scores to calculate the video-level prediction value p i ;

[0134] Then given a mini-batch containing B samples, with video-level Ground-truth y i , therefore, the binary cross entropy based on multi-instance learning is expressed as:

[0135]

[0136] B represents the number of video samples in a minibatch, not a single video sample. Each minibatch usually contains multiple video samples for parallel processing and accelerated calculation during training. n represents the nth video sample in a minibatch, which refers to an index in minibatch B, used to traverse each video sample in the minibatch.

[0137] Binary cross entropy loss: The final loss function is obtained by predicting the video level label p i and the true label y i The binary cross entropy between . Therefore, the average of k anomaly confidences from the normal video and the highest anomaly confidence value from the anomaly video are selected as the video-level prediction of the test video. The classification loss is then calculated using the binary cross entropy between this video-level prediction and the true label;

[0138] In the testing phase, the predicted anomaly score of a video is expressed as where s j is the anomaly score of the jth segment. The goal of anomaly detection is to accurately locate the interval and determine whether the video contains anomalies based on the predicted anomaly score.

[0139] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A weakly supervised video anomaly detection method based on text interaction context features, characterized in that: Including the global local context framework GLCA, the cross-modal interaction framework CIM, feature fusion and multi-instance learning framework; Obtain an anomaly detection video data set and a text dictionary consisting of one-to-one corresponding texts, and divide the data into a normal video data set and an abnormal video data set, wherein the video clips in the normal video data set are all normal behaviors, and the video clips in the abnormal video data set contain normal behaviors and abnormal behaviors; In the global-local context framework, it includes I3D feature extraction network, global context module, local context module, adaptive fusion module and multi-layer perceptron. The global context attention module adopts the Transformer attention module to capture the global context dependencies of the video clips to model the complete temporal relationship; The local context attention module also adopts the Transformer attention module, and adds mask operation and temporal mask operation to it. The output of the local context attention module is expressed by formula (2): Among them, D k is the dimension of the embedding vector. m (i,j)=-[|ij| / e β ] represents the distance between the i-th position and the j-th position in time, β is a sensitivity factor used to balance the local distance; f v (X I3D ) is the value vector V after linear projection; M ij is the output feature of the masking operation; softmax[T m ·f v (X I3D )] indicates the timing Mask operation; is the local context feature; M ij The expression is: Where T represents the size of the mask window; j and i range from 0 to T-1; w is a hyperparameter; The normal video dataset and the abnormal video dataset are respectively subjected to the I3D feature extraction network to obtain I3D features, and then enter the global and local context modules as well as the adaptive fusion module and the multi-layer perceptron MLP to generate normal / abnormal context video features; In the cross-modal interaction framework, the video dataset and the corresponding text dictionary are trained from the CLIP model to generate video features and text features respectively, and enter the cross-modal interaction module to calculate the weights between text and frame features and generate text-guided video features. The expression of the cross-modal interaction module is formula (7) and (8): Among them, α k represents the text conditional weight of the kth frame in the video feature v', f' k represents the temporal features of the kth frame in the video feature v'; τ represents a scaling factor; is the text feature; F t It is the text-guided video feature; In the feature fusion and multi-instance learning framework, normal context video features and normal text-guided video features, abnormal context video features and abnormal text-guided video features are fused together for multi-instance learning loss ranking to obtain weakly supervised video anomaly detection results.

2. The method according to claim 1, characterized in that The specific process in the global-local context framework is as follows: the output of the I3D feature extraction network enters the global context module and the local context module respectively. The outputs of the global context module and the local context module are processed by the adaptive fusion module and then residually connected with the output of the I3D feature extraction network through normalization. Finally, the context video features are obtained after processing by the multi-layer perceptron MLP and layer normalization LN.

3. The method according to claim 1, characterized in that The expression of the adaptive fusion module is: Among them, ,λ and 1-λ represent the global context feature A and the local context feature respectively. is the fusion weight of the context, and X is the global and local features output by the adaptive fusion module.

4. The method according to claim 1, characterized in that: In the feature enhancement fusion and multi-instance learning framework, a feature fusion enhancement module is used to transform the normal context video features into and normal text guided video features Perform feature fusion, similarly, abnormal context video features and abnormal text-guided video features Perform feature fusion to obtain the normal fusion feature F a and abnormal fusion feature F n , the formula is: Among them, ψ(·) is the feature fusion enhancement module, which represents the feature alignment and fusion operation; Construct a normal / abnormal behavior classification model, input the normal fusion features and abnormal fusion features into the multi-layer perceptron and causal convolution layer respectively to predict the abnormal score S of the video clip a and S n , the formula of the normal / abnormal behavior classification model is: Among them, f t (·) is a causal convolutional layer with a kernel size of Δt, σ(·) is a sigmoid activation function, is the anomaly score of the i-th video clip in the anomaly fusion feature; is the abnormality score of the i-th video clip in the normal fusion feature; MLP is a multi-layer perceptron; The multi-instance learning loss function uses binary cross entropy loss to obtain the binary cross entropy between the frequency-level prediction and the true label to calculate the classification loss, and determines whether the video contains anomalies based on the predicted anomaly score.

Images