A weakly supervised video anomaly detection method

Abstract

The application relates to a weakly supervised video anomaly detection method, comprising the following steps: a visual-text information competitive fusion module is constructed, complementary information of the two is aggregated, a competitive mechanism is adopted to reduce information redundancy of the two, high-level semantics of a video are better captured, and discriminative semantic information is obtained; then, a local-global time dependence module is further established to model short-term and long-term time dependence between video clips and adjust information weight in the clips; finally, two loss functions are designed in a loss function, multiple abnormal clips possibly existing in an abnormal video are maximally utilized for effective training; and the method improves accuracy of video anomaly detection and enhances robustness of video anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of video anomaly detection technology, and in particular to a weakly supervised video anomaly detection method. Background Technology

[0002] Video anomaly detection (VAD) aims to detect anomalous events in undressed video. This task not only meets the growing security needs of daily life but also significantly reduces human labor, providing crucial support for building smart cities. Due to the sparsity and unpredictability of anomalous events, manual screening is inefficient, time-consuming, and labor-intensive, and may result in missed detections due to human visual fatigue, making it difficult to handle large-scale video streams. Therefore, employing deep learning algorithms to automatically detect anomalous events in videos is of great significance.

[0003] Due to the sparsity and diversity of anomalous events, previous studies have mostly relied on semi-supervised settings, employing a one-class classification paradigm and training models using normal video data. This approach assumes that the model can learn normal patterns from normal videos and classifies patterns deviating from these normal patterns as anomalous. However, collecting all possible normal events for modeling is impractical. Therefore, this method of anomaly identification is overly idealistic, ignoring the diversity of normal events. Furthermore, the learned model struggles to find a reasonable balance between representational and generalization capabilities: insufficient representational power may lead to misidentification of unseen normal events as anomalous; conversely, excessive generalization may result in misclassification of anomalous events as normal. Consequently, in complex scenarios, the detection performance of such methods is often unsatisfactory. Weakly supervised video anomaly detection (WS-VAD), by introducing video-level labels and anomalous videos during the training phase, employs a multiple instance learning (MIL) mechanism and ranking loss to detect anomalous events. This approach enables the model to extract features from both normal and anomalous events simultaneously. Therefore, this method is more effective at detecting anomalous events in videos compared to semi-supervised methods.

[0004] Weakly supervised methods provide video-level labels but not the temporal location of anomalous instances. Most current MIL-based video anomaly detection methods treat videos as packets, and individual video segments as instances within those packets. Anomalous videos are considered positive packets, while normal videos are considered negative packets. These methods typically assume that the instance with the highest anomalous score in the positive packet is most likely anomalous, and increase the anomalous score difference between the instance with the highest anomalous score in the positive packet and the normal instance with the highest anomalous score in the negative packet, thus allowing the most likely anomalous instance in the positive packet to receive a higher anomalous score.

[0005] Although this approach incorporates anomalous event information during training, existing MIL-based weakly supervised methods still face several challenges. First, most existing methods utilize spatiotemporal visual features, such as 3D ConvNet (C3D) or Inflated3DConvNet (I3D), to extract visual features from videos and then leverage these features for anomaly detection. However, in complex scenes, anomalous events may not be visually obvious. For example, in the case of an explosion, if the flames occupy only a small portion of the frame or are obscured by smoke, the visual information in the scene may not adequately express the complete semantics of the event. Relying on a single visual feature often fails to capture all the high-level semantic information in the video, making the detection of certain anomalous events difficult. Furthermore, textual features emphasize capturing abstract concepts that are difficult for visual features to directly capture, providing a new perspective for understanding video events. To overcome the limitations of visual features, introducing textual information, which is better suited to expressing the semantics of events, is a crucial step in improving model performance. Chen et al., drawing inspiration from video captioning models, extracted textual features containing rich semantic information to supplement visual features, thereby improving the accuracy and robustness of weakly supervised video anomaly detection tasks. Although textual and visual features provide complementary information, there is still some information redundancy between them. When fusing visual and textual features, Chen et al. used only simple methods such as concatenation, addition, and multiplication. This simple feature cascading method cannot effectively model the complementary information and correlation between the two modalities. While textual and visual features are complementary, there is still some information redundancy. Therefore, effectively combining them will help the model maintain robust performance in different contexts. Secondly, videos consist of continuous segments with natural temporal dependencies. However, most of the aforementioned weakly supervised methods locate anomalies based on feature differences within a single segment, and their exploration of temporal dependencies between segments, especially long-term and short-term temporal dependencies, remains very limited. These temporal dependencies between segments are crucial for anomaly detection. Furthermore, in the loss function, the MIL-based ranking loss aims to maximize the difference in anomaly scores between the segment with the highest anomaly score in the abnormal and normal videos to achieve better detection performance. However, abnormal events often consist of multiple consecutive abnormal segments; simply selecting the instance with the highest anomaly score as the optimization target of the ranking loss may lead to missed anomaly segments in the abnormal video. Furthermore, abnormal videos often contain a lot of noise from normal segments, which can also interfere with model training and optimization. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a weakly supervised video anomaly detection method. This method improves the accuracy and robustness of video anomaly detection by three aspects: competitive fusion of visual-text features, mining the temporal dependence between segments, and increasing the distinguishability between abnormal and normal segments.

[0007] The technical solution adopted in this invention is a weakly supervised video anomaly detection method, which includes the following steps:

[0008] Step 1: Construct a temporally enhanced visual-text competition fusion network model; the temporally enhanced visual-text competition fusion network model includes, in sequence, parallel visual and text branches, a local-global temporal dependency module, a visual-text information competition fusion module, and a MIL classifier;

[0009] Step 2: Train the temporally enhanced visual-text competition fusion network model to obtain the trained temporally enhanced visual-text competition fusion network model;

[0010] Step 3: Input the video to be detected into the trained temporally enhanced visual-text competition fusion network model to perform video anomaly detection;

[0011] In step two, the specific process of training the temporally enhanced visual-text competition fusion network model includes:

[0012] S1. Given an input video V, the video-level label of the input video V is Y, where Y∈{0,1}; divide the input video V into T segments, i.e. Each video segment is generated by v i express;

[0013] S2. For each video segment, the video segment is input into both the visual branch and the text branch. The visual branch is used to extract the visual features F of the video segment. Vis , And extract the visual features F Vis Output; the text branch is used to extract the text features F corresponding to the video clip. Text , And the extracted text features F Text Output;

[0014] S3, the visual features F output by the visual branch. Vis and the text features F output by the text branch Text The inputs are processed by the local-global temporal dependency module, which then outputs the local-global temporal dependency visual features F.V and local-global time-dependent text features F T ,in,

[0015] S4, the F V and F T Simultaneously, the visual-text information is input into the competitive fusion module for fusion, resulting in the fused feature F. V&T ,in,

[0016] S5. The fusion feature F V&T The input is used to perform segment-level anomaly score prediction in the MIL classifier, resulting in a set of segment anomaly scores S = {S1, S2, ..., S...}. T}, where S i This represents the anomaly score of the i-th video segment;

[0017] S6. Calculate the training loss of the MIL classifier using the loss function, and train the MIL classifier through gradient backpropagation to obtain the trained MIL classifier.

[0018] S7. Using the next input video, return to step S1 and repeat until the trained temporally enhanced visual-text competition fusion network model is obtained.

[0019] The beneficial effects of this invention are as follows: The aforementioned weakly supervised video anomaly detection method aggregates complementary visual and textual information through a Visual-Text Information Competition Fusion (VTICF) module and employs a competition mechanism to reduce information redundancy, thereby better capturing the high-level semantics of the video and obtaining discriminative semantic information. Furthermore, a Local-Global Temporal Dependency (LGDT) module is established to model the short-term and long-term temporal dependencies between video segments and adjust the information weights within segments. Finally, a loss function is used to train the MIL classifier. This method can improve the accuracy and robustness of video anomaly detection.

[0020] Preferably, in step S3, the local-global temporal dependency module sequentially includes a multi-scale temporal feature fusion module, a multi-head self-attention module, and an SENet module; the visual features F output by the visual branch... Vis The specific process of inputting into the local-global time-dependent module for processing includes the following steps:

[0021] S3.1 In the multi-scale temporal feature fusion module, a one-dimensional convolution with a kernel size of s is used to apply the visual feature F Vis The convolution operation is represented as: F Vis (s)=Conv1D (s) (F Vis ); where s∈{3,7},Conv1D (s) (·) denotes a one-dimensional convolution operation with a kernel size of s, F Vis (s) Representing visual feature F Vis The output features after applying one-dimensional convolution in the time dimension Visual features F are extracted through convolution operations. Vis (3) and visual features F Vis (7) ;

[0022] S3.2, Visual features F Vis (3) Visual features F Vis (7) and original visual features F Vis The fusion is performed to obtain multi-scale temporal fusion features F. Vis_L ,

[0023] S3.3, The multi-scale temporal fusion feature F Vis_L In the multi-head self-attention module, F is first processed through three fully connected layers. Vis_L Convert this into a query q, key k, and value v; where, as well as Then q, k, and v are respectively divided into q c k c and v c ;in, as well as Next, the multi-head self-attention module calculates the self-attention O for each head separately. c Its expression is: O c =SA(q) c ,k c ,v c The self-attention results from each head are then merged to obtain the final output feature F. Vis_LG :F Vis_LG =MHSA(F Vis_L = Concat(O1,...,O) H );in,

[0024] S3.4, the output feature F Vis_LG The fragment feature weights are adjusted in the SENet module; specifically, the first step is a Squeeze operation, which adjusts the weights of the output feature F. Vis_LGPerform global average pooling, where each segment feature is represented by a constant. express, Then comes the excitation operation, which uses a linear mapping to first reduce the dimensionality and then increase it, learning the importance of fragment features, i.e., obtaining the weights w of the fragment features, w∈R. T Finally, there is the scaling operation, which sets the attention weights w and F. Vis_LG Multiplication yields the final feature output F. V ,

[0025] Preferably, in step S4, the F V and F T The specific process of fusion in the simultaneous input visual-text information competition fusion module includes the following steps:

[0026] S4.1, the F V and F T The text information is fed into the first cross-attention mechanism, which embeds the text information into the visual features. The specific process is as follows:

[0027] S4.01, the F V and F T Mapped to query Q respectively v and key K t Sum V t , is represented as: in, and Represents the learnable linear projection parameters;

[0028] S4.11 Calculate the cross-attention scores of text features and visual features, and apply residual connections to obtain the visual features F that fuse text semantics. V_T , is represented as: in, d vt Q represents v and K t The feature dimension;

[0029] S4.2, Visual features F V_T In the second cross-attention mechanism, the visual features F are again processed by the second cross-attention mechanism. V_T The specific process of embedding into text features is as follows:

[0030] S4.02, F T and F V_T The input is a second cross-attention mechanism, and through a similar mapping process, F... T and FV_T Mapped to query Q respectively t and key K vt Sum V vt It is represented as: in, and Represents the learnable linear projection parameters;

[0031] S4.12, Calculate F T and F V_T The cross-attention scores are calculated and residual connections are applied to obtain the text features F that fuse visual information. T_V It is represented as: Where, d tv Q represents t and K vt Feature dimension,

[0032] S4.3. A cross-modal feature dynamic competitive activation mechanism is adopted to dynamically weigh the relative importance of visual and textual information, and to form the final visual feature F that integrates textual semantics by aggregating information from different features. V_T And finally, the text features F that fuse visual information. T_V The final visual feature F of the fused text semantics V_T Represented as:

[0033] The final text feature F of the fused visual information T_V Represented as:

[0034] Where the number of channels in the feature is T, and the feature dimension is d. vt ,

[0035] Visual-text information complementary enhancement features Represented as: Where: (f i vt ) d =φ d ,(f i tv ) d =θ d max(,) represents the maximum activation operation.

[0036] Preferably, in step S6, the loss function is expressed as: Where μ and σ both represent weights; BCE(S,Y) represents the binary cross-entropy loss between the anomaly score set S and the label set Y; L outer Represents the Top-K out-of-packet loss function. K max (S) represents the anomaly scores s in the set S of anomaly scores for a video segment. i The set of the largest K elements and These represent the anomalous scores of segments in abnormal videos and the anomalous scores of segments in normal videos, respectively; L inner L represents the k-maxmin in-packet loss function. inner =L pos +L neg L pos This represents the k-maxmin positive bag in-loss function. K min (S) The set of outlier scores S of the video clip contains outlier scores s. i The smallest set of K elements, L neg This represents the k-maxmin negative in-pack loss function. L smooth Represents the smoothing loss function.

[0037] Preferably, the specific process of inputting the video to be detected into the trained temporally enhanced visual-text competitive fusion network model for video anomaly detection is as follows: the video to be detected is input into the trained temporally enhanced visual-text competitive fusion network model, the trained MIL classifier predicts the anomaly scores of all video segments in the video to be detected, a threshold is set, and the type of each video segment is classified according to the threshold, that is, video segments with anomaly scores greater than the threshold are abnormal segments, and video segments with anomaly scores less than the threshold are normal segments. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the framework of the temporally enhanced visual-text competition fusion network model in this invention;

[0039] Figure 2 This is a schematic diagram of the framework of the local-global time-dependent module in this invention;

[0040] Figure 3 This is a schematic diagram of the framework of the visual-text information competition fusion module in this invention;

[0041] Figure 4 This is a schematic diagram of the visualization results of the method of the present invention on the UCF-Crime dataset. Detailed Implementation

[0042] The invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can implement it based on the description. The scope of protection of the invention is not limited to these specific embodiments.

[0043] This invention relates to a weakly supervised video anomaly detection method, which includes the following steps:

[0044] Step 1: Construct as follows Figure 1 The Temporally Enhanced Visual-Text Competition Fusion Network Model (TE-VTCF) shown is described. The Temporally Enhanced Visual-Text Competition Fusion Network Model includes, in sequence, parallel visual and text branches, a local-global temporal dependency module, a visual-text information competition fusion module, and a MIL classifier.

[0045] Step 2: Train the temporally enhanced visual-text competition fusion network model to obtain the trained temporally enhanced visual-text competition fusion network model;

[0046] Step 3: Input the video to be detected into the trained temporally enhanced visual-text competition fusion network model to perform video anomaly detection;

[0047] In step two, the specific process of training the temporally enhanced visual-text competition fusion network model includes:

[0048] S1. Given an input video V, the video-level label of the input video V is Y, where Y∈{0,1}; divide the input video V into T segments, i.e. Each video segment is generated by v i express;

[0049] S2. For each video segment, the video segment is input into both the visual branch and the text branch. The visual branch is used to extract the visual features F of the video segment. Vis , And extract the visual features F Vis Output; the text branch is used to extract the text features F corresponding to the video clip. Text , And the extracted text features F Text Output;

[0050] S3, the visual features F output by the visual branch. Vis and the text features F output by the text branch Text The inputs are processed by the local-global temporal dependency module, which then outputs the local-global temporal dependency visual features F. V and local-global time-dependent text features FT ,in,

[0051] S4, the F V and F T Simultaneously, the visual-text information is input into the competitive fusion module for fusion, resulting in the fused feature F. V&T ,in,

[0052] S5. The fusion feature F V&T The input is used to perform segment-level anomaly score prediction in the MIL classifier, resulting in a set of segment anomaly scores S = {S1, S2, ..., S...}. T}, where S i This represents the anomaly score of the i-th video segment;

[0053] S6. Calculate the training loss of the MIL classifier using the loss function, and train the MIL classifier through gradient backpropagation to obtain the trained MIL classifier.

[0054] S7. Using the next input video, return to step S1 and repeat until the trained temporally enhanced visual-text competition fusion network model is obtained.

[0055] Inspired by the significant achievements of attention mechanisms in computer vision, this invention proposes a Local-Global Temporal Dependency Module (LGTD) to model short-term and long-term temporal dependencies between video segments and adjust the information weights within each segment. The LGTD module comprises a multi-scale temporal feature fusion module, a multi-head self-attention module, and an SENet module, with the network structure as follows: Figure 2 As shown. First, considering that anomalous events are often composed of multiple consecutive anomalous segments, obtaining the local temporal dependencies between adjacent segments is very helpful in improving anomaly detection performance. Second, there may be multiple anomalous events in an anomalous video, and these anomalous events may be correlated. Therefore, obtaining the global temporal dependencies between video segments is also beneficial in enhancing the discriminative power of video segment anomalies.

[0056] Given that anomalous events in videos often occur within multiple consecutive segments, this invention proposes a multi-scale temporal feature fusion module. This module uses multiple one-dimensional convolutional kernels of varying sizes to act on the temporal dimension of video features to obtain local temporal dependencies between instances at different scales. Since both visual and textual features are processed through this module, this invention uses the visual features of the input video V as the basis for its design. Using the input as an example, we will illustrate the overall architecture of this module. This invention uses a one-dimensional convolution with a kernel size of s to process video features F. VisPerform a convolution operation, where s∈{3,7}. The one-dimensional convolution operation is defined as follows:

[0057] F Vis (s) =Conv1D (s) (F Vis (1)

[0058] Among them, Conv1D (s) (·) represents a one-dimensional convolution operation with a kernel size of s. F Vis (s) Representing video features F Vis The output features are obtained by applying one-dimensional convolution in the time dimension; finally, the present invention extracts the features F from the one-dimensional convolution. Vis (3) and F Vis (7) With original feature F Vis Fusion yields multi-scale temporal fusion features. The fused features possess both the spatiotemporal characteristics of the original video segments and rich temporal local correlations between segments. This invention believes that this is beneficial for fully utilizing the information from different receptive fields to obtain the local temporal correlations between video segments.

[0059] An abnormal video may contain more than one abnormal event, and these events may have certain logical connections. Furthermore, not only are there connections between several consecutive abnormal segments that constitute an abnormal event, but they may also be reasonably related to segments of normal events. Therefore, capturing the global temporal dependencies between segments plays an important role in improving the discriminative power between normal and abnormal segments. Therefore, this invention employs a multi-head self-attention module to generate global attention to calculate the global temporal dependencies between segments in the video. For example, for an input S, whose query, key, and value are Q, K, and V respectively, the self-attention is defined as follows:

[0060]

[0061] in

[0062] Q=S·Wq, K=S·Wk, V=S·Wv (3)

[0063] Wq, Wk, and Wv represent learnable linear projection parameters, softmax(·) is the softmax function, and dk represents the feature dimensions of Q and K.

[0064] F-scale features of the video Vis_L As input to the multi-head self-attention module, it is transformed into a query q, a key k, and a value v using three fully connected (FC) layers to reduce feature dimensionality and computational cost. and Then, the feature fragments are divided into H segments (number of heads). Afterwards, q, k, and v are divided into q segments. c k c and v c , and c∈{1,.,H}. The multi-head self-attention module calculates the self-attention O for each head separately. c The results are then combined to obtain the final output.

[0065] O c =SA(q) c ,k c ,v c (4)

[0066] F Vis_LG =MHSA(F Vis_L = Concat(O1,...,O) H ) (5)

[0068] To further model the global correlation between segment features and allow the model to focus more on information within segments, this invention treats each segment in the video as a corresponding channel. Finally, this invention uses feature F... Vis_LG The fragment feature weights are fed into the SENet module to adjust the segment feature weights. The SENet module mainly includes three operations: compression, expansion, and weight transformation. The first is the Squeeze operation, which adjusts the weights of the F segments. Vis_LG Perform global average pooling, where each segment feature is represented by a constant. Indicated, i∈[1,T]. Next is the Excitation operation, which uses a linear mapping to first reduce dimensionality and then increase it, learning the importance of fragment features, i.e., obtaining the weights w∈R of the fragment features. T Finally, there's the scaling operation, which sets the attention weights w and F... Vis_LG Multiplication yields the final feature output.

[0069] In complex scenes, especially those with complex visual information or obscured visual cues, relying solely on the visual modality may cause the model to overlook visually inconspicuous anomalies. These subtle differences are often key to understanding deep semantic information in videos. To overcome the limitations of a single modality in acquiring video-level semantic information, this invention proposes a Visual-Text Information Competitive Fusion (VTICF) module. This module not only captures coherent dynamic information using visual features but also understands more abstract concepts and details through textual features. It extracts high-level semantic information from videos through cross-attention and cross-modal feature competitive activation mechanisms. This not only reduces false positives and false negatives caused by scene complexity but also enhances the model's ability to detect anomalies while maintaining high accuracy.

[0070] To make full use of visual features Text features The complementary nature of the information between the two features is addressed through two cross-attention mechanisms to promote deep information fusion. First, textual information is embedded into visual features. This allows the model to guide visual features to focus on visual elements closely related to the text description, guided by the context of the text content. Second, the text-enriched visual features are re-embedded into the textual features, enabling the model to enhance its understanding of textual features through visual cues and strengthening its ability to interpret textual features.

[0071] To embed textual information into visual features, F V and F T It is fed into the first cross-attention mechanism. First, it is fed into three different fully connected layers, where F... V and F T Mapped to query Q respectively v and key K t Sum V t , is represented as:

[0072]

[0073] in, and The learnable linear projection parameters are represented. Next, the cross-attention scores of textual and visual features are calculated, and residual connections are applied to obtain visual features that fuse textual semantics. The calculation formula is as follows:

[0074]

[0075] Where, d vtQ represents v and K t The feature dimension.

[0076] In obtaining visual-text fusion features F V_T Next, a second cross-attention mechanism is used to re-embed the text features, which are then enriched with textual information, into the text features to optimize the text representation. The specific steps are as follows: F... T and F V_T Input cross-attention mechanism, and through a similar mapping process, F T and F V_T Mapped to query Q respectively t and key K vt Sum V vt , is represented as:

[0077]

[0078] in, and The text features are derived by first representing learnable linear projection parameters, and then obtaining text features that incorporate visual information by calculating cross-attention scores and residual connections. The calculation formula is as follows:

[0079]

[0080] Where, d tv Q represents t and K vt The feature dimension.

[0081] While combining visual and textual features to utilize their complementary information, this invention also addresses the potential information redundancy between these two types of features. Although existing technologies, such as those by Chen et al., have attempted to merge these two features through simple concatenation, multiplication, or addition, utilizing their complementary information to some extent, this may introduce unnecessary information redundancy, thus interfering with the model's training process. To address this challenge, inspired by research on cross-domain face recognition, which improves recognition performance by activating the differences in feature weights across different domains, this invention proposes a cross-modal feature competitive activation mechanism. This mechanism introduces a competition between visual and textual features, weighing the relative importance of visual and textual information to aggregate video representation information from different features. This strengthens the model's selectivity in extracting key features, thereby effectively reducing information redundancy. (Video feature fusion) The number of channels is T, and the feature dimension is Similarly, The implementation steps of this module are similar to those of the Maxout operation, using visual-textual information complementary enhancement features. The calculation method is as follows:

[0082]

[0083]

[0084] By activating the max operation in formula (11), aggregate F V_T and F T_V It extracts key information from mid-length video content and effectively enhances the compactness of feature representation.

[0085] This invention treats video anomaly detection as a regression task, aiming for the model to predict higher anomaly scores for anomalous instances and lower anomaly scores for normal instances. The ranking loss based on the MIL classifier aims to maximize the difference in anomaly scores between the instances with the highest anomaly scores in the positive and negative packets to achieve better detection performance. However, anomalous events often consist of multiple anomalous segments. Simply selecting the instance with the highest anomaly score as the optimization unit for the ranking loss may lead to missed anomaly instances and incorrect normal instances in anomaly video detection. Therefore, this invention proposes a Top-K out-of-packet loss function, selecting the top k instances with the highest anomaly scores in both the positive and negative packets, and using the average of these anomaly scores as the optimization unit. This approach has a higher probability of selecting anomalous instances from the positive packet, and the model can maximize the use of the anomaly instances present in the positive packet for training. The Top-K out-of-packet loss is defined as follows:

[0086]

[0087] Among them, K max (S) represents the set of exception scores S containing the exception scores of instances in a package, where s is the exception score. i The set of the largest K elements and These represent the abnormal scores of segments in abnormal and normal videos, respectively.

[0088] For the positive packet, the presence of numerous normal instances within it creates noise, negatively impacting model detection performance. Therefore, this invention proposes the k-maxmin positive packet in-packet loss, which widens the gap in outlier scores among instances within the positive packet to reduce noise interference from normal instances and improve model detection performance. The k-maxmin positive packet in-packet loss is defined as follows:

[0089]

[0090] Among them, K min(S) Returns the set of exception scores S for an instance, containing the exception scores s. i The smallest set of K elements.

[0091] For negative packets, there may be difficult instance samples with high outlier scores, affecting the model's convergence speed and detection performance. This invention aims to minimize the difference in outlier scores among all instances in the negative packet. Therefore, this invention proposes a k-maxmin negative packet intra-packet loss, which minimizes the difference between the average outlier scores of the k highest and k lowest outlier instances in the negative packet. The k-maxmin negative packet intra-packet loss of this invention is as follows:

[0092]

[0093] Then, this invention combines the k-maxmin positive bag loss and the negative bag loss to form the k-maxmin bag loss, defined as follows:

[0094] L inner =L pos +L neg (14)

[0095] Finally, referring to the smoothing constraints on instance anomaly scores in existing technologies, this invention adds a smoothing loss function to the anomaly scores of positive and negative packet instances to enhance the temporal smoothness between the anomaly scores of temporally adjacent instances. The smoothing loss of this invention is defined as follows:

[0096]

[0097] The MIL loss function is defined as follows:

[0098] L mil =L outer +L inner +L smooth (16)

[0099] Finally, this invention incorporates binary cross-entropy loss as the classification loss function, and the total loss function is defined as follows:

[0100] L final =μ·L mil +σ·BCE(S,Y) (17)

[0101] Here, BCE is the binary cross-entropy loss between the anomaly score set S and the label set Y.

[0102] In step three, the specific process of inputting the video to be detected into the trained temporally enhanced visual-text competitive fusion network model for video anomaly detection is as follows: the video to be detected is input into the trained temporally enhanced visual-text competitive fusion network model, the trained MIL classifier predicts the anomaly scores of all video segments in the video to be detected, a threshold is set, and the type of each video segment is divided according to the threshold, that is, video segments with anomaly scores greater than the threshold are abnormal segments, and video segments with anomaly scores less than the threshold are normal segments.

[0103] This invention proposes a temporally enhanced visual-text competitive fusion (TE-VTCF) model framework, which enhances the discriminativeness and robustness of anomaly detectors from three aspects: competitive fusion of visual and textual features, mining temporal dependencies between segments, and increasing the discriminative power between anomalous and normal segments. Specifically, a visual-text information competitive fusion module (VTICF) is constructed, which aggregates complementary information from both and employs a competitive mechanism to reduce information redundancy, thereby better capturing high-level semantics of the video and obtaining discriminative semantic information. Furthermore, a local-global temporal dependency module (LGDT) is established to model short-term and long-term temporal dependencies between video segments and adjust the information weights within segments. Finally, two loss functions are designed in the loss function. The Top-K out-of-package loss selects the k instances with the highest anomalous scores from both anomalous and normal videos and widens the gap between these instances' anomalous scores, thereby maximizing the utilization of multiple anomalous segments that may exist in the anomalous video for effective training. Meanwhile, the K-maxmin intra-bag loss utilizes the latent temporal structure of abnormal and normal events, and further reduces noise interference from normal segments in abnormal videos by adjusting the distribution of abnormal scores in segments within abnormal and normal videos, and maximizes the separation of normal and abnormal segments within the video.

[0104] This invention's method fuses visual and textual features from videos through a cross-attention mechanism and a cross-modal feature competitive activation mechanism, aggregating complementary information while reducing information redundancy. Simultaneously, a local-global temporal dependency module captures the temporal dependencies between segment features, further mining high-level semantic information in the video so that the model can clearly separate normal and abnormal segments. Building upon existing ranking losses, this invention introduces Top-K out-of-bag loss and K-maxmin in-bag loss. Inspired by the latent temporal structure of anomalous and normal events in videos, these loss functions aim to maximize the use of multiple potential anomalous segments in anomalous videos for effective training, while reducing noise interference from normal segments. This invention's method outperforms state-of-the-art methods on two benchmark datasets and demonstrates better robustness.

[0105] The following experiments demonstrate the superiority of the weakly supervised video anomaly detection method of the present invention.

[0106] This invention uses two large-scale anomaly detection datasets for experiments and presents the experimental results, namely the UCF-Crime dataset and the XD-Violence dataset.

[0107] UCF-Crime: This dataset consists of diverse real-world scenarios, totaling 128 hours in length. It includes 1900 untrimmed videos from real-world street and doorway surveillance cameras, covering 13 common anomalous events in daily life. Both the training and test sets contain normal and anomalous videos. The training set contains 1610 training videos, all with video-level labels, while the test set contains 290 test videos with frame-level labels.

[0108] XD-Violence: This dataset is a large-scale dataset with a total duration of 217 hours, containing 4754 untrimmed videos collected from multiple sources. Both the training and test sets contain normal and anomalous videos. The training set contains 3954 training videos, all with video-level labels, while the test set contains 800 test videos with frame-level labels.

[0109] Evaluation metrics: Area UnderofCurve (AUC) of the frame-level Receiver Operating Characteristics (ROC) was used as the evaluation metric for the TE-VTCF model on the UCF-Crime dataset. Mean Accuracy (AP) was used as the evaluation metric for the TE-VTCF model on the XD-Violence dataset. In video anomaly detection tasks, higher AUC and AP values ​​indicate better performance of the TE-VTCF model.

[0110] Using the method of this invention:

[0111] 1) Feature extraction:

[0112] Each video is divided into 32 non-overlapping video segments, i.e., T=32. For the UCF-Crime and XD-Violence datasets, we use a pre-trained I3D network to extract visual features from these two datasets, with each segment feature having a dimension D. vis All are 1024. Furthermore, the captions for the videos in the dataset were generated using pre-trained SwinBERT, and the generated captions were converted into sentence embedding vectors using the Semantic Similarity Self-Supervised Contrastive Learning (SimCSE) framework as text features. The dimension D of the text features for each segment is... text It is 768.

[0113] 2) Network structure:

[0114] The one-dimensional convolutional kernels of the multi-scale feature fusion module are set to sizes of 3 and 7; the number of heads in the multi-head self-attention module is set to 4, i.e., H = 4, d vis 'Set to 64.' Finally, the temporal enhanced visual features are obtained. and temporal enhanced text features fragment feature dimension d v and d t All values ​​were set to 64. For all fully connected layers, 60% dropout was used to prevent overfitting. Finally, based on experience, the loss function's k-value was set to 3, and the hyperparameter λ1 was set to 8e-5. On the UCF-Crime dataset, μ = 0.7 and σ = 0.3 were set. On the XD-Violence dataset, μ = 0.01 and σ = 1 were set.

[0115] 3) Training details:

[0116] The TE-VTCF model was implemented using PyTorch on a single RTX 3090 GPU. During training, the Adam optimizer was used, with a batch size of 32, a learning rate of 0.001, a weight decay factor of 0.005, and a training duration of 100 epochs.

[0117] To fairly evaluate the performance of our model, the method of this invention was compared with several existing weakly supervised learning methods. Specific experimental results are shown in Tables 1 and 2. The selected existing methods include: MIL, TCN-IBL, Motion-Aware, GCN-AD, NVAD, MIST, RTFM, MSL, BN-SVP, and TEVAD.

[0118] Comparative Experiments on the UCF-Crime Dataset: Table 1 shows the comparison results of the proposed method and other state-of-the-art weakly supervised anomaly detection methods on the UCF-Crime test set, with the proposed method achieving an AUC score of 86.01%. Compared with MIL, TCN-IBL, Motion-Aware, GCN-AD, NVAD, and MIST methods, the proposed method shows a significant improvement in detection performance. Although the TCN-IBL method designs a ranking loss to distinguish between anomalous and normal segments, it fails to capture the different relationships between different segments. Furthermore, although the Motion-Aware method uses PWC-Net to extract optical flow as local motion cues and enhances the MIL ranking model by incorporating bottom-up attention modules, the TE-VTCF model of the proposed method still outperforms it by 6.57%. This indicates that using only local motion representations is insufficient to model the temporal structure in videos. Even compared with the state-of-the-art TEVAD, our results lead by 1.11%.

[0119] Table 1 compares the frame-level AUC (%) performance on the UCF-Crimes dataset. * This indicates that the method is retrained using I3D features.

[0120]

[0121]

[0122] Comparative experiments on the XD-Violence dataset: Table 2 shows the comparison results of the proposed method and other state-of-the-art weakly supervised anomaly detection methods on the XD-Violence test set. The proposed method achieves an AP score of 81.62%. Compared with MIL, NVAD, and RTFM methods, our method shows a significant improvement in detection performance. Even compared with the state-of-the-art TEVAD, the proposed method outperforms it by 1.82%.

[0123] Table 2 compares the performance of frame-level AP (%) on the XD-Violence dataset.

[0124] MIL C3D-RGB 73.20 NVAD I3D-RGB 75.41 RTFM C3D-RGB 75.89 RTFM I3D-RGB 77.81 MSL VideoSwin-RGB 78.59 TEVAD I3D-RGB+SwinBERT 79.80 This invention I3D-RGB+SwinBERT 81.62

[0125] In summary, the method proposed in this invention consistently outperforms state-of-the-art (SOTA) methods on two benchmark datasets in the field of video anomaly detection.

[0126] This invention conducted ablation studies on network modules across different datasets to demonstrate the effectiveness of the main components of the TE-VTCF model in this invention: LGTD, VTICF, and the introduction of textual features. The results are shown as percentages in Table 3. Table 3 presents the AUC results for the UCF-Crime dataset and the AP results for the XD-Violence dataset. As can be seen from the table, adding LGTD to process visual features resulted in performance improvements across all datasets. This indicates that extracting temporal dependencies from visual features helps improve model detection performance. Furthermore, adding textual features resulted in consistent performance improvements across all datasets. Additionally, using LGTD to process textual features further improves performance. This demonstrates that introducing textual features and extracting temporal dependencies from textual features contributes to improved model detection performance. To evaluate the impact of VTICF, we compared it with the concat method. The results show consistent performance improvements across all datasets, indicating that VTICF better combines the high-level semantic information of visual and textual features. In summary, compared with using the original visual features alone, the TE-VTCF model improves performance on UCF-Crime and XD-Violence by 5.1% and 8.16%, respectively.

[0127] Table 3 shows the ablation study results of the network modules. Vanilla represents the original video features obtained using the feature extractor, LGTD represents the temporal dependency between segment features enhanced using the LGTD module, and concat and vticf represent the fusion methods for visual and textual features, respectively.

[0128]

[0129] This invention conducts ablation experiments on different datasets using the loss function employed in the TE-VTCF method to demonstrate its effectiveness. The results are shown as percentages in Table 4. To evaluate the effectiveness of the loss function, we compare it with the ranking loss function L... rank A comparison was made. As can be seen from the table, using L alone... outer and L inner The model showed improved detection performance across all datasets. Finally, by comprehensively using L... outer and L inner This can further improve performance. In summary, compared with using ranking loss alone, the TE-VTCF model improves performance by 3.44% and 4.75% on UCF-Crime and XD-Violence, respectively.

[0130] Table 4.4 Results of ablation study on loss function

[0131]

[0132] Finally, the anomaly detection performance before and after introducing the main components of the TE-VTCF model was visualized. Figure 4 The two rows of images in the image show a comparison of anomaly detection performance in two different videos after incorporating different components of the TE-VTCF model. Specifically, the first column displays the anomaly detection results using visual features and the MIL method. Figure 4 (a) Figure 4 As shown in (d), because anomaly detection relies solely on raw visual features, the model's predicted anomaly scores exhibit significant volatility. Furthermore, false positives and false negatives are quite frequent. The second column presents the detection results after introducing a Local-Global Temporal Dependency Module (LGTD) into the MIL method. Figure 4 (b) Figure 4 As shown in (e), the model's volatility over time was suppressed; however, false negatives and false negatives still exist. The last column demonstrates the significant improvement in detection results achieved by incorporating textual features and implementing competitive fusion of visual and textual features in the TE-VTCF model, as shown in... Figure 4 (c) and Figure 4 As shown in (f). This is because after integrating visual and textual features, the model has a deeper understanding of the high-level semantic information of the video content, which significantly improves the accuracy of abnormal region localization.

Claims

1. A weakly supervised video anomaly detection method, characterized in that: The method includes the following steps: Step 1: Construct a temporally enhanced visual-text competition fusion network model; the temporally enhanced visual-text competition fusion network model includes, in sequence, parallel visual and text branches, a local-global temporal dependency module, a visual-text information competition fusion module, and a MIL classifier; Step 2: Train the temporally enhanced visual-text competition fusion network model to obtain the trained temporally enhanced visual-text competition fusion network model; Step 3: Input the video to be detected into the trained temporally enhanced visual-text competition fusion network model to perform video anomaly detection; In step two, the specific process of training the temporally enhanced visual-text competition fusion network model includes: S1. Given an input video V, the video-level label of the input video V is Y. The input video V is divided into T segments, i.e. Each video segment is composed of express; S2. For each video segment, the video segment is input into a visual branch and a text branch respectively. The visual branch is used to extract the visual features of the video segment. and the extracted visual features Output; the text branch is used to extract text features corresponding to the video clips. and extract the text features Output; S3, Visual features of the output of the visual branch and the text features output by the text branch The inputs are processed by the local-global temporal dependency module, which then outputs the local-global temporal dependency visual features. and local-global time-dependent text features ,in, ; S4, the aforementioned and Simultaneously, the visual-text information is input into the visual-text information competition fusion module for fusion to obtain fused features. ,in, Specifically: S4.1, the aforementioned and The information is fed into the first cross-attention mechanism, which embeds the textual information into the visual features. The specific process is as follows: S4.01, the aforementioned and Mapped to queries respectively and key Sum , is represented as: , , ;in, , and Represents the learnable linear projection parameters; S4.11 Calculate the cross-attention scores of text features and visual features, and apply residual connections to obtain visual features that fuse text semantics. , is represented as: ; , express The feature dimension; S4.2, Visual features In the second cross-attention mechanism, visual features are again processed by the second cross-attention mechanism. The specific process of embedding into text features is as follows: S4.02, will and The input is a second cross-attention mechanism, and through a similar mapping process, ... and Mapped to queries respectively and key Sum It is represented as: = · , = · , = · ;in, , and Represents the learnable linear projection parameters; S4.12, Calculation and By calculating the cross-attention scores and applying residual connections, text features incorporating visual information are obtained. It is represented as: + ;in, express and Feature dimension, ; S4.

3. A cross-modal feature dynamic competitive activation mechanism is adopted to dynamically weigh the relative importance of visual and textual information, and to form the final visual features that fuse textual semantics by aggregating information from different features. And finally, the text features that integrate visual information. The final visual features of fused text semantics Represented as: The final text features fused with visual information Represented as: The number of channels for the feature is The feature dimension is , , Visual-text information complementary enhancement features Represented as: ,in, , This represents the operation of maximizing activation; S5. The fusion feature is then... The segments are input into the MIL classifier for segment-level anomaly score prediction, resulting in a set of segment anomaly scores for the video. ,in, This represents the anomaly score of the i-th video segment; S6. Calculate the training loss of the MIL classifier using the loss function, and train the MIL classifier through gradient backpropagation to obtain the trained MIL classifier. S7. Using the next input video, return to step S1 and repeat until the trained temporally enhanced visual-text competition fusion network model is obtained.

2. The weakly supervised video anomaly detection method according to claim 1, characterized in that: In step S3, the local-global temporal dependency module sequentially includes a multi-scale temporal feature fusion module, a multi-head self-attention module, and an SE Net module; the visual features output by the visual branch... The specific process of inputting into the local-global time-dependent module for processing includes the following steps: S3.1 In the multi-scale temporal feature fusion module, a one-dimensional convolution with a kernel size of s is used to fuse visual features. The convolution operation is represented as follows: ;in, This represents a one-dimensional convolution operation with a kernel size of s. Representing visual features The output features after applying one-dimensional convolution in the time dimension Visual features are extracted through convolution operations. and visual features ; S3.2, Visual features Visual features and original visual features By performing fusion, multi-scale temporal fusion features are obtained. ; S3.3, The multi-scale temporal fusion features are described above. In the multi-head self-attention module, the input is first processed through three fully connected layers. Convert to query ,key and value ;in, , as well as Then , as well as Divided into , as well as ;in, , as well as c∈ {1, ..., H}; then the multi-head self-attention module calculates the self-attention of each head separately. Its expression is: = SA( , , The self-attention results from each head are then merged to obtain the final output features. = MHSA( = Concat( , . . . , );in, ; S3.4, The output features The fragment feature weights are adjusted in the SE Net module; specifically, the first step is a Squeeze operation, which adjusts the weights of the output features. Perform global average pooling, where each segment feature is represented by a constant. express, Then comes the excitation operation, which uses a linear mapping to first reduce dimensionality and then increase it, learning the importance of fragment features, i.e., obtaining the weights of the fragment features. Finally, there's the scaling operation, which adjusts the attention weights. and Multiplication yields the final feature output. .

3. The weakly supervised video anomaly detection method according to claim 2, characterized in that: In step S6, the loss function is expressed as: ;in, and All represent weights; This represents the binary cross-entropy loss between the set of outlier scores S and the set of labels Y; Represents the Top-K out-of-packet loss function. , The abnormal scores in the set S of abnormal scores for video segments represent the abnormal scores. The set of the largest K elements and These represent the abnormality scores of segments in abnormal videos and the abnormality scores of segments in normal videos, respectively. This represents the k-maxmin in-packet loss function. , This represents the k-maxmin positive bag in-loss function. , The abnormal scores in the set S of abnormal scores of video clips The smallest set of K elements. This represents the k-maxmin negative in-pack loss function. ; Represents the smoothing loss function. .

4. The weakly supervised video anomaly detection method according to claim 1, characterized in that: In step three, the specific process of inputting the video to be detected into the trained temporally enhanced visual-text competitive fusion network model for video anomaly detection is as follows: the video to be detected is input into the trained temporally enhanced visual-text competitive fusion network model, the trained MIL classifier predicts the anomaly scores of all video segments in the video to be detected, a threshold is set, and the type of each video segment is divided according to the threshold, that is, video segments with anomaly scores greater than the threshold are abnormal segments, and video segments with anomaly scores less than the threshold are normal segments.

Images