Weakly supervised video anomaly detection method based on prompt enhanced multi-instance learning
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF CHINESE ACAD OF SCI
- Filing Date
- 2024-04-16
- Publication Date
- 2026-07-21
Smart Images
Figure CN118537768B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a weakly supervised video anomaly detection method based on cue-enhanced multi-instance learning, belonging to the field of computer vision. Background Technology
[0002] Video anomaly detection is used to identify frame-level anomalies in videos. To achieve good generalization ability across different scenarios, weakly supervised video-level labeling (wVAD) is often used for video anomaly detection. However, two key issues arise when using weak supervision with video-level labels: 1) detecting complex anomaly patterns in various scenarios, where the temporal relationships and visual appearances of anomalies exhibit significant differences; 2) generating clear boundaries for anomalous events without fine-grained boundary annotations.
[0003] To address the aforementioned issues, existing technologies have proposed a method for handling weakly supervised VADs by generating noisy frame-level anomaly labels and subsequently reducing the noise. However, this method limits its generalization ability in invisible scenes.
[0004] Existing technologies also utilize Multiple Instance Learning (MIL) to address weakly supervised Visual Anomaly Detection (VAD). To detect anomalous events through frame-level anomaly scores, MIL-based methods typically apply ranking loss to encourage the highest score in anomalous videos to be higher than the highest score in normal videos. In this way, MIL can learn unique patterns within anomalous instances. However, it is well-known that MIL is weak in defining ambiguous event boundaries and modeling different anomaly patterns in complex scenes. To address this, some methods modify the objective function or training strategy to learn clear event boundaries; or introduce auxiliary information, such as knowledge bases, to model anomalous patterns in complex scenes. Despite these advancements, two major challenges remain:
[0005] 1) Binary labels are insufficient to capture complex anomaly patterns because they only capture a general anomaly pattern. Furthermore, binary labels ignore semantic relevance that can represent the similarity and uniqueness of patterns that exist between anomalous events.
[0006] 2) The coupling between exceptions and contextual patterns hinders the learning of clear boundaries.
[0007] Therefore, it is necessary to further study existing weakly supervised video anomaly detection methods to improve the ability to identify frame-level anomalies in videos. Summary of the Invention
[0008] To overcome the above problems, the inventors conducted in-depth research and designed a weakly supervised video anomaly detection method based on cue-enhanced multi-instance learning, characterized by comprising:
[0009] S101. Extract video and audio features, and set contextual cue features;
[0010] S102. Set the time feature fusion module to perform time feature fusion on video features, audio features, and contextual cue features to obtain temporal features;
[0011] S103. Set the prediction head to predict the temporal features, obtain the anomaly score, and determine whether there is an anomaly in the video based on the anomaly score.
[0012] In a preferred embodiment, in S102, the video features and audio features are concatenated along the feature dimension and then concatenated with the contextual cue features along the length to obtain multimodal features. Multimodal features are used as input to the temporal feature fusion module.
[0013] In a preferred embodiment, the temporal feature fusion module is an attention mechanism architecture, including a three-layer self-attention mechanism, a Norm layer, and a fourth linear layer; wherein, multimodal features After passing through three linear layers, numerical features f are generated. v Query feature f q and key features f k Query feature f q and key features f k The dot product yields the similarity matrix, which is then normalized to obtain the global map A. g , global map A g With numerical characteristics f v Contextual attention features are obtained after dot product.
[0014] In the time feature fusion module, dynamic position encoding is also performed on the similarity matrix to obtain the similarity matrix. Represented as:
[0015]
[0016] ε j，k =exp(-|γ(jk)) 2 +β|)
[0017] Where, ε=[ε j，k ], j∈[1,N], k∈[1,N], j and k represent the indices of the video and audio segments, N represents the number of segments, γ represents the learnable weights, β represents the learnable bias term, f q () indicates the extraction of query features, f k () indicates a key feature, and the superscript T indicates transpose.
[0018] In a preferred embodiment, the time feature fusion module further includes a mask similarity matrix. After normalization, region map A is obtained. l Map A of the region l With numerical characteristics f v Dot product yields event context features context attention features Event context features Weighted fusion is performed, and the fused features are then passed through a Norm layer and a fourth linear layer before being combined with multimodal features. The data is then concatenated and passed through a normalization layer to obtain temporal features.
[0019] In a preferred embodiment, in S103, the prediction head includes a perceptron, which can be represented as:
[0020]
[0021] in, Represents semantic features, Dropout() represents the Dropout layer, GELU() represents the GELU activation function, and Conv s1 This represents a one-dimensional convolutional layer with a stride of s1. Indicates temporal characteristics;
[0022] The obtained abnormal score Represented as:
[0023]
[0024] Where σ() represents the sigmoid activation function, f t () denotes a causal convolutional layer, Dropout() denotes a Dropout layer, GELU() denotes the GELU activation function, and Conv s2 () represents a one-dimensional convolutional layer with a stride of s2. Indicates semantic features.
[0025] In a preferred embodiment, before performing video anomaly detection, the temporal feature fusion module and the prediction head need to be trained to obtain suitable network parameters. The training includes the following steps:
[0026] S201. Based on the abnormal scenarios in the videos in the video training set, set up a semantic prompt set;
[0027] S202. Extract video features and audio features from the video training set and the corresponding audio training set. Without setting contextual cue features, simply concatenate the video features and audio features along the feature dimension to obtain the multimodal features. The network parameters are then trained sequentially through the time feature fusion module and the prediction head.
[0028] In a preferred embodiment, in S201, the semantic cue set is a set of video annotations, which are used to describe whether the video is normal or an abnormal situation.
[0029] In a preferred embodiment, the contextual hint feature is obtained in the following manner:
[0030] S401. After completing the training of the temporal feature fusion module and the prediction head, freeze the network parameters of the temporal feature fusion module and the prediction head.
[0031] S402. Randomly set context cue features, extract video and audio features from the video training set and the corresponding audio training set, concatenate the video and audio features along the feature dimension, and then concatenate them with the context cue features along the length to obtain multimodal features. The anomaly score is obtained after passing through the temporal feature fusion module and the prediction head in sequence.
[0032] S403. Set the video label to 0, set the loss function, and repeat step S402 to train using the video training set and audio training set to obtain contextual cue features.
[0033] The present invention also provides an electronic device, comprising:
[0034] At least one processor; and
[0035] A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in any of the preceding descriptions.
[0036] The present invention also provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method according to any one of the preceding claims.
[0037] The beneficial effects of this invention include:
[0038] (1) Use anomaly perception prompts to integrate semantic information into video features, and then introduce normal context prompts to enrich boundary context features, thereby identifying different anomaly patterns and generating clear boundaries, and improving the anomaly recognition rate.
[0039] (2) Introduce normal contextual cues as a sum of normal patterns. Such cues enrich the contextual features of the boundary to reveal a more discriminative feature, which can be used to decouple the context and the anomaly, thereby producing clear event boundaries and improving the anomaly recognition rate. Attached Figure Description
[0040] Figure 1 A flowchart is shown for a weakly supervised video anomaly detection method based on cue-enhanced multiple instance learning according to a preferred embodiment of the present invention. Detailed Implementation
[0041] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the present invention will become clearer and more apparent.
[0042] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.
[0043] This invention provides a weakly supervised video anomaly detection method based on cue-enhanced multi-instance learning, such as... Figure 1 As shown, it includes:
[0044] S101. Extract video and audio features, and set contextual cue features;
[0045] S102. Set the time feature fusion module to perform time feature fusion on video features, audio features, and contextual cue features to obtain temporal features;
[0046] S103. Set the prediction head to predict the temporal features, obtain the anomaly score, and determine whether there is an anomaly in the video based on the anomaly score.
[0047] In S101, for the input video and its corresponding audio Use a pre-trained video encoder to extract from video Extracting video features Use a pre-trained audio encoder to extract audio Extracting audio features
[0048] Preferably, the input video is pre-loaded before feature extraction. and audio The video is divided into multiple non-overlapping frames, and features are extracted from each frame to form segment-level video features. and fragment-level audio features Where N is the number of segments, D v It is the feature dimension of the video, D a It is the feature dimension of the audio.
[0049] Preferably, the multiple frames are 16 frames.
[0050] By dividing the data into multiple non-overlapping frames, computational overhead can be saved and recognition efficiency can be improved.
[0051] In this invention, the specific network structure of the video encoder and the audio encoder is not limited. Those skilled in the art can use commonly used encoders, such as I3D for video encoder and VG Gish for audio encoder.
[0052] The contextual hint features are used to help generate clear boundaries of abnormal events, and their specific feature values can be set by those skilled in the art based on experience.
[0053] By setting contextual cue features, the potential normal event distribution of the trained model can be summarized, and clear boundaries of abnormal events can be generated by enriching fuzzy contextual features.
[0054] In S102, the Temporal Feature Fusion (TFF) module is set to process and fuse video and audio features to obtain temporal features.
[0055] Specifically, video and audio features are concatenated along their feature dimensions and then concatenated with contextual cue features along their length to obtain multimodal features. Multimodal features are used as input to the temporal feature fusion module.
[0056] The temporal feature fusion module is an attention mechanism architecture for interpretable multi-time step prediction, including a self-attention mechanism with three linear layers, a Norm layer, and a fourth linear layer.
[0057] Among them, multimodal features After passing through three linear layers, numerical features f are generated. v Query feature f q and key features f k Query feature f q and key features f k The dot product yields the similarity matrix, which is then normalized to obtain the global map A. g , global map A g With numerical characteristics f v Contextual attention features are obtained after dot product. The process is the same as that of a traditional Transformer module (such as a TFT module).
[0058] Unlike traditional Transformer modules, the temporal feature fusion module performs dynamic positional encoding on the similarity matrix to obtain the desired similarity matrix. Represented as:
[0059]
[0060] ε j，k =exp(-|γ(jk)) 2 +β|)
[0061] Where, ε=[ε j，k ], j∈[1,N], k∈[1,N], j and k represent the indices of the video and audio segments, N represents the number of segments, γ represents the learnable weights, β represents the learnable bias term, f q (·) indicates the extraction of query features, f k (·) indicates a key feature, and the superscript T indicates transpose.
[0062] Furthermore, unlike the traditional Transformer module, the temporal feature fusion module also includes a mask similarity matrix. By analyzing the similarity matrix Obtained by masking, as follows:
[0063]
[0064]
[0065] in, Represents the mask similarity matrix The element i represents the index of the video segment, and w represents the window size.
[0066] Furthermore, the mask similarity matrix After normalization, region map A is obtained. l .
[0067] Map A of the region l With numerical characteristics f v Dot product yields event context features Represented as:
[0068]
[0069] Among them, D h This represents the hidden layer dimension in the attention mechanism architecture.
[0070] Furthermore, contextual attention features Event context features Weighted fusion is performed, and the fused features are then passed through a Norm layer and a fourth linear layer before being combined with multimodal features. The data is then concatenated and passed through a normalization layer to obtain temporal features. Represented as:
[0071]
[0072]
[0073] in, Let f represent the weighted fusion features, where α is the learnable weight and f is the weight. o The output features are represented by Norm(), which represents the Norm layer, and LayerNorm represents the normalization layer.
[0074] In this invention, by setting a mask similarity matrix and weighted fusion, local attention is improved, the problem of long-term noise is solved, and anomaly identification is made more accurate.
[0075] Furthermore, after adding contextual cue features, the temporal feature fusion module can capture the unique features of anomalies and contexts through an attention mechanism. It leverages the high feature correlation to dynamically integrate the abnormal context with the rich content of the normal domain, thereby amplifying the feature differences. This allows the subsequent prediction head to use these differences to accurately determine event boundaries.
[0076] In S103, the prediction head includes a perceptron, which can be represented as:
[0077]
[0078] in, Represents semantic features, Dropout() represents the Dropout layer, GELU() represents the GELU activation function, and Conv s1 This represents a one-dimensional convolutional layer with a stride of s1. It represents the temporal characteristics.
[0079] The obtained abnormal score Represented as:
[0080]
[0081] Where σ() represents the sigmoid activation function, f t () denotes a causal convolutional layer, Dropout() denotes a Dropout layer, GELU() denotes the GELU activation function, and Conv s2 () represents a one-dimensional convolutional layer with a stride of s2. Indicates semantic features.
[0082] According to the present invention, the causal convolutional layer, the Dropout layer, and the convolutional layer are all commonly used neural network structures, and their specific structures will not be described in detail in the present invention.
[0083] According to the present invention, when the anomaly score is lower than a preset value, it is considered that there is no anomaly in the video; otherwise, it is considered that there is anomaly in the video.
[0084] According to the present invention, before performing video anomaly detection, the temporal feature fusion module and the prediction head need to be trained to obtain suitable network parameters.
[0085] In this invention, unlike traditional neural network training, the training includes the following steps:
[0086] S201. Based on the abnormal scenarios in the videos in the video training set, set up a semantic prompt set;
[0087] S202. Extract video features and audio features from the video training set and the corresponding audio training set. Without setting contextual cue features, simply concatenate the video features and audio features along the feature dimension to obtain the multimodal features. The network parameters are then trained sequentially through the time feature fusion module and the prediction head.
[0088] In S201, the semantic prompt set is a set of video annotations, which are used to describe whether the video is normal or abnormal. For example, a traffic video with a car accident is annotated as "car accident"; a traffic video without abnormalities is annotated as "normal".
[0089] For any given video, the annotations in the semantic cue set can be divided into three categories: positive annotations, negative annotations, and related annotations.
[0090] Positive annotations describe unusual situations that occur in the video. For example, in a traffic video with a car accident, "car accident" is a positive annotation.
[0091] The negative class annotation is "normal", used to describe normal videos, that is, videos in which no abnormal situations have occurred;
[0092] Related categories are scenarios that did not appear in this video but appeared in other videos in the video set.
[0093] In S202, during training, for abnormal videos, the top q abnormal scores are selected to enhance the abnormal features, and the maximum score of normal videos is sampled to reduce the abnormal scores of normal videos.
[0094] Preferably, q is set as follows:
[0095]
[0096] Where y=1 represents abnormal video, y=0 represents normal video, and a represents the number of frames when divided into multiple non-overlapping segments.
[0097] Preferably, the average of the first k anomaly scores is used as the anomaly prediction for the anomaly video.
[0098] Preferably, the loss function during training Set to:
[0099]
[0100] Preferably, the objective function in the training is obtained by the following method:
[0101] S301, using the scaling anomaly score as the activation value, separates video-level events and contextual features, represented as follows:
[0102]
[0103]
[0104]
[0105] in, Indicates the characteristics of the event, This represents the contextual features, and μ represents the configurable scale factor. This represents the visual features obtained by concatenating event features with contextual features.
[0106] Since videos may contain examples of events and context, aligning all segments with the same cue can confuse the model with unclear semantics. Step S303 can solve the above problem.
[0107] S302. Convert the annotations in the semantic hint set into text features;
[0108] Preferably, the annotations in the semantic cue set are first converted into embedding tensors, then learnable cue is connected to the embedding tensors, and text features are obtained through a text encoder.
[0109] Since the original category annotations are too concise to summarize complex events and lack rich semantics, this invention introduces learnable hints into the original text embedding, which can improve generalization ability and derive semantically rich text features.
[0110] Preferably, the embedding tensor representation of positive class annotations is as follows:
[0111]
[0112] The learnable hints and embedded tensor connection process is represented as follows:
[0113]
[0114] The positive class annotation text features obtained through the text encoder are represented as T. p ,
[0115] Similarly, obtain the relevant class annotation text features T. r Negative class annotation text features T n The text features of positive class annotations, related class annotations, and negative class annotations are concatenated to form the text feature T for each label:
[0116] T = {T p T r T n}
[0117] S303. Obtain the visual text relevance distribution P, represented as:
[0118]
[0119]
[0120] Where C represents the anomaly category number, τ represents the temperature factor, and k represents the fragment index.
[0121] S304. Dynamically calculate semantic relevance to form an alignment target, represented as follows:
[0122]
[0123] Where c is the scaling factor, which is equal to 1 if it is a normal video. The superscripts e and c correspond to the features of the event and context, and the subscripts p, r, and n refer to the positive class annotation, related class annotation, and negative class annotation of the text features.
[0124] S305, Obtaining Instant Learning Loss Represented as:
[0125]
[0126]
[0127] in, Indicates the target distribution. Expressing expectations, Indicates the goal.
[0128] Preferably, it further includes: S306, setting a prompt constraint loss.
[0129]
[0130] Among them, T l Indicates a learning prompt, T init This indicates the initial prompt.
[0131] Hints on setting constraint losses to ensure consistency between learnable hints and class annotations.
[0132] More preferably, during the training phase, the overall objective function is set as follows:
[0133]
[0134] The hyperparameters λ and β are used to balance the loss.
[0135] By optimizing the objective function, the method in this invention can generate more discriminative features using semantically rich features, thereby enabling accurate detection of various abnormal patterns.
[0136] Preferably, the contextual cue feature is obtained in the following way:
[0137] S401. After completing the training of the temporal feature fusion module and the prediction head, freeze the network parameters of the temporal feature fusion module and the prediction head.
[0138] S402. Randomly set context cue features, extract video and audio features from the video training set and the corresponding audio training set, concatenate the video and audio features along the feature dimension, and then concatenate them with the context cue features along the length to obtain multimodal features. The anomaly score is obtained after passing through the temporal feature fusion module and the prediction head in sequence.
[0139] S403. Set the video label to 0, set the loss function, and repeat step S402 to train using the video training set and audio training set to obtain contextual cue features.
[0140] Preferably, during training, the loss function Set to:
[0141]
[0142] in, Let i represent the abnormal score of segment i, and N represent the total number of segments.
[0143] Various embodiments of the methods described above in this invention can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0144] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0145] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0146] To provide interaction with a user, the methods and apparatus described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0147] The methods and apparatus described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0148] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.
[0149] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.
[0150] Example
[0151] Example 1
[0152] Experiments were conducted on the XD Violence dataset. The ShanghaiTech dataset includes 13 fixed-view campus scenes from surveillance systems, containing 238 videos in the training set and 199 videos in the test set. XDViolence is currently the most extensive dataset available for wVAD, including videos collected from various sources, including movies, games, and car cameras. This dataset includes 3954 training videos with video-level annotations and 800 test videos with frame-level labels, covering six different anomaly categories.
[0153] Experiments were conducted using a weakly supervised video anomaly detection method based on cue-enhanced multi-instance learning, including:
[0154] S101. Extract video and audio features, and set contextual cue features;
[0155] S102. Set the time feature fusion module to perform time feature fusion on video features, audio features, and contextual cue features to obtain temporal features;
[0156] S103. Set the prediction head to predict the temporal features, obtain the anomaly score, and determine whether there is an anomaly in the video based on the anomaly score.
[0157] In S101, the video is encoded into 1024-dimensional video features using an RGB stream I3D video encoder, which is pre-trained on the Kinetics dataset. For audio features, VGGish, pre-trained on the YouTube dataset, is used to divide the input video and audio into 16 non-overlapping segments before feature extraction.
[0158] In S102, video features and audio features are concatenated along the feature dimension and then concatenated with contextual cue features along the length to obtain multimodal features. Multimodal features are used as input to the temporal feature fusion module;
[0159] In the time feature fusion module, dynamic position encoding is also performed on the similarity matrix to obtain the similarity matrix. Represented as:
[0160]
[0161] ε j,k＝ exp(-|γ(jk) 2 +β|)
[0162] The time feature fusion module also includes a mask similarity matrix. After normalization, region map A is obtained. l Map A of the regionl With numerical characteristics f v Dot product yields event context features context attention features Event context features Weighted fusion is performed, and the fused features are then passed through a Norm layer and a fourth linear layer before being combined with multimodal features. The data is then concatenated and passed through a normalization layer to obtain temporal features.
[0163] In the XD-Violence dataset, the context cue feature length is set to 35. During the network parameter training phase, steps S201-S202 and S401-S403 are used, where λ and β are 1 and 8 respectively, and the ratios of s1 and s2 are 2 and 3 respectively.
[0164] Comparative Example 1
[0165] Using the same dataset as in Example 1, the same experiments as in Example 1 were conducted using the OCSVM method, Conv-AE method, MIL-Rank method, CA-VAD method, RTFM method, CRFD method, DDL method, MSL method, S3R method, MGFN method, UR-DMU method, Zhang et al. method, CMA-LA method, MACIL-SD method, CoMo method, PEL4VAD method, and HyperVD method.
[0166] Among them, the OCSVM method can be found in the literature Xiaogang Peng, Hao Wen, Yikai Luo, Xiao Zhou, KeyangYu, Yigang Wang, and ZizhaoWu. Learning weakly supervised audio-visualviolence detection in hyperbolic space. arXiv preprint arXiv:2305.18797, 2023.
[0167] For the Conv-AE method, see the literature: Tianyu Gao, Adam Fisch, and Danqi Chen. Making pretrained language models better few-shot learners. arXiv preprint arXiv:2012.15723,2020.
[0168] MIL-Rank Configuration SystemYujiang Pu and Xiaoyu Wu.Audio-guided attentionnetwork for weakly supervised violence detection.In 2022 2nd InternationalConference on Consumer Electronics and Computer Engineering(ICCECE),pages219–223.IEEE,2022.
[0169] CA-VAD was developed by Joao Carreira and Andrew Zisserman.Quo vadis, action recognition?a new model and the kinetics dataset.
[0170] Proceedings of the RTFM Method Yujiang Pu and Xiaoyu Wu.Locality-aware attentionnetwork with discriminative dynamics learning for weakly supervised anomaly detection.In 2022IEEE International Conference on Multimedia and Expo(ICME),pages 1–6.IEEE,2022.
[0171] CRFD features include Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, and QingmingHuang.Smart:Syntax-calibrated multi-aspect relation transformer for changecaptioning.IEEE Transactions on Pattern Analysis and Machine Intelligence,2024.
[0172] The DDL method can be found in the literature: Hui Lv, Zhongqi Yue, Qianru Sun, Bin Luo, Zhen Cui, and Hanwang Zhang. Unbiased multiple instance learning for weakly supervised video anomaly detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 8022–8031, 2023.
[0173] The MSL method can be found in the literature: Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for effcient video understanding. In European Conference on Computer Vision, pages 105–124. Springer, 2022.
[0174] The S3R method can be found in the literature: Yunbin Tu, Liang Li, Li Su, Shengxiang Gao, Chenggang Yan, Zheng-Jun Zha, Zhengtao Yu, and Qingming Huang. I2 transformer: Intra-and inter-relation embedding transformer for tv show captioning. IEEE Transactions on Image Processing, 31:3565–3577, 2022.
[0175] The MGFN method can be found in the literature: Shuning Chang, Yanchao Li, Shengmei Shen, Jiashi Feng, and Zhiying Zhou. Contrastive attention for video anomaly detection. IEEE Transactions on Multimedia, 24:4067–4076, 2021.
[0176] The UR-DMU method can be found in the literature: Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pretraining for open-world detection. Advances in Neural Information Processing Systems, 35: 9125–9138, 2022.
[0177] The Zhang et al. method can be found in the literature: Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision, page 322–339, 2020.
[0178] The CMA-LA method can be found in the literature: Hui Lv, Chuanwei Zhou, Zhen Cui, Chunyan Xu, Yong Li, and Jian Yang. Localizing anomalies from weakly labeled videos. IEEE Transactions on Image Processing, page 4505–4515, 2021.
[0179] The MACIL-SD method can be found in the literature: Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472, 2021.
[0180] For the CoMo method, please refer to the literature Yingxian Chen, Zhengzhe Liu, Baoheng Zhang, Wilton Fok, Xiaojuan Qi, and Yik-ChungWu. Mgfn: Magnitude contrastive glance-and-focus network for weakly-supervised video anomaly detection. In Proceedings of the AAAI Conference on Artifcial Intelligence, pages 387–395, 2023.
[0181] For the PEL4VAD method, see Seongheon Park, Hanjae Kim, Minsu Kim, Dahye Kim, and Kwanghoon Sohn. Normality guided multiple instance learning for weaklysupervised video anomaly detection. In Proceedings of the IEEE / CVF WinterConference on Applications of Computer Vision, pages 2665–2674, 2023.
[0182] For the HyperVD method, see the reference Cewu Lu, Jianping Shi, and Jiaya Jia. Abnormal event detection at 150fps in matlab. In Proceedings of the IEEE international conference on computer vision, pages 2720–2727, 2013.
[0183] The experimental results of Comparative Example 1 and Comparative Example 1 are shown in Table 1, where the average precision (AP) is used as a metric.
[0184] Table 1
[0185]
[0186] In Comparative Example 1, the OCSVM method and the Conv-AE method are semi-supervised methods, while the other methods are weakly supervised methods.
[0187] As can be seen from Table 1, the method in Example 1 outperforms all previous semi-supervised and weakly supervised methods (i.e., the method in Comparative Example 1). Notably, when using the same I3D-RGB video features and VGGish audio features, the method in Example 1 achieves an absolute gain of 2.54% in AP compared to the best method in Comparative Example 1. This superior result stems from the accurate modeling of divergent anomaly patterns by the method in Example 1, and from its ability to generate clear event anomaly scores and eliminate false positives, thus contributing to more accurate anomaly detection results.
[0188] The present invention has been described above with reference to preferred embodiments; however, these embodiments are merely exemplary and illustrative. Various substitutions and modifications can be made to the present invention based on these embodiments, all of which fall within the scope of protection of the present invention.
Claims
1. A weakly supervised video anomaly detection method based on cue-enhanced multi-instance learning, characterized in that, include: S101. Extract video and audio features, and set contextual cue features; S102. Set the time feature fusion module to perform time feature fusion on video features, audio features, and contextual cue features to obtain temporal features; S103. Set the prediction head to predict the temporal features, obtain the anomaly score, and determine whether there is an anomaly in the video based on the anomaly score. In S102, video features and audio features are concatenated along the feature dimension and then concatenated with contextual cue features along the length to obtain multimodal features. Multimodal features are used as input to the temporal feature fusion module; The temporal feature fusion module employs an attention mechanism architecture, comprising a self-attention mechanism with three linear layers, a Norm layer, and a fourth linear layer; among which, multimodal features... Numerical features are generated after passing through three linear layers. Query features and key features Query features and key features The dot product yields the similarity matrix, which is then normalized to obtain the global map. , global map With numerical characteristics Contextual attention features are obtained after dot product. In the time feature fusion module, dynamic position encoding is also performed on the similarity matrix to obtain the similarity matrix. Represented as: in, , Indicates the sequence number of video or audio segments. N Indicates the number of segments, Represents the learnable weights. This represents a learnable bias term. ( ) indicates the extraction of query features. () indicates key features, and the superscript T indicates transpose; The time feature fusion module also includes a mask similarity matrix. The region map is obtained after normalization. , regional map With numerical characteristics Dot product yields event context features Context attention features Event context features Weighted fusion is performed, and the fused features are then passed through a Norm layer and a fourth linear layer before being combined with multimodal features. The data is then concatenated and passed through a normalization layer to obtain temporal features. .
2. The weakly supervised video anomaly detection method based on cue-enhanced multiple instance learning according to claim 1, characterized in that, In S103, the prediction head includes a perceptron, which can be represented as: in, Represents semantic features, Dropout() represents the Dropout layer, and GELU() represents the GELU activation function. Indicates step size is One-dimensional convolutional layer, Indicates temporal characteristics; The obtained abnormal score Represented as: in, This represents the sigmoid activation function. This represents a causal convolutional layer, Dropout() represents a Dropout layer, and GELU() represents the GELU activation function. ( ) indicates a step size of 1. One-dimensional convolutional layer, Indicates semantic features.
3. The weakly supervised video anomaly detection method based on cue-enhanced multiple instance learning according to claim 1, characterized in that, Before performing video anomaly detection, the temporal feature fusion module and prediction head need to be trained to obtain suitable network parameters. The training includes the following steps: S201. Based on the abnormal scenarios in the videos in the video training set, set up a semantic prompt set; S202. Extract video and audio features from the video training set and the corresponding audio training set. Without setting contextual cue features, simply concatenate the video and audio features along the feature dimension to obtain the multimodal features. Then, the network parameters are trained sequentially through the time feature fusion module and the prediction head.
4. The weakly supervised video anomaly detection method based on cue-enhanced multiple instance learning according to claim 3, characterized in that, In S201, the semantic cue set is a set of video annotations, which are used to describe whether the video is normal or abnormal.
5. The weakly supervised video anomaly detection method based on cue-enhanced multi-instance learning according to claim 1, characterized in that, The contextual cue features are obtained in the following ways: S401. After completing the training of the temporal feature fusion module and the prediction head, freeze the network parameters of the temporal feature fusion module and the prediction head. S402. Randomly set context cue features, extract video and audio features from the video training set and the corresponding audio training set, concatenate the video and audio features along the feature dimension, and then concatenate them with the context cue features along the length to obtain multimodal features. The anomaly score is obtained after passing through the time feature fusion module and the prediction head in sequence. ; S403. Set the video label to 0, set the loss function, and repeat step S402 to train using the video training set and audio training set to obtain contextual cue features.
6. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
7. A computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.