Weakly supervised video anomaly detection method based on visual language instance perception learning
By employing a visual language instance-based perception learning method, we have addressed the issues of local dominance and cross-modal semantic gap in weakly supervised video anomaly detection, thereby improving detection accuracy and stability, providing interpretable anomaly judgments, and adapting to complex monitoring environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGZHOU RES INST OF XIAN UNIV OF ELECTRONIC SCI & TECH
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-05
AI Technical Summary
Existing weakly supervised video anomaly detection methods suffer from local dominance problems and cross-modal semantic gaps, resulting in high false negative rates for latent anomalies and inaccurate detection accuracy due to noise interference from generated captions.
We construct a visual-language instance-aware learning method. By combining visual and linguistic features through multi-granular semantic extraction at the focusing, saccade, and global levels, we build a parallel dual-branch instance-aware learning architecture for multi-stage differentiated training. We utilize latent positive sample mining and dynamic semantic integration to reduce the impact of noise and improve detection accuracy.
It significantly reduces the risk of missing latent anomalies, improves detection accuracy and stability, provides interpretable anomaly judgment results, and adapts to complex and ever-changing monitoring environments.
Smart Images
Figure CN122156748A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of video anomaly detection technology, specifically relating to a weakly supervised video anomaly detection method based on visual language instance perception learning. Background Technology
[0002] With the widespread adoption of urban security systems, video anomaly detection (VAD) has become a research hotspot in computer vision. However, due to the extreme scarcity of real-world anomaly samples and the inexhaustible nature of anomaly types, fully supervised learning methods are difficult to implement. Therefore, weakly supervised video anomaly detection (WSVAD) has become mainstream, using only video-level labels rather than frame-level labels for training. Current mainstream methods are mostly based on the Multi-Instance Learning (MIL) framework, detecting anomalies by maximizing the difference in feature magnitude between instances (video clips) in the positive packet (abnormal video) and the negative packet (normal video).
[0003] Despite some progress in existing technologies, the following significant problems still limit their application in complex real-world scenarios: The Local Domination Problem: Existing MIL methods rely excessively on feature magnitude, causing the model to focus only on the most visually striking anomalous segments in the video, while systematically ignoring events with smooth visual features but semantically abnormalities. This "winner-takes-all" mechanism results in an extremely high rate of missed detections of latent anomalies.
[0004] Modality Gap: Recent research has attempted to introduce visual-language models (such as CLIP) to assist in detection. However, directly applying image-text pre-trained models to video tasks results in a severe modality gap. Videos exhibit temporal continuity, with extremely high visual redundancy between adjacent frames. Existing image-text contrastive learning methods lack modeling of this temporal correlation, causing textual knowledge to be unable to be accurately anchored to the temporal evolution of the video.
[0005] Noise interference in subtitle generation: Existing multimodal video descriptor (VAD) methods typically generate video subtitles directly using image captioning models. Due to a lack of targeted design, the generated subtitles often contain a lot of noise, or only describe local actions (ignoring the global background), or only describe static scenes (ignoring dynamic behaviors). This low-quality text information can mislead the detection model and affect detection accuracy. Summary of the Invention
[0006] The purpose of this invention is to provide a weakly supervised video anomaly detection method based on visual language instance perception learning.
[0007] The technical solution for achieving the objective of this invention is a weakly supervised video anomaly detection method based on visual language instance perception learning, the method comprising the following steps:
[0008] Step A: Focus and scan the video to extract multi-granular video semantics;
[0009] Step B: Perform video-specific visual and linguistic contrast loss and construct a loss function for video temporal characteristics;
[0010] Step C: Construct a parallel dual-branch instance-aware learning architecture to simulate human perception and cognition channels respectively;
[0011] Step D: Implement multi-stage differentiated training and search for abnormal clues in the remaining video clips.
[0012] Further preferred: Step A includes a focusing layer, used to divide the input video into N temporal segments, randomly sample keyframes in each segment, generate detailed descriptions of the keyframes, and capture details of subtle objects and local limb movements in the scene.
[0013] The saccade layer is used to generate segment-level event descriptions using a video description model, and to capture short- to medium-length continuous motion streams and local context of events.
[0014] The global layer is used to analyze the entire video using a long video understanding model, generate a global summary, and provide macro-level scene information and event background, while providing environmental constraints for local anomalies.
[0015] Dynamic semantic integration and cleaning are used to construct a candidate text pool from the above three layers of text descriptions. For each frame in the video, the text description that best matches the visual content of the current frame is dynamically selected as the final semantic input of the frame based on the similarity between the visual features and all text features in the candidate text pool.
[0016] A further preferred embodiment is: Step B includes latent positive sample mining for strong temporal correlation based on video frames, and constructing a similarity matrix for contrastive learning, which is then used to classify the current frame... Its corresponding text It will be considered a positive sample, and will also be considered if its visual similarity to the current frame exceeds a threshold. adjacent frames The corresponding text Also marked as "potential positive sample";
[0017] Modify the comparison matrix to construct the corrected assignment matrix, which includes real positive samples and mined potential positive samples;
[0018] Loss calculation is used to weight and correct the standard noise-contrast estimation loss function (InfoNCE) using a matrix, and to force the text encoder to adapt to the continuous changes in the video, so that the general language knowledge can be smoothly and accurately aligned to the continuous video frame sequence.
[0019] A further preferred embodiment is that step C includes a visual perception branch, the input of which is the visual features of the original video frame, and modeling is performed using a local-global temporal network;
[0020] The knowledge recognition branch takes extracted and aligned text features as input, processes them through a local-global temporal network with shared or independent parameters, and outputs a "semantic anomaly score".
[0021] Category alignment mapping maps the feature vectors output by the two visual perception branches and the knowledge cognition branches to the semantic spaces of the two categories, respectively, and calculates the cosine similarity as the final classification basis, so that the features have clear semantic orientation.
[0022] A further preferred method is: in step D, basic instance perception is included, and only the K instances with the highest abnormal scores in the positive packet are selected to calculate the classification loss;
[0023] Differentiated instance perception calculates the cosine similarity between the temporal attention distributions of the visual perception branch and the knowledge cognition branch, and minimizes this similarity;
[0024] Potential anomaly detection involves masking or removing anomalous segments that have already been identified with high confidence in basic instance perception and differentiated instance perception, and then searching for anomalous clues in the remaining video segments.
[0025] Compared with existing technologies, the present invention has the following positive effects: The present invention constructs a parallel dual-branch instance-aware learning architecture, which simulates human perception and cognition channels respectively. With dual-branch collaboration as the core, the video anomaly detection is upgraded from a paradigm that simply relies on appearance / motion mutations to a paradigm of joint discrimination of "visual evidence + semantic prior". It can simultaneously cover both significant and latent anomalies under weak supervision, avoid the model being dominated by only a few high-response segments, improve the integrity and stability of anomaly localization from a mechanism perspective, and also help improve detection accuracy.
[0026] By supplementing and correcting abnormal clues, the ability to identify scenes with "insignificant visual signals but clear semantic direction" such as logical anomalies and intentional anomalies is significantly enhanced, effectively reducing the risk of missing latent anomalies.
[0027] By introducing differentiated training and potential anomaly detection mechanisms, the model can still maintain convergence and stability even under conditions of weak supervision, large pseudo-label noise, and long-tailed anomaly distribution. It also improves the coverage of different anomaly forms and enhances its comprehensive capabilities, enabling it to better adapt to the complex and ever-changing anomaly definitions and data distribution drift in real monitoring environments.
[0028] It not only outputs anomaly scores, but also provides textual evidence and semantic explanations consistent with the anomaly judgment, making the alarm results verifiable. This significantly lowers the threshold for using black-box models in actual business operations and provides a more engineering-feasible solution for anomaly detection and alarm in extreme environments. Attached Figure Description
[0029] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:
[0030] Figure 1 This is a flowchart of the present invention;
[0031] Figure 2 This is a structural architecture diagram of the present invention. Detailed Implementation
[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0033] Example
[0034] See Figure 1 and Figure 2 As shown, a weakly supervised video anomaly detection method based on visual language instance perception learning is proposed. The method includes the following steps:
[0035] Step A: Focusing and scanning the video for multi-granular video semantic extraction; Step A includes a focusing layer, which is used to divide the input video into N temporal segments, such as 24, 32 or other numbers of temporal segments. Keyframes are randomly sampled in each segment. The keyframes are generated by using a multimodal large model with high-resolution perception capabilities to generate detailed descriptions of the keyframes and capture subtle objects (such as knives, flames, etc.) and local limb movement details in the scene.
[0036] The scanning layer is used to generate segment-level event descriptions using a video description model and to capture short- to medium-length continuous motion streams and local context of events. It uses a sliding window mechanism (e.g., a 64-frame window with a step size of 16 frames) to extract video segments, focusing on capturing short- to medium-length continuous actions (such as running and fighting) and local context of events.
[0037] The global layer is used to analyze the entire video using a long video understanding model, generate a global summary, and provide macro-level scene information (such as a nighttime parking lot) and event background, providing environmental constraints for local anomalies.
[0038] Dynamic semantic integration and cleaning are used to construct a candidate text pool from the above three layers of text descriptions. For each frame in the video, the text description that best matches the visual content of the current frame is dynamically selected as the final semantic input of the frame based on the similarity between the visual features and all text features in the candidate text pool.
[0039] Step B involves applying video-specific visual and linguistic contrastive loss and constructing a loss function for video temporal characteristics. Step B includes latent positive sample mining for strong temporal correlation based on video frames. In this process, adjacent frames are considered highly semantically consistent. A similarity matrix based on contrastive learning is constructed, and the current frame... Its corresponding text It will be considered a positive sample, and will also be considered if its visual similarity to the current frame exceeds a threshold. adjacent frames The corresponding text Also marked as "potential positive sample";
[0040] Modify the comparison matrix to construct the corrected assignment matrix M, which includes real positive samples and mined potential positive samples;
[0041] Loss calculation is used to weight and correct the standard noise-contrast estimation loss function (InfoNCE) using matrix M, and to force the text encoder to adapt to the continuous changes in the video, so that the general language knowledge can be smoothly and accurately aligned to the continuous video frame sequence, mainly to eliminate the modal gap.
[0042] Step C: Construct a parallel dual-branch instance-aware learning architecture to simulate human perception and cognition channels respectively; Step C includes a visual perception branch, the input of which is the visual features of the original video frames, and is modeled using a Local-Global Temporal Network (LGTN);
[0043] LGTN includes a Transformer Encoder for capturing short-term dependencies and a Graph Convolutional Network (GCN) for capturing long-term dependencies; this branch outputs a "visual anomaly score" that focuses on identifying physical-level abrupt changes (such as drastic changes in velocity or optical flow).
[0044] The knowledge recognition branch takes extracted and aligned text features as input, processes them through a local-global temporal network with shared or independent parameters, and outputs a "semantic anomaly score". This branch focuses on identifying anomalies at the logical level (such as the sociological meaning of the action "someone is picking a lock").
[0045] Category alignment mapping introduces learnable prompts (such as "Anomalous Event" and "Normal Event"), mapping the feature vectors output by the two visual perception branches and the knowledge cognition branch to the semantic spaces of the two category prompts respectively, and calculating the cosine similarity.
[0046] As the final classification criterion, the features have a clear semantic orientation.
[0047] Step D: Implement multi-stage differentiated training and search for anomaly clues in the remaining video clips. Step D includes basic instance perception, using the Top-K MIL strategy, which selects only the K instances with the highest anomaly scores in the positive packet to calculate the classification loss; the purpose of this stage is to enable the model to quickly learn to identify the most significant and easily detectable anomalies (such as explosions).
[0048] Differential instance perception introduces a differential loss, which calculates the cosine similarity between the temporal attention distributions of the visual perception branch and the knowledge cognition branch, and minimizes this similarity. This mechanism forces the visual branch and the text branch to focus on different regions in the video, and expands the detection coverage by leveraging the complementarity of multimodal approaches.
[0049] Potential anomaly detection involves introducing a masking mechanism to cover up or remove anomalous segments already identified with high confidence in basic instance detection and differential instance detection, while searching for anomalous clues in the remaining video segments. This strategy is specifically designed to uncover "hard samples" that are easily overlooked, such as covert theft.
[0050] This invention has been validated on publicly available benchmark datasets, demonstrating stable and significant performance in weakly supervised video anomaly detection tasks: achieving an AUC of 88.75% on the UCF-Crime dataset and an AP of 85.38% on the XD-Violence dataset. Compared to the representative pure vision method DMU, this invention improves the AP by 3.72 percentage points on XD-Violence (85.38% vs 81.66%) and the AUC by 1.78 percentage points on UCF-Crime, demonstrating that "multi-granular semantic construction, video-specific cross-modal alignment, and perceptual / cognitive dual-branch collaboration" can significantly enhance the ability to discriminate complex scenes and latent anomalies. Furthermore, in a setting relying solely on text modal reasoning, this invention achieves relative improvements of 4.40 and 9.93 percentage points on UCF-Crime and XD-Violence, respectively, compared to the comparative method, indicating that it maintains strong robustness and interpretable anomaly recognition capabilities even when visual signals are impaired or scenes change significantly.
[0051] Compared with existing technologies, the present invention has the following positive effects: The present invention constructs a parallel dual-branch instance-aware learning architecture, which simulates human perception and cognition channels respectively. With dual-branch collaboration as the core, the video anomaly detection is upgraded from a paradigm that simply relies on appearance / motion mutations to a paradigm of joint discrimination of "visual evidence + semantic prior". It can simultaneously cover both significant and latent anomalies under weak supervision, avoid the model being dominated by only a few high-response segments, improve the integrity and stability of anomaly localization from a mechanism perspective, and also help improve detection accuracy.
[0052] By supplementing and correcting abnormal clues, the ability to identify scenes with "insignificant visual signals but clear semantic direction" such as logical anomalies and intentional anomalies is significantly enhanced, effectively reducing the risk of missing latent anomalies.
[0053] By introducing differentiated training and potential anomaly detection mechanisms, the model can still maintain convergence and stability even under conditions of weak supervision, large pseudo-label noise, and long-tailed anomaly distribution. It also improves the coverage of different anomaly forms and enhances its comprehensive capabilities, enabling it to better adapt to the complex and ever-changing anomaly definitions and data distribution drift in real monitoring environments.
[0054] It not only outputs anomaly scores, but also provides textual evidence and semantic explanations consistent with the anomaly judgment, making the alarm results verifiable. This significantly lowers the threshold for using black-box models in actual business operations and provides a more engineering-feasible solution for anomaly detection and alarm in extreme environments.
[0055] The standard parts used in this embodiment can be purchased directly from the market, and the non-standard structural parts described in the instruction manual can also be processed without any doubt based on existing technical common sense. At the same time, the connection methods of each component adopt mature conventional methods in the existing technology, and the machinery, parts and equipment all adopt conventional models in the existing technology, so they will not be described in detail here.
[0056] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, these obvious variations or modifications derived from the essential spirit of the present invention still fall within the scope of protection of the present invention.
Claims
1. A weakly supervised video anomaly detection method based on visual language instance perception learning, characterized in that: The method includes the following steps: Step A: Focus and scan the video to extract multi-granular video semantics; Step B: Perform video-specific visual and linguistic contrast loss and construct a loss function for video temporal characteristics; Step C: Construct a parallel dual-branch instance-aware learning architecture to simulate human perception and cognition channels respectively; Step D: Implement multi-stage differentiated training and search for abnormal clues in the remaining video clips.
2. The weakly supervised video anomaly detection method based on visual language instance perception learning according to claim 1, characterized in that: Step A includes a focusing layer, which is used to divide the input video into N temporal segments, randomly sample keyframes in each segment, generate detailed descriptions of the keyframes, and capture details of small objects and local body movements in the scene. The saccade layer is used to generate segment-level event descriptions using a video description model, and to capture short- to medium-length continuous motion streams and local context of events. The global layer is used to analyze the entire video using a long video understanding model, generate a global summary, and provide macro-level scene information and event background, while providing environmental constraints for local anomalies. Dynamic semantic integration and cleaning are used to construct a candidate text pool from the above three layers of text descriptions. For each frame in the video, the text description that best matches the visual content of the current frame is dynamically selected as the final semantic input of the frame based on the similarity between the visual features and all text features in the candidate text pool.
3. The weakly supervised video anomaly detection method based on visual language instance perception learning according to claim 2, characterized in that: Step B includes latent positive sample mining for strong temporal correlation based on video frames, and constructing a similarity matrix for contrastive learning, which is then used to identify the current frame. Its corresponding text It will be considered a positive sample, and will also be considered if its visual similarity to the current frame exceeds a threshold. adjacent frames The corresponding text is also labeled as "potential positive sample"; Modify the comparison matrix to construct the corrected assignment matrix, which includes real positive samples and mined potential positive samples; Loss calculation is used to weight and correct the standard noise-contrast estimation loss function (InfoNCE) using a matrix, and to force the text encoder to adapt to the continuous changes in the video, so that the general language knowledge can be smoothly and accurately aligned to the continuous video frame sequence.
4. The weakly supervised video anomaly detection method based on visual language instance perception learning according to claim 3, characterized in that: Step C includes a visual perception branch, which takes the visual features of the original video frames as input and uses a local-global temporal network for modeling. The knowledge recognition branch takes extracted and aligned text features as input, processes them through a local-global temporal network with shared or independent parameters, and outputs a "semantic anomaly score". Category alignment mapping maps the feature vectors output by the two visual perception branches and the knowledge cognition branches to the semantic spaces of the two categories, respectively, and calculates the cosine similarity as the final classification basis, so that the features have clear semantic orientation.
5. The weakly supervised video anomaly detection method based on visual language instance perception learning according to claim 4, characterized in that: Step D includes basic instance perception, in which only the K instances with the highest abnormal scores in the positive packet are selected to calculate the classification loss; Differentiated instance perception calculates the cosine similarity between the temporal attention distributions of the visual perception branch and the knowledge cognition branch, and minimizes this similarity; Potential anomaly detection involves masking or removing anomalous segments that have already been identified with high confidence in basic instance perception and differentiated instance perception, and then searching for anomalous clues in the remaining video segments.