A video anomaly detection method based on scene-conditional normalized stream
By segmenting the foreground target and jointly modeling scene features, the problems of insufficient high-level semantic feature expression and scene-dependent anomaly detection in existing technologies are solved, and stable and consistent anomaly detection in complex backgrounds is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGZHOU RES INST OF XIAN UNIV OF ELECTRONIC SCI & TECH
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-02
AI Technical Summary
Existing video anomaly detection methods model in pixel space or single target feature space, lacking effective expression of high-level semantic features, making it difficult to handle scene-dependent anomaly events, and background changes or environmental disturbances affect model stability.
By segmenting the foreground target into blocks, extracting temporal and spatial features, embedding them into block-level feature sequences, and using a Transformer encoding network for joint modeling, conditional probability mapping is performed in conjunction with scene features to determine abnormal events.
It improves the stability and robustness of anomaly detection, maintains consistent anomaly judgment criteria in complex backgrounds and dynamic scenes, and enhances detection capabilities under cross-scene conditions.
Smart Images
Figure CN122135296A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of video anomaly detection, and particularly relates to a video anomaly detection method based on scene condition standardized flow. BACKGROUND
[0002] Video anomaly detection is a key technology in the field of intelligent video monitoring, and is widely used in public safety monitoring, traffic management, intelligent security and other scenes. The main task is to automatically identify abnormal events deviating from the regular behavior pattern from video sequences. In practical applications, abnormal events usually have the characteristics of low occurrence frequency, various types and difficulty in enumeration, making it difficult to implement full-supervised learning methods based on a large number of abnormal samples. Therefore, the semi-supervised anomaly detection method using only normal samples for training is generally used in the prior art.
[0003] According to the processing object of video data, the existing semi-supervised video anomaly detection method can be mainly divided into two categories: one is a frame-centered detection method, and the other is a foreground target-centered detection method.
[0004] Among them, the frame-centered detection method usually constructs a reconstruction model or a future frame prediction model of video frames, and uses the reconstruction error or the prediction error as the basis for anomaly judgment. This kind of method mainly models the image in the pixel space, although the implementation is relatively direct, but there are obvious shortcomings. On the one hand, pixel-level modeling contains a large amount of background redundant information, making it difficult to effectively extract high-level semantic features related to abnormal behavior. On the other hand, when there are changes in illumination, background dynamic interference or slight camera jitter in the monitored scene, the changes in the background area will also cause the reconstruction error to increase, thereby affecting the stability of the anomaly judgment and reducing the generalization ability of the model.
[0005] The foreground target-centered detection method usually detects or tracks the moving target in the video first, then models the appearance features or motion trajectories of the target, and judges the target deviating from the normal appearance or motion pattern as abnormal. This kind of method reduces the background interference to a certain extent, but it mainly focuses on the feature distribution of the target itself, ignoring the context information of the environment where the target is located. In fact, in many application scenarios, the judgment of abnormality has obvious scene dependence, that is, the same behavior may have different semantics in different scenes. For example, a vehicle staying in the middle of the road may be abnormal, but in a designated parking area it is a normal behavior. The existing method lacks effective modeling ability for scene semantic information, and it is difficult to handle such scene-dependent anomalies, resulting in limited detection accuracy.
[0006] The above-mentioned video anomaly detection methods generally have the following technical deficiencies:
[0007] It mainly models in pixel space or single target feature space, lacking effective expression of high-level semantic features;
[0008] Insufficient utilization of scene context information makes it difficult to handle scene-dependent abnormal events;
[0009] Some methods have limited input information and ignore appearance features or anomalies of non-human targets, resulting in a narrow range of applications;
[0010] Background changes or environmental disturbances can easily affect model stability, and detection robustness needs to be improved. Summary of the Invention
[0011] The purpose of this invention is to provide a video anomaly detection method based on scene condition normalized stream, which aims to address the aforementioned technical deficiencies in the prior art.
[0012] The technical solution for achieving the objective of this invention is a video anomaly detection method based on scene-condition-normalized streams, the method comprising the following steps:
[0013] Step A: Process the continuous video frame sequence to obtain the foreground target sequence, and complete the modeling of the foreground target region;
[0014] Step B: Divide the foreground target sequence into blocks to obtain a block-level feature sequence, and extract the corresponding temporal and spatial features;
[0015] Step C: Embed the temporal and spatial features into the block-level feature sequence and overlay them to obtain complete spatiotemporal structural information features;
[0016] Step D: Input the complete spatiotemporal structural information features into the encoding network, and jointly model the correlation of the foreground target in the time and space dimensions to obtain the foreground features of high-level semantics;
[0017] Step E: Extract the overall scene information corresponding to the foreground target from the original video frames, and extract features from the relatively stable environmental structures in the video to obtain scene features;
[0018] Step F: Perform conditional probability mapping modeling on scene features;
[0019] Step G: Map the foreground target sequence extracted from the video to be detected to the latent space, analyze the probability value or log-likelihood value under the corresponding scene conditions, and determine whether the corresponding foreground target or video is an abnormal event.
[0020] A further preferred embodiment is: in step A, using a continuous video frame sequence As input, where Indicates the first A video frame at a given moment;
[0021] The video frame sequence is processed by a target detection module to detect and crop out the foreground target region from each frame, thereby obtaining a continuous foreground target image sequence in the time dimension. ,in Indicates the first The foreground target image corresponding to the frame.
[0022] A further preferred method is: In step B, each frame of the foreground target image is segmented into several fixed-size image blocks, and each image block is mapped into a vector form, thereby constructing a block-level feature sequence representation of the foreground target.
[0023]
[0024] In order to preserve the spatial structure and temporal sequence information of the foreground objects in the subsequent modeling process.
[0025] A further preferred embodiment is: in step C, spatial location embedding and temporal embedding are introduced into the block-level features, and these are superimposed with the block features to obtain a feature representation containing complete spatiotemporal structural information.
[0026]
[0027] in, Used to represent the spatial relationship of image patches within a foreground target. Used to represent the sequential relationship of image patches in a time series.
[0028] A further preferred approach is as follows: In step D, the enhanced foreground features are input into a Transformer encoding network that combines one-dimensional temporal modeling and two-dimensional spatial modeling. A self-attention mechanism is used to jointly model the correlation between the foreground target in the temporal and spatial dimensions, resulting in a high-level semantic representation of the foreground features.
[0029]
[0030] This stage can effectively capture the movement patterns, appearance changes, and relationships between different spatial regions of the foreground target over a continuous period of time, thereby forming a comprehensive spatiotemporal representation of normal behavior patterns.
[0031] A further preferred method is: In step E, the overall scene information corresponding to the foreground target is extracted from the original video frame, and the scene image is fed into the scene coding network as input to extract features from the relatively stable environmental structure in the video, thereby obtaining the scene feature representation.
[0032] .
[0033] A further preferred approach is to perform conditional probability mapping modeling on the foreground features in step F. Specifically, using a Transformer Flow structure composed of multiple reversible transformations, a modulation mechanism conditioned on scene features is introduced in each transformation layer. This ensures that the foreground features are constrained by the corresponding scene information during the mapping process. The overall mapping process can be represented as follows:
[0034]
[0035] in, This represents an invertible mapping function under specific scenario conditions. The feature representation mapped to the latent probability space;
[0036] In the latent space, it is assumed that under given scenario conditions, normal behavioral characteristics follow a pre-defined probability distribution model, and a Gaussian mixture model is used to model them:
[0037]
[0038] in, , and They represent the first The weights, mean, and covariance matrices of each Gaussian component.
[0039] A further preferred embodiment is that in step G, when the probability value is lower than a preset threshold or its deviation from the normal distribution exceeds a preset range, the corresponding foreground target or video segment is determined to be an abnormal event.
[0040] Compared with existing technologies, this invention has the following positive effects: By using probabilistic modeling, the spatiotemporal behavior modeling of the foreground target is jointly constrained with scene environment information, which effectively avoids the misjudgment problem caused by the same behavior being uniformly modeled in different scenes in existing methods. The overall method is simple and can effectively reduce the interference caused by background areas and irrelevant pixel information.
[0041] Furthermore, it can extract higher-level feature representations with more semantic consistency, thereby improving the stability of anomaly detection in complex backgrounds and dynamic scenes;
[0042] Meanwhile, it can maintain a relatively consistent anomaly judgment standard when facing changes in scene, differences in regional semantics, or different environmental structures, effectively enhancing the detection robustness under cross-scene conditions.
[0043] Furthermore, anomaly detection is based on the degree of deviation from the conditional probability distribution, avoiding the instability caused by relying solely on reconstruction errors or simple distance metrics, thus enabling better continuity and consistency of anomaly detection results under different discrimination thresholds. Attached Figure Description
[0044] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:
[0045] Figure 1 This is a structural block diagram of the present invention.
[0046] In the attached diagram, OD stands for Object Detection; Patchify for Patch Partitioning; Position Embedding for Position Embedding; Temporal Embedding for Temporal Embedding; 1D+2D Transformer for 1D+2D Transformer module; Squeeze for Dimension Compression; Cross Attenuation for Cross Attention module; Split for Dimension Separation; GMM for Gaussian Mixture Model; and Transformer Flow for Transformer-based Normalized Flow Model. Detailed Implementation
[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0048] Example
[0049] See Figure 1 As shown, a video anomaly detection method based on scene-conditional normalized streams includes the following steps:
[0050] Step A: Process the continuous video frame sequence to obtain the foreground target sequence, and complete the modeling of the foreground target region; wherein, in Step A, the continuous video frame sequence As input, where Indicates the first A video frame at a given moment;
[0051] The video frame sequence is processed by a target detection module to detect and crop out the foreground target region from each frame, thereby obtaining a continuous foreground target image sequence in the time dimension. ,in Indicates the first The foreground target image corresponding to the frame.
[0052] Step B: The foreground target sequence is segmented to obtain a block-level feature sequence, and the corresponding temporal and spatial features are extracted. In Step B, each frame of the foreground target image is segmented into several fixed-size image blocks, and each image block is mapped into a vector form, thereby constructing a block-level feature sequence representation of the foreground target.
[0053]
[0054] In order to preserve the spatial structure and temporal sequence information of the foreground objects in the subsequent modeling process.
[0055] Step C: Embed the temporal and spatial features into the block-level feature sequence and superimpose them to obtain complete spatiotemporal structural information features; in step C, spatial location embedding and temporal embedding are introduced into the block-level features, and they are superimposed with the block features to obtain a feature representation containing complete spatiotemporal structural information:
[0056]
[0057] in, Used to represent the spatial relationship of image patches within a foreground target. Used to represent the sequential relationship of image patches in a time series.
[0058] This approach enables the model to simultaneously perceive the spatial distribution and temporal evolution characteristics of foreground targets during subsequent processing.
[0059] Step D: Input the complete spatiotemporal structural information features into the encoding network, and jointly model the correlation of the foreground target in the temporal and spatial dimensions to obtain the foreground features with high-level semantics; In Step D, the enhanced foreground features are input into the Transformer encoding network that combines one-dimensional temporal modeling and two-dimensional spatial modeling, and the correlation of the foreground target in the temporal and spatial dimensions is jointly modeled through a self-attention mechanism to obtain the foreground feature representation with high-level semantics:
[0060]
[0061] This stage can effectively capture the movement patterns, appearance changes, and relationships between different spatial regions of the foreground target over a continuous period of time, thereby forming a comprehensive spatiotemporal representation of normal behavior patterns.
[0062] Step E: Extract the overall scene information corresponding to the foreground target from the original video frames, and perform feature extraction on the relatively stable environmental structures in the video to obtain scene features; In Step E, extract the overall scene information corresponding to the foreground target from the original video frames, feed the scene image as input into the scene coding network, and perform feature extraction on the relatively stable environmental structures in the video to obtain scene feature representation:
[0063] .
[0064] To ensure that the scene features can stably and accurately reflect the video environment information, the present invention also uses a scene decoding network to reconstruct and constrain the scene features, so that the learned scene features can maintain discriminative ability while having good expressive consistency.
[0065] In obtaining high-level characteristics of the prospect With scene features Subsequently, this invention introduces a Transformer-based normalized flow model. Step F involves conditional probability mapping modeling of scene features; specifically, step F involves conditional probability mapping modeling of foreground features. Through a Transformer Flow structure composed of multiple reversible transformations, a modulation mechanism conditioned on scene features is introduced in each transformation layer, so that the foreground features are constrained by corresponding scene information during the mapping process. The overall mapping process can be expressed as follows:
[0066]
[0067] in, This represents an invertible mapping function under specific scenario conditions. This is a feature representation mapped to the latent probability space; in this way, normal behavioral features in different scenarios are mapped to different latent distribution regions, thereby effectively distinguishing scenario-dependent behavioral patterns.
[0068] In the latent space, it is assumed that under given scenario conditions, normal behavioral characteristics follow a pre-defined probability distribution model, and a Gaussian mixture model is used to model them:
[0069]
[0070] in, , and They represent the first The weights, mean, and covariance matrices of each Gaussian component.
[0071] Step G: Map the foreground target sequence extracted from the video to be detected to the latent space, analyze the probability value or log-likelihood value under the corresponding scene conditions, and determine whether the corresponding foreground target or video is an anomalous event. In Step G, when the probability value is lower than a preset threshold or its deviation from the normal distribution exceeds a preset range, the corresponding foreground target or video segment is determined to be an anomalous event.
[0072] Through a series of comparative experiments, two metrics, micro-AUROC and macro-AUROC, were used to evaluate the model's performance. Micro-AUROC, by uniformly statistically analyzing all test samples, measures the model's overall ability to distinguish between abnormal and normal events at different discrimination thresholds. Macro-AUROC, on the other hand, calculates the detection performance separately for different scenarios and averages it to reflect the model's detection stability and consistency under cross-scenario conditions, providing a more intuitive representation of its ability to identify scene-dependent anomalies. Experimental results show that the method of this invention achieves detection results of 92.7% micro-AUROC and 91.4% macro-AUROC on the Avenue dataset, and 80.2% micro-AUROC and 87.3% macro-AUROC on the ShanghaiTech dataset. Overall, its performance outperforms most comparative methods that do not explicitly incorporate scene modeling.
[0073] On the NWPU-Campus dataset, where anomaly detection is highly dependent on environmental semantics and the same behavior may have different attributes in different scenarios, the macro-AUROC of the method in this invention reaches 76.4%, which is significantly better than existing methods. This indicates that by introducing scene-related conditional constraints and constructing a scene-related probability distribution of normal behavior, the detection capability of scene-dependent anomalies can be effectively improved. The overall experimental results verify the effectiveness, stability, and cross-scene adaptability of the method in complex monitoring scenarios.
[0074] Compared with existing technologies, this invention has the following positive effects: By using probabilistic modeling, the spatiotemporal behavior modeling of the foreground target is jointly constrained with scene environment information, which effectively avoids the misjudgment problem caused by the same behavior being uniformly modeled in different scenes in existing methods. The overall method is simple and can effectively reduce the interference caused by background areas and irrelevant pixel information.
[0075] Furthermore, it can extract higher-level feature representations with more semantic consistency, thereby improving the stability of anomaly detection in complex backgrounds and dynamic scenes;
[0076] Meanwhile, it can maintain a relatively consistent anomaly judgment standard when facing changes in scene, differences in regional semantics, or different environmental structures, effectively enhancing the detection robustness under cross-scene conditions.
[0077] Furthermore, anomaly detection is based on the degree of deviation from the conditional probability distribution, avoiding the instability caused by relying solely on reconstruction errors or simple distance metrics, thus enabling better continuity and consistency of anomaly detection results under different discrimination thresholds.
[0078] The standard parts used in this embodiment can be purchased directly from the market, and the non-standard structural parts described in the instruction manual can also be processed without any doubt based on existing technical common sense. At the same time, the connection methods of each component adopt mature conventional methods in the existing technology, and the machinery, parts and equipment all adopt conventional models in the existing technology, so they will not be described in detail here.
[0079] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, these obvious variations or modifications derived from the essential spirit of the present invention still fall within the scope of protection of the present invention.
Claims
1. A video anomaly detection method based on scene-conditional normalized stream, characterized in that: The method includes the following steps: Step A: Process the continuous video frame sequence to obtain the foreground target sequence, and complete the modeling of the foreground target region; Step B: Divide the foreground target sequence into blocks to obtain a block-level feature sequence, and extract the corresponding temporal and spatial features; Step C: Embed the temporal and spatial features into the block-level feature sequence and overlay them to obtain complete spatiotemporal structural information features; Step D: Input the complete spatiotemporal structural information features into the encoding network, and jointly model the correlation of the foreground target in the time and space dimensions to obtain the foreground features of high-level semantics; Step E: Extract the overall scene information corresponding to the foreground target from the original video frames, and extract features from the relatively stable environmental structures in the video to obtain scene features; Step F: Perform conditional probability mapping modeling on scene features; Step G: Map the foreground target sequence extracted from the video to be detected to the latent space, analyze the probability value or log-likelihood value under the corresponding scene conditions, and determine whether the corresponding foreground target or video is an abnormal event.
2. The video anomaly detection method based on scene-conditional normalized stream according to claim 1, characterized in that: In step A, a continuous video frame sequence As input, where Indicates the first A video frame at a given moment; The video frame sequence is processed by a target detection module to detect and crop out the foreground target region from each frame, thereby obtaining a continuous foreground target image sequence in the time dimension. ,in Indicates the first The foreground target image corresponding to the frame.
3. The video anomaly detection method based on scene condition normalized stream according to claim 2, characterized in that: In step B, each frame of the foreground target image is divided into several fixed-size image blocks, and each image block is mapped into a vector form, thereby constructing a block-level feature sequence representation of the foreground target: In order to preserve the spatial structure and temporal sequence information of the foreground objects in the subsequent modeling process.
4. The video anomaly detection method based on scene condition normalized stream according to claim 3, characterized in that: In step C, spatial location embedding and temporal embedding are introduced into the block-level features, and these are superimposed with the block features to obtain a feature representation containing complete spatiotemporal structural information: in, Used to represent the spatial relationship of image patches within a foreground target. Used to represent the sequential relationship of image patches in a time series.
5. The video anomaly detection method based on scene condition normalized stream according to claim 4, characterized in that: In step D, the enhanced foreground features are input into a Transformer encoding network that combines one-dimensional temporal modeling and two-dimensional spatial modeling. A self-attention mechanism is used to jointly model the correlation between the foreground target in the temporal and spatial dimensions, resulting in a high-level semantic representation of the foreground features. This stage can effectively capture the movement patterns, appearance changes, and relationships between different spatial regions of the foreground target over a continuous period of time, thereby forming a comprehensive spatiotemporal representation of normal behavior patterns.
6. The video anomaly detection method based on scene condition normalized stream according to claim 5, characterized in that: In step E, the overall scene information corresponding to the foreground target is extracted from the original video frames. The scene image is then fed into the scene coding network as input to extract features from the relatively stable environmental structures in the video, resulting in a scene feature representation. .
7. The video anomaly detection method based on scene condition normalized stream according to claim 6, characterized in that: In step F, conditional probability mapping modeling is performed on the foreground features. Specifically, a Transformer Flow structure consisting of multiple reversible transformations is used. In each transformation layer, a modulation mechanism conditioned on scene features is introduced, so that the foreground features are constrained by the corresponding scene information during the mapping process. The overall mapping process can be represented as follows: ,in, This represents an invertible mapping function under specific scenario conditions. The feature representation mapped to the latent probability space; In the latent space, it is assumed that under given scenario conditions, normal behavioral characteristics follow a pre-defined probability distribution model, and a Gaussian mixture model is used to model them: in, , and They represent the first The weights, mean, and covariance matrices of each Gaussian component.
8. A video anomaly detection method based on scene condition normalized stream according to claim 7, characterized in that: In step G, when the probability value is lower than a preset threshold or its deviation from the normal distribution exceeds a preset range, the corresponding foreground target or video segment is determined to be an abnormal event.