A scene-dependent video anomaly detection method and device

Abstract

The application relates to a scene-dependent video anomaly detection method and device. The method comprises: acquiring events and scene features. After feature preprocessing, random disturbance is applied to the scene features, and the event features are updated through time dimension self-attention. Modulation parameters are generated through a scene condition fusion block, the event features are modulated and fused, and residual connection is performed, a scene condition classification head is input, and discriminative learning is performed with a camera ID as a label. Meanwhile, the features are processed for comparative learning, positive and negative sample pairs are constructed in combination with a memory bank, and bidirectional symmetric comparative learning is performed. Two types of anomaly scores are calculated, adaptive fusion is performed based on distribution difference and local correlation, the maximum fusion score of each frame is taken as a frame-level score, and output is performed after median filtering and smoothing, so that accurate anomaly detection is realized. The method can improve the detection accuracy and practicability in a complex multi-scene environment.

Description

Technical Field

[0001] This application relates to the field of video anomaly detection technology, and in particular to a scene-dependent video anomaly detection method and apparatus. Background Technology

[0002] With the acceleration of urbanization and the increasing demand for public safety, intelligent video surveillance technology is being applied more and more widely in fields such as urban security, campus security, and traffic monitoring. Video anomaly detection, as a core function of intelligent video surveillance, aims to automatically identify abnormal events that violate normal behavioral patterns from massive amounts of surveillance video, providing timely early warnings for security and prevention. Due to the low frequency and rarity of abnormal events, most existing video anomaly detection methods adopt a semi-supervised learning paradigm, using only a large number of normal videos to train a "normality" model. During the inference phase, anomalies are determined by judging the degree of deviation between the video to be detected and the normal pattern. Traditional methods are mostly focused on single-scene or scene-independent detection tasks. A typical approach is to use reconstruction or predictive generative models, calculating anomaly scores based on reconstruction errors or prediction errors.

[0003] However, significant scene dependence exists in real-world surveillance scenarios—the normality of the same behavior varies drastically across different scenes. For example, running is normal in a playground but abnormal in a bank lobby; cycling on a campus road is reasonable but abnormal in a teaching building corridor. With the emergence of scene-dependent video anomaly detection datasets such as NWPUCampus and ShanghaiTech-sd, researchers have begun to incorporate scene information into generative frameworks to improve detection performance through conditional generation. However, existing technologies still have significant shortcomings: in multi-camera surveillance systems, each camera's unique ID naturally corresponds to a fixed shooting scene and can serve as a scene category label. However, existing methods only use scene images or features as conditional inputs to the generative model, without using scene labels as supervisory signals for discriminative learning, resulting in insufficient utilization of scene information; existing generative methods often focus on pixel-level or feature-level reconstruction / prediction errors, lacking semantic constraints on whether events and scenes match, making it difficult to characterize the core essence of scene-dependent anomalies; existing solutions often employ a single generative paradigm, failing to simultaneously consider classification semantic supervision and fine-grained relationship modeling, leading to limited detection robustness. Summary of the Invention

[0004] Therefore, it is necessary to provide a scene-dependent video anomaly detection method and device that can improve the detection accuracy and practicality in complex and multi-scene environments, addressing the aforementioned technical problems.

[0005] A scene-dependent video anomaly detection method, the method comprising:

[0006] Each frame of the multi-camera surveillance video is preprocessed, and a set of foreground target boxes is obtained through target detection and optical flow estimation. Based on the set of foreground target boxes, a spatiotemporal cube corresponding to each foreground target is constructed. The spatiotemporal cube and the whole frame scene image of the current frame are input into the visual feature extraction model to obtain event features and scene features.

[0007] The event features and scene features are preprocessed, and random perturbation is applied to the preprocessed scene features. Then, a time-dimensional self-attention operation is performed on the preprocessed event features to obtain the updated event features.

[0008] The scene conditional fusion block is constructed to map the perturbed scene features to modulation parameters. The modulation parameters are used to conditionally modulate the updated event features. The modulated event features and the updated event features are weighted and fused together, and then residually connected with the preprocessed event features to obtain the fused event features.

[0009] The fused event features are input into the scene condition classification head, and discriminative learning is performed using the camera ID as the scene label to obtain discriminative correlation output.

[0010] The preprocessed event features are subjected to temporal average pooling and then combined with scene features for feature transformation and normalization to obtain event features and scene features for contrastive learning. Based on scene labels, event-scene positive and negative sample pairs are constructed, and more positive and negative sample pairs are constructed by combining historical features stored in the memory bank. Bidirectional symmetric contrastive learning of event-scene and scene-event is carried out to obtain contrastive correlation output.

[0011] Based on discriminant correlation output and contrastive correlation output, discriminant anomaly score and contrastive anomaly score are calculated respectively; based on the distribution difference and local correlation of the two types of anomaly scores, event-level fused anomaly score is calculated through an adaptive fusion function.

[0012] The maximum fused anomaly score of all events in each frame of video is taken as the frame-level anomaly score. The frame-level anomaly score sequence is smoothed by sliding window mid-range filtering to obtain the final frame-level anomaly score sequence, thus realizing video anomaly detection.

[0013] A scene-dependent video anomaly detection device, the device comprising:

[0014] The event and scene feature extraction module is used to preprocess each frame of the multi-camera surveillance video. It obtains a set of foreground target boxes through target detection and optical flow estimation, constructs a spatiotemporal cube corresponding to each foreground target based on the set of foreground target boxes, and inputs the spatiotemporal cube and the whole frame scene image of the current frame into the visual feature extraction model to obtain event features and scene features.

[0015] The discriminative learning module preprocesses event features and scene features, applies random perturbations to the preprocessed scene features, and performs a time-dimensional self-attention operation on the preprocessed event features to obtain updated event features. A scene conditional fusion block is constructed to map the perturbed scene features to modulation parameters. These modulation parameters are used to conditionally modulate the updated event features. The modulated event features and the updated event features are then weighted and fused, and a residual connection is established with the preprocessed event features to obtain fused event features. These fused event features are then input into a scene conditional classification head, and discriminative learning is performed using the camera ID as the scene label to obtain discriminative correlation output.

[0016] The contrastive learning module is used to perform temporal average pooling on the preprocessed event features and to perform feature transformation and normalization together with the scene features to obtain event features and scene features for contrastive learning. Based on the scene labels, event-scene positive and negative sample pairs are constructed, and more positive and negative sample pairs are constructed by combining the historical features stored in the memory bank. Bidirectional symmetric contrastive learning of event-scene and scene-event is carried out to obtain contrastive correlation output.

[0017] The anomaly fusion scoring module is used to calculate discriminative anomaly scores and contrastive anomaly scores based on discriminative correlation output and contrastive correlation output, respectively. Based on the distribution differences and local correlations of the two types of anomaly scores, an event-level fusion anomaly score is calculated through an adaptive fusion function. The maximum fusion anomaly score of all events in each frame of video is taken as the frame-level anomaly score. The frame-level anomaly score sequence is smoothed by sliding window mid-range filtering to obtain the final frame-level anomaly score sequence, thereby realizing video anomaly detection.

[0018] The aforementioned scene-dependent video anomaly detection method and apparatus innovatively combine object detection and optical flow estimation to obtain a set of foreground object bounding boxes. It accurately captures foreground objects of known categories using YOLOv13 and supplements unidentified motion regions with SEA-RAFT, ensuring no foreground localization is missed. The construction of a spatiotemporal cube transforms the continuous motion process of a single foreground object into a processable event unit. Combined with event features and scene features extracted by the DINOv2 model, it provides high-quality input rich in spatiotemporal information and scene context for subsequent learning, guaranteeing detection accuracy from the source. The discriminative learning stage explicitly utilizes the camera ID as a scene label, upgrading scene information from a simple conditional input to a supervisory signal. Combined with a scene conditional fusion block and a classification head, the model directly learns the classification relationship of whether an event belongs to the current scene. Compared to generative methods that rely solely on scene images, this approach more fully leverages the value of scene information, significantly improving the ability to distinguish scene-dependent anomalies. Meanwhile, random perturbations of scene features and self-attention operations on event features further enhance the model's adaptability to scene changes and its ability to model short-term motion patterns, making classification judgments more robust. Positive and negative sample pairs are constructed through scene labels, and a memory bank is introduced to expand sample diversity. The bidirectional contrastive learning mechanism makes the model more sensitive to fine-grained behavioral differences in complex scenes, effectively overcoming the limitations of traditional methods that only focus on pixel or feature-level errors, further improving the accuracy of anomaly detection. Finally, the adaptive fusion function dynamically adjusts the weights based on the distribution differences and local correlations of the two types of scores, avoiding the limitations of fixed fusion methods and making event-level anomaly scores more consistent with actual scenarios. Taking the maximum fusion score within a frame as the frame-level score ensures that no abnormal events are missed; sliding window midpoint filtering effectively removes noise, making the detection results more stable and reliable. Furthermore, the modular design allows this application to flexibly adapt to different visual backbone networks and target detectors, facilitating upgrades and deployments in existing monitoring systems, significantly improving the practicality and scalability of the technology. The collaborative optimization of multiple stages ultimately achieved a dual improvement in detection accuracy and robustness, making it more valuable for application in complex and multi-scenario environments. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a scene-dependent video anomaly detection method in one embodiment;

[0020] Figure 2 This is a block diagram of a scene-dependent video anomaly detection method in one embodiment;

[0021] Figure 3 This is a structural block diagram of a scene-dependent video anomaly detection device in one embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] In one embodiment, such as Figure 1 and Figure 2 As shown, a scene-dependent video anomaly detection method is provided, including the following steps:

[0024] Step 102: Preprocess each frame of the multi-camera surveillance video, obtain a set of foreground target boxes through target detection and optical flow estimation, construct a spatiotemporal cube corresponding to each foreground target based on the set of foreground target boxes, and input the spatiotemporal cube and the whole frame scene image of the current frame into the visual feature extraction model to obtain event features and scene features.

[0025] The YOLOv13 real-time object detector is used to detect objects in each frame of the video. Object boxes with too low confidence or too high overlap are filtered out based on confidence and overlap thresholds to obtain the final object detection boxes. Since the target detector is limited by a fixed category, motion compensation is performed using a SEA-RAFT optical flow estimator. A binary motion map is obtained based on a gradient threshold, and regions covered by the target bounding box are removed to avoid double counting. Furthermore, pseudo-motion blocks with excessively small areas or aspect ratios exceeding a specified threshold are discarded to obtain the final optical flow motion map. Finally, we will use the object detection bounding box. And optical flow motion frame By merging the results, we obtain a set of foreground target boxes for each video frame. For each foreground target bounding box, we crop its corresponding image patch from the current frame and place it in front of the current frame. Image blocks are cropped from the same position in the frame, and all image blocks are stacked sequentially to a length of [length missing]. The spatiotemporal cube is used as a local video segment corresponding to a single event in a video frame. Then, the extracted spatiotemporal cube corresponding to the video frame event and the current video frame scene are input into the frozen visual feature extraction model DINOv2 to extract the corresponding event features. and scene features Event features and scene features will serve as common inputs for both discriminative and contrastive learning.

[0026] Multi-camera surveillance video contains multiple fixed shooting scenes, with each camera corresponding to a unique scene label (camera ID). The core purpose of preprocessing is to accurately locate foreground targets from complex backgrounds—that is, moving targets that need to be monitored, such as pedestrians and vehicles—and to model the continuous motion process of a single foreground target as an "event." Target detection is used to quickly identify foreground targets of known categories, while optical flow estimation is used to supplement the blind spots of target detection, capturing unidentified but moving target regions. The combination of the two can improve the completeness of foreground localization. The spatiotemporal cube is a feature carrier formed by stacking local image patches of a single foreground target in a continuous T frames in chronological order, which can completely represent the spatiotemporal motion information of the target; the visual feature extraction model is used to convert the spatiotemporal cube in image form and the whole-frame scene image into vector-form features that can be processed by computers—event features and scene features.

[0027] Step 104: Preprocess the event features and scene features, apply random perturbation to the preprocessed scene features, and perform a time-dimensional self-attention operation on the preprocessed event features to obtain the updated event features.

[0028] Event characteristics and scene features Spatial average pooling is performed separately, followed by uniform linear transformation, normalization, and activation processing to obtain the processed event features. and scene features To avoid the classifier relying too heavily on static scene appearance, scene features are... Apply random perturbation. Specifically, sample random numbers from the interval (0,1). Calculate the scene features after perturbation Regarding the characteristics of the event In the time dimension Several layers of self-attention operations are applied to enhance the modeling ability of short-term motion patterns, resulting in updated event features. .

[0029] The purpose of feature preprocessing is to unify feature dimensions and standardize feature distribution, providing high-quality input for subsequent learning processes. This includes operations such as spatial average pooling, linear transformation, normalization, and activation processing. Spatial average pooling reduces feature dimensionality while retaining key information, linear transformation unifies feature dimensions, and normalization and activation processing enhance the expressive power of features. Applying random perturbations to scene features avoids the classifier's over-reliance on static scene appearance and improves the model's generalization ability; this is achieved by introducing a random scaling factor. Self-attention allows the model to automatically focus on key temporal correlations of event features, enhancing its ability to model short-term motion patterns and thus more accurately capturing the motion patterns of foreground targets.

[0030] Step 106: Construct a scene conditional fusion block to map the perturbed scene features to modulation parameters, use the modulation parameters to conditionally modulate the updated event features, weight and fuse the modulated event features with the updated event features, and then perform residual connection with the preprocessed event features to obtain fused event features.

[0031] In the discriminant network, multiple scene condition fusion blocks are set up, and in each fusion block:

[0032] 1) Using perturbed scene embedding The parameters are mapped to several modulation parameters by a multilayer perceptron (MLP).

[0033] 2) Apply the modulation parameters to the AdaLN-Zero style attention branch and feedforward network branch respectively to achieve conditional modulation of event features, thereby obtaining modulated event features. ;

[0034] 3) Update event features after self-attention operation using learnable sample-aware fusion weights. Modulated event characteristics Perform weighted fusion and combine with initial event features Perform residual connection.

[0035] The scene conditional fusion block is the core structure for achieving deep fusion of scene information and event features. By mapping scene features to modulation parameters, scene information can regulate the processing of event features, imbuing them with scene attributes. Weighted fusion uses learnable sample-aware fusion weights to dynamically balance the importance of event features updated through self-attention and those modulated. Residual connections alleviate the vanishing gradient problem in deep network training, preserving effective information from the initial features. The resulting fused event features contain both the event's own motion information and the scene's constraint information.

[0036] Step 108: Input the fused event features into the scene condition classification head, use the camera ID as the scene label for discriminative learning, and obtain discriminative correlation output.

[0037] In a multi-camera surveillance system, the IDs of different cameras are used as scene category labels. A scene-conditional discriminant network is used, with event features and scene features jointly input. Classification learning is performed directly using the scene labels as the supervised objective, and the cross-entropy loss of the classification labels is used as the discriminant loss function.

[0038] ;

[0039] Labeling real-world scenarios The corresponding predicted distribution probability.

[0040] The scene conditional classification head is a network structure specifically designed for scene classification tasks. It is also optimized through modulation parameters of scene feature mapping to ensure a close correlation between the classification process and scene information. Camera IDs, as natural scene category labels, provide explicit supervision signals for discriminative learning. The core of discriminative learning is to teach the model whether "the current event belongs to the scene corresponding to its respective camera." By continuously optimizing model parameters through a classification loss function, the model can accurately distinguish the matching relationship between events and scenes. The discriminative correlation output is specifically the predicted probability distribution of events belonging to each scene category.

[0041] Step 110: Perform time-averaged pooling on the preprocessed event features, and perform feature transformation and normalization together with the scene features to obtain event features and scene features for contrastive learning; construct event-scene positive and negative sample pairs based on scene labels, and construct more positive and negative sample pairs by combining historical features stored in the memory bank, and perform bidirectional symmetric contrastive learning of event-scene and scene-event to obtain contrastive correlation output.

[0042] Contrastive deep neural network module: This module integrates event features. In the time dimension Time-averaged pooling is performed on the data, and then combined with scene features. The processed event features are obtained separately through a multilayer perceptron and a single layer. and scene features 。 Based on the scene labels of event features and scene features, positive and negative sample pairs are defined: if the scene labels of the event features and scene features are the same, they constitute a positive sample pair in contrastive learning; if the scene labels are different, they constitute a negative sample pair. A similarity function based on cosine similarity is defined:

[0043] ;

[0044] in, This refers to the temperature parameter.

[0045] Employs symmetrical contrast loss in two directions: event->scene and scene->event.

[0046] 1) For the event->scene direction, maximize the similarity of each event with scenes with the same label and minimize the similarity with scenes with different labels;

[0047] 2) For the scene-to-event direction, maximize the similarity between each scene and events with the same label, and minimize the similarity with events with different labels. The average of these two values ​​yields the overall contrastive loss function. .

[0048] To enrich the comparison samples, this application establishes a memory bank to store historical event features, scene features, and their labels. During each training iteration, the features of the current batch are concatenated with the features in the memory bank to construct more positive and negative sample pairs. After training, a first-in, first-out (FIFO) strategy is used to update the memory bank, ensuring that it continuously retains representative historical samples within a given capacity.

[0049] The contrastive loss function is:

[0050] ;

[0051] and These are the bidirectional comparison losses for event->scene and scene->event, respectively.

[0052] Temporal average pooling is used to aggregate event features over time, resulting in a more stable event representation. Feature transformation is achieved through a multilayer perceptron, mapping the original features to a feature space suitable for contrastive learning. Normalization ensures the consistency of feature distribution, preventing differences in feature scale from affecting the contrastive learning effect. The resulting contrastive learning event features and scene features can be used more accurately to calculate the matching similarity between events and scenes.

[0053] The construction of positive and negative sample pairs is the foundation of contrastive learning. Event features and scene features under the same scene label constitute positive sample pairs (representing a "matching" relationship), while features under different scene labels constitute negative sample pairs (representing a "mismatching" relationship). The introduction of a memory bank is to expand the sample size and solve the problem of insufficient positive and negative samples in single-batch training. By storing features and labels from the historical training process, richer positive and negative sample pairs can be constructed in each training session, improving the stability and effectiveness of learning. Bidirectional symmetric contrastive learning trains simultaneously from two dimensions: "event matching scene" and "scene matching event," allowing the model to more comprehensively learn the matching rules between events and scenes. The contrastive correlation output is specifically the similarity value between events and scenes.

[0054] Step 112: Calculate the discriminant anomaly score and the contrastive anomaly score based on the discriminant correlation output and the contrastive correlation output, respectively; calculate the event-level fused anomaly score based on the distribution differences and local correlations of the two types of anomaly scores using an adaptive fusion function.

[0055] For any video frame, the th Event characteristics Discriminant anomaly scores were obtained respectively. Comparative anomaly scores Based on the distributional differences of the two types of scores across the entire test sequence (measured by a simplified KL divergence) and the correlation between the two types of scores within the local neighborhood of the same event (e.g., Pearson correlation coefficient), a pre-defined adaptive fusion function is employed. Calculate the event-level final fusion anomaly score:

[0056]

[0057] Discriminative anomaly scores are calculated based on the classification results of discriminative learning. A high discriminative anomaly score indicates a low match between the event and its surrounding scene. Contrastive anomaly scores are calculated based on the similarity between the event and its scene. A high contrastive anomaly score indicates a low similarity between the two. These two types of anomaly scores reflect the degree of anomaly from different dimensions: discriminative anomaly scores focus on "category attribution," while contrastive anomaly scores focus on "feature matching." Combining the two provides a more comprehensive characterization of anomalous features.

[0058] Because the effectiveness of the two types of anomaly scores varies across different scenarios, direct weighted fusion is unlikely to achieve optimal results. Distributional differences are assessed using a simplified KL divergence metric to determine the overall consistency between the two types of scores; local correlation is measured using the Pearson correlation coefficient to determine the degree of synchronous change in the two types of scores within a local time window. The adaptive fusion function dynamically adjusts the fusion weights of the two types of anomaly scores based on these two indicators, making the fusion result more closely reflect the actual situation of the current scenario. The resulting event-level fused anomaly score can comprehensively reflect the degree of anomaly of the event.

[0059] Step 114: Take the maximum fused anomaly score of all events in each video frame as the frame-level anomaly score, and smooth the frame-level anomaly score sequence by sliding window mid-range filtering to obtain the final frame-level anomaly score sequence, thereby realizing video anomaly detection.

[0060] For each video frame, the maximum fusion score of all event features in that frame is taken as the frame-level anomaly score:

[0061]

[0062] Finally, a sliding window smoothing based on median filtering is applied to the frame-level anomaly score sequence to eliminate isolated noise points, resulting in a smoothed frame-level anomaly score sequence for final video frame anomaly assessment.

[0063] A single frame of video may contain multiple events. The maximum fused anomaly score is taken as the frame-level anomaly score because if a frame contains even one event with a high degree of anomaly, that frame must be classified as an anomaly frame, meeting the early warning requirements of real-world monitoring scenarios. A sliding window mid-range filter is used to eliminate isolated noise points in the frame-level anomaly score sequence, preventing misjudgments of individual events from distorting the overall detection results. This makes the final frame-level anomaly score sequence smoother and more reliable, enabling accurate detection and early warning of video anomalies.

[0064] The aforementioned scene-dependent video anomaly detection method and apparatus innovatively combine object detection and optical flow estimation to obtain a set of foreground object bounding boxes. It accurately captures foreground objects of known categories using YOLOv13 and supplements unidentified motion regions with SEA-RAFT, ensuring no foreground localization is missed. The construction of a spatiotemporal cube transforms the continuous motion process of a single foreground object into a processable event unit. Combined with event features and scene features extracted by the DINOv2 model, it provides high-quality input rich in spatiotemporal information and scene context for subsequent learning, guaranteeing detection accuracy from the source. The discriminative learning stage explicitly utilizes the camera ID as a scene label, upgrading scene information from a simple conditional input to a supervisory signal. Combined with a scene conditional fusion block and a classification head, the model directly learns the classification relationship of whether an event belongs to the current scene. Compared to generative methods that rely solely on scene images, this approach more fully leverages the value of scene information, significantly improving the ability to distinguish scene-dependent anomalies. Meanwhile, random perturbations of scene features and self-attention operations on event features further enhance the model's adaptability to scene changes and its ability to model short-term motion patterns, making classification judgments more robust. Positive and negative sample pairs are constructed through scene labels, and a memory bank is introduced to expand sample diversity. The bidirectional contrastive learning mechanism makes the model more sensitive to fine-grained behavioral differences in complex scenes, effectively overcoming the limitations of traditional methods that only focus on pixel or feature-level errors, further improving the accuracy of anomaly detection. Finally, the adaptive fusion function dynamically adjusts the weights based on the distribution differences and local correlations of the two types of scores, avoiding the limitations of fixed fusion methods and making event-level anomaly scores more consistent with actual scenarios. Taking the maximum fusion score within a frame as the frame-level score ensures that no abnormal events are missed; sliding window midpoint filtering effectively removes noise, making the detection results more stable and reliable. Furthermore, the modular design allows this application to flexibly adapt to different visual backbone networks and target detectors, facilitating upgrades and deployments in existing monitoring systems, significantly improving the practicality and scalability of the technology. The collaborative optimization of multiple stages ultimately achieved a dual improvement in detection accuracy and robustness, making it more valuable for application in complex and multi-scenario environments.

[0065] In one embodiment, preprocessing of event features and scene features, and applying random perturbation to the preprocessed scene features, includes:

[0066] The preprocessing of event features and scene features includes spatial average pooling, linear transformation, normalization, and activation processing performed sequentially; random perturbations of scene features are achieved by sampling random numbers from the interval (0,1). α The preprocessed scene features are combined with α Multiplying the results yields the perturbed scene characteristics.

[0067] In one embodiment, the target detection uses the YOLOv13 target detector, the optical flow estimation uses the SEA-RAFT optical flow estimator, and the visual feature extraction model is the DINOv2 model. In the target detection process, low-confidence and excessively overlapping target boxes are filtered according to the confidence threshold and the overlap threshold. In the optical flow estimation process, a binary motion map is obtained based on the gradient threshold. The area covered by the target box is removed and pseudo-motion blocks with an area lower than the preset threshold or an aspect ratio exceeding the threshold are eliminated. The target detection box and the optical flow motion box are merged to obtain a set of foreground target boxes.

[0068] In one embodiment, the discriminative anomaly score and the contrastive anomaly score are calculated based on the discriminative correlation output and the contrastive correlation output, respectively, including:

[0069] For any video frame, the th Event characteristics The discriminant anomaly score is calculated as follows:

[0070] ;

[0071] in, Labeling real-world scenarios The corresponding predicted distribution probability.

[0072] In one embodiment, event characteristics In the time dimension Time-averaged pooling is performed on the data, and then combined with scene features. The processed event features are obtained separately through a multilayer perceptron and a single layer. and scene features ;

[0073] The similarity of sample pairs is calculated based on the processed event features and scene features, and the comparative anomaly score is calculated using the sample pair similarity.

[0074] In one embodiment, a contrastive anomaly score is calculated using sample pair similarity, including:

[0075] The contrastive anomaly score is calculated using sample pair similarity:

[0076] ;

[0077] ;

[0078] in, Indicates the similarity between sample pairs. This refers to the temperature parameter.

[0079] In one embodiment, based on the distributional differences and local correlations of the two types of anomaly scores, an event-level fused anomaly score is calculated using an adaptive fusion function, including:

[0080] Based on the distributional differences and local correlations of the two types of outlier scores, an adaptive fusion function is used. The event-level fusion anomaly score is calculated as follows:

[0081] ;

[0082] in, Let i be the fusion anomaly score of the i-th event. Let be the discriminant anomaly score for the i-th event. Let be the comparative anomaly score of the i-th event.

[0083] In one embodiment, a scene conditional fusion block is constructed to map the perturbed scene features to modulation parameters. The updated event features are conditionally modulated using these modulation parameters. The modulated event features and the updated event features are then weighted and fused, and a residual connection is performed with the preprocessed event features to obtain fused event features, including:

[0084] Set up multiple scene condition fusion blocks, and use the perturbed scene features in each fusion block. The parameters are mapped through a multilayer perceptron and then converted into several modulation parameters.

[0085] The modulation parameters are applied to the attention branch and feedforward branch of the AdaLN-Zero style network respectively to achieve conditional modulation of the event features, resulting in the modulated event features. ;

[0086] Event features updated after self-attention operation are processed using learnable sample-aware fusion weights. Modulated event characteristics Perform weighted fusion and combine with initial event features Perform residual connections to obtain the fused event characteristics.

[0087] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0088] In one embodiment, such as Figure 3 As shown, a scene-dependent video anomaly detection device is provided, comprising: an event and scene feature extraction module 302, a discriminative learning module 304, a contrastive learning module 306, and an anomaly fusion scoring module 308, wherein:

[0089] The event and scene feature extraction module 302 is used to preprocess each frame of the multi-camera surveillance video, obtain a set of foreground target boxes through target detection and optical flow estimation, construct a spatiotemporal cube corresponding to each foreground target based on the set of foreground target boxes, and input the spatiotemporal cube and the whole frame scene image of the current frame into the visual feature extraction model to obtain event features and scene features;

[0090] The discriminative learning module 304 is used to preprocess event features and scene features, apply random perturbation to the preprocessed scene features, and perform a time-dimensional self-attention operation on the preprocessed event features to obtain updated event features. A scene conditional fusion block is constructed to map the perturbed scene features to modulation parameters. The updated event features are conditionally modulated using these modulation parameters. The modulated event features and the updated event features are then weighted and fused, and a residual connection is made with the preprocessed event features to obtain fused event features. The fused event features are input into the scene conditional classification head, and discriminative learning is performed using the camera ID as the scene label to obtain discriminative correlation output.

[0091] The contrastive learning module 306 is used to perform temporal average pooling on the preprocessed event features and to perform feature transformation and normalization together with the scene features to obtain event features and scene features for contrastive learning; it constructs event-scene positive and negative sample pairs based on scene labels, and constructs more positive and negative sample pairs by combining historical features stored in the memory bank, and performs bidirectional symmetric contrastive learning of event-scene and scene-event to obtain contrastive correlation output;

[0092] The anomaly fusion scoring module 308 is used to calculate discriminative anomaly scores and contrastive anomaly scores based on discriminative correlation output and contrastive correlation output, respectively; based on the distribution differences and local correlations of the two types of anomaly scores, an event-level fusion anomaly score is calculated through an adaptive fusion function; the maximum fusion anomaly score of all events in each frame of video is taken as the frame-level anomaly score, and the frame-level anomaly score sequence is smoothed by sliding window mid-range filtering to obtain the final frame-level anomaly scoring sequence, thereby realizing video anomaly detection.

[0093] For specific limitations regarding a scene-dependent video anomaly detection device, please refer to the limitations of a scene-dependent video anomaly detection method described above, which will not be repeated here. Each module in the aforementioned scene-dependent video anomaly detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0094] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A scene-dependent video anomaly detection method, characterized in that, The method includes: Each frame of the multi-camera surveillance video is preprocessed, and a set of foreground target boxes is obtained through target detection and optical flow estimation. Based on the set of foreground target boxes, a spatiotemporal cube corresponding to each foreground target is constructed. The spatiotemporal cube and the whole frame scene image of the current frame are input into the visual feature extraction model to obtain event features and scene features. The event features and scene features are preprocessed, and random perturbation is applied to the preprocessed scene features. Then, a time-dimensional self-attention operation is performed on the preprocessed event features to obtain the updated event features. The scene conditional fusion block is constructed to map the perturbed scene features to modulation parameters. The modulation parameters are used to conditionally modulate the updated event features. The modulated event features and the updated event features are weighted and fused together, and then residually connected with the preprocessed event features to obtain the fused event features. The fused event features are input into the scene condition classification head, and discriminative learning is performed using the camera ID as the scene label to obtain discriminative correlation output. The preprocessed event features are subjected to temporal average pooling and then combined with scene features for feature transformation and normalization to obtain event features and scene features for contrastive learning. Based on scene labels, event-scene positive and negative sample pairs are constructed, and more positive and negative sample pairs are constructed by combining historical features stored in the memory bank. Bidirectional symmetric contrastive learning of event-scene and scene-event is carried out to obtain contrastive correlation output. Based on discriminant correlation output and contrastive correlation output, discriminant anomaly score and contrastive anomaly score are calculated respectively; based on the distribution difference and local correlation of the two types of anomaly scores, event-level fused anomaly score is calculated through an adaptive fusion function. The maximum fused anomaly score of all events in each frame of video is taken as the frame-level anomaly score. The frame-level anomaly score sequence is smoothed by sliding window mid-range filtering to obtain the final frame-level anomaly score sequence, thus realizing video anomaly detection.

2. The method according to claim 1, characterized in that, Preprocessing the event features and scene features, and applying random perturbations to the preprocessed scene features, includes: The preprocessing of event features and scene features includes spatial average pooling, linear transformation, normalization, and activation processing performed sequentially; random perturbations of scene features are achieved by sampling random numbers from the interval (0,1). α The preprocessed scene features are combined with α Multiplying the results yields the perturbed scene characteristics.

3. The method according to claim 1, characterized in that, The target detection uses the YOLOv13 target detector, the optical flow estimation uses the SEA-RAFT optical flow estimator, and the visual feature extraction model is the DINOv2 model. In the target detection process, low-confidence and excessively overlapping target boxes are filtered according to the confidence threshold and the overlap threshold. In the optical flow estimation process, a binary motion map is obtained based on the gradient threshold. The regions covered by the target boxes are removed, and pseudo-motion blocks with an area lower than the preset threshold or an aspect ratio exceeding the threshold are eliminated. The target detection boxes and optical flow motion boxes are merged to obtain a set of foreground target boxes.

4. The method according to any one of claims 1 to 3, characterized in that, Based on the discriminant correlation output and the contrastive correlation output, the discriminant anomaly score and the contrastive anomaly score are calculated respectively, including: For any video frame, the th Event characteristics The discriminant anomaly score is calculated as follows: in, Labeling real-world scenarios The corresponding predicted distribution probability.

5. The method according to claim 4, characterized in that, The method further includes: Event characteristics In the time dimension Time-averaged pooling is performed on the data, and then combined with scene features. The processed event features are obtained separately through a multilayer perceptron and a single layer. and scene features ; The similarity of sample pairs is calculated based on the processed event features and scene features, and the comparative anomaly score is calculated using the sample pair similarity.

6. The method according to claim 5, characterized in that, Calculating a contrastive anomaly score using the similarity of the sample pairs includes: The contrastive anomaly score is calculated using the similarity of the sample pairs: in, Indicates the similarity between sample pairs. This refers to the temperature parameter.

7. The method according to claim 1, characterized in that, Based on the distributional differences and local correlations of the two types of anomaly scores, an event-level fused anomaly score is calculated using an adaptive fusion function, including: Based on the distributional differences and local correlations of the two types of outlier scores, an adaptive fusion function is used. The event-level fusion anomaly score is calculated as follows: in, Let i be the fusion anomaly score of the i-th event. Let be the discriminant anomaly score for the i-th event. Let be the comparative anomaly score of the i-th event.

8. The method according to claim 1, characterized in that, A scene conditional fusion block is constructed to map the perturbed scene features to modulation parameters. These modulation parameters are then used to conditionally modulate the updated event features. The modulated and updated event features are then weighted and fused together, and finally, a residual connection is made with the preprocessed event features to obtain the fused event features, including: Set up multiple scene condition fusion blocks, and use the perturbed scene features in each fusion block. The parameters are mapped through a multilayer perceptron and then converted into several modulation parameters. The modulation parameters are applied to the AdaLN-Zero style attention branch and feedforward network branch respectively to achieve conditional modulation of event features, resulting in modulated event features. ; Event features updated after self-attention operation are processed using learnable sample-aware fusion weights. Modulated event characteristics Perform weighted fusion and combine with initial event features Perform residual connections to obtain the fused event characteristics.

9. A scene-dependent video anomaly detection device, characterized in that, The device includes: The event and scene feature extraction module is used to preprocess each frame of the multi-camera surveillance video. It obtains a set of foreground target boxes through target detection and optical flow estimation, constructs a spatiotemporal cube corresponding to each foreground target based on the set of foreground target boxes, and inputs the spatiotemporal cube and the whole frame scene image of the current frame into the visual feature extraction model to obtain event features and scene features. The discriminative learning module preprocesses event features and scene features, applies random perturbations to the preprocessed scene features, and performs a time-dimensional self-attention operation on the preprocessed event features to obtain updated event features. A scene conditional fusion block is constructed to map the perturbed scene features to modulation parameters. These modulation parameters are used to conditionally modulate the updated event features. The modulated event features and the updated event features are then weighted and fused, and a residual connection is established with the preprocessed event features to obtain fused event features. These fused event features are then input into a scene conditional classification head, and discriminative learning is performed using the camera ID as the scene label to obtain discriminative correlation output. The contrastive learning module is used to perform temporal average pooling on the preprocessed event features and to perform feature transformation and normalization together with the scene features to obtain event features and scene features for contrastive learning. Based on the scene labels, event-scene positive and negative sample pairs are constructed, and more positive and negative sample pairs are constructed by combining the historical features stored in the memory bank. Bidirectional symmetric contrastive learning of event-scene and scene-event is carried out to obtain contrastive correlation output. The anomaly fusion scoring module is used to calculate discriminative anomaly scores and contrastive anomaly scores based on discriminative correlation output and contrastive correlation output, respectively. Based on the distribution differences and local correlations of the two types of anomaly scores, an event-level fusion anomaly score is calculated through an adaptive fusion function. The maximum fusion anomaly score of all events in each frame of video is taken as the frame-level anomaly score. The frame-level anomaly score sequence is smoothed by sliding window mid-range filtering to obtain the final frame-level anomaly score sequence, thereby realizing video anomaly detection.

Images