Unsupervised video anomaly detection method based on time-series dynamic saliency and boundary modeling

Abstract

The application belongs to the field of computer vision and image processing, and proposes an unsupervised video anomaly detection method based on time sequence dynamic saliency and boundary modeling. The method realizes fine-grained anomaly recognition by constructing a time sequence dynamic saliency reasoning module and a boundary enhancement module, introducing a normality prior propagation mechanism and a channel attention mechanism. In the training stage, the model learns and optimizes the anomaly detector based on the visual features of the original video extracted by the basic branch, the time sequence dynamic modeling and boundary modeling optimization, and the pseudo label obtained by combining the normality propagation. In the inference stage, the trained anomaly detector is used to identify the abnormal time period in the video. The method greatly improves the recognition accuracy in an unsupervised environment, and the architecture is refined and easy to engineer. It not only has strong robustness in complex scenes, but also takes into account the efficiency of video detection, and is suitable for intelligent monitoring and behavior recognition scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing, and relates to the use of neural networks for fine-grained anomaly recognition in videos. Specifically, it relates to an unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling. Background Technology

[0002] In scenarios such as smart security and urban governance, video anomaly detection (VAD) is a key visual task widely used in areas such as behavior recognition, risk warning, and event backtracking. Unsupervised detection methods can significantly reduce the workload of manually labeling data, making them a recent research focus. Traditional methods such as SDOR (Pang et al., CVPR 2020) and DND (Yu et al., CVPR 2022) optimize anomaly detectors through self-training, achieving frame-level anomaly detection without any manual labeling. However, this self-training paradigm is prone to confirmation bias; once the model classifies a normal behavior as an anomaly, it is difficult to correct itself, leading to serious false positives.

[0003] In recent years, some studies have attempted to introduce classic deep learning methods into unsupervised video anomaly detection tasks. For example, GCL (Zaheer et al., CVPR 2022) transformed the adversarial relationship in Generative Adversarial Networks (GANs) into a cooperative one; DiffVAD (Tur et al. ICIP 2023) utilized the reconstruction capability of diffusion models to train anomaly detectors; C2FPL (Al-Lahham et al., WACV 2024) improved training reliability by processing pseudo-labels in stages; and NPU (Shi et al., ECCV 2024) further optimized the generation quality of pseudo-labels through normality prior knowledge, achieving some progress. However, existing methods generally suffer from the following technical bottlenecks: The problem of insufficient consideration of the temporal dynamic changes of features: Traditional methods for video anomaly detection often rely heavily on obvious appearance differences. For subtle anomalies that are visually very similar to normal behavior but violate temporal logic or motion patterns, the above methods are often insufficient to generate discriminative power, resulting in poor detection performance. The temporal structure and dynamic evolution of the video are largely ignored.

[0004] Ignoring environmental interference in video: Surveillance videos may be affected by surrounding environmental interference (such as camera shake, lighting changes, and background movement), causing the model to incorrectly identify these interfering factors as abnormal behavior. Existing technologies fail to consider the interference caused by environmental factors in anomaly detection, and this noise leads to unstable model performance.

[0005] Against this backdrop, while existing unsupervised video anomaly detection techniques have made some progress through paradigm reconstruction or normality priors, they still suffer from insurmountable limitations: on the one hand, over-reliance on the similarity comparison of explicit visual appearances makes the model insensitive to dynamic mutations hidden in drastic temporal evolution, and prone to missed detections due to ignoring temporal dynamic changes; on the other hand, existing prediction mechanisms often lead to fragmented results and lack the ability to resist environmental noise such as camera shake and lighting changes. Therefore, a new technical solution is urgently needed that can amplify the differences in anomaly patterns by explicitly modeling the saliency of temporal dynamics, achieve smooth refinement of anomaly boundaries by combining global context interaction, and introduce a noise perception mechanism to suppress environmental interference. Summary of the Invention

[0006] This invention provides an unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling, aiming to address the technical bottlenecks of existing unsupervised techniques, such as over-reliance on appearance similarity, insensitivity to temporal logical anomalies, fragmented prediction results, and severe environmental noise interference. By explicitly modeling the temporal dynamic evolution of the video, this invention can accurately identify subtle anomalies that resemble normal behavioral visual features but violate temporal logic, effectively improving detection accuracy and boundary integrity in complex real-world environments. It has broad application prospects in fields such as intelligent surveillance and video behavior detection.

[0007] The technical solution of this invention is as follows: An unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling includes the following steps: The first step is data preparation: The initial videos without any editing or modification form the dataset, which is then divided into training and testing sets. No manual annotation is required for the training set. For the testing set, each video needs to be annotated with frame-level anomalies. All video data in the dataset (whether in the training or testing set) are processed by the ResNeXt3D visual feature network to extract spatiotemporal segment features. These spatiotemporal segment features deeply integrate static appearance and short-term motion patterns, providing high-dimensional semantic support for the subsequent model to capture temporal dynamic saliency. The second step is temporal dynamic saliency inference: For a video segment feature sequence, the difference value sequence is obtained by subtracting the temporally adjacent segment feature pairs, which serves as the basis for the saliency of the video's temporal dynamic changes. Then, based on the magnitude of the difference value, the segment features are divided into non-significant segment features and significant segment features. The third step is anomaly boundary enhancement: global feature enhancement is performed on the selected non-significant segment features and significant segment features respectively. The method is to pass the two types of features through the channel information interaction unit and the time series information interaction unit in turn, and then generate segment features with anomaly boundary enhancement through linear weighting operation and residual connection operation. The fourth step is the implementation of the anomaly detector: pseudo-labels are generated through graph propagation based on prior knowledge of the normality of the beginning and end of the video. A noise perception mechanism based on channel attention units is added to the anomaly detector to perform adaptive recalibration of the boundary enhancement features on the channel. During the training phase, the anomaly detector is optimized based on the pseudo-labels. During the inference phase, the trained anomaly detector is used to detect the time periods in the video where anomalies occur. Step 5: Construct the loss function and train the network. It measures the difference between the anomaly score output by the anomaly detector and the pseudo-label in the training data.

[0008] The beneficial effects of this invention are: This invention constructs an anomaly recognition method suitable for unsupervised video anomaly detection based on temporal dynamic saliency and boundary modeling. Combining saliency inference, boundary enhancement, and noise perception mechanisms, it ultimately obtains accurate anomaly detection results and has the following characteristics: 1. It achieves fully unsupervised end-to-end detection, significantly reducing the cost of practical applications and effectively solving the problems of difficult and costly large-scale data annotation in smart security scenarios, with extremely strong flexibility in practical deployment; 2. Significantly enhances the ability to capture subtle dynamic anomalies, overcoming the reliance of traditional methods on explicit appearances. This invention constructs a temporal dynamic saliency inference module, utilizing the gradient differences in features between adjacent segments to explicitly model the evolution of the video; 3. It ensures the temporal continuity and boundary integrity of the detection results. Addressing the fragmentation problem in prediction results that can easily occur due to local saliency detection, it integrates the global context through channel and temporal information interaction units, achieving feature enhancement and smoothing. 4. Excellent environmental interference resistance and detection robustness. This invention introduces a noise perception mechanism based on channel attention units in the anomaly detection module, which can adaptively recalibrate boundary enhancement features, ensuring the performance stability of the model in complex and uncontrolled real-world monitoring environments.

[0009] 5. We constructed an efficient unsupervised optimization loop, significantly improving the model's accuracy in identifying abnormal patterns. Thanks to the modular design, our method achieves excellent detection accuracy. Attached Figure Description

[0010] Figure 1 This is a diagram illustrating the specific implementation network framework. Detailed Implementation

[0011] The unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling of the present invention will be described in detail below with reference to embodiments and accompanying drawings.

[0012] An unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling includes the following steps: The first step is data preparation: The initial videos without any editing or modification form the dataset, which is then divided into training and testing sets. No manual annotation is required for the training set. For the testing set, each video needs to be annotated with frame-level anomalies. All video data in the dataset (whether in the training or testing set) are processed by the ResNeXt3D visual feature network to extract spatiotemporal segment features. These spatiotemporal segment features deeply integrate static appearance and short-term motion patterns, providing high-dimensional semantic support for the subsequent model to capture temporal dynamic saliency. The second step is temporal dynamic saliency inference: For a video segment feature sequence, the difference value sequence is obtained by subtracting the temporally adjacent segment feature pairs, which serves as the basis for the saliency of the video's temporal dynamic changes. Then, based on the magnitude of the difference value, the segment features are divided into non-significant segment features and significant segment features. The third step is anomaly boundary enhancement: global feature enhancement is performed on the selected non-significant segment features and significant segment features respectively. The method is to pass the two types of features through the channel information interaction unit and the time series information interaction unit in turn, and then generate segment features with anomaly boundary enhancement through linear weighting operation and residual connection operation. The fourth step is the implementation of the anomaly detector: pseudo-labels are generated through graph propagation based on prior knowledge of the normality of the beginning and end of the video. A noise perception mechanism based on channel attention units is added to the anomaly detector to perform adaptive recalibration of the boundary enhancement features on the channel. During the training phase, the anomaly detector is optimized based on the pseudo-labels. During the inference phase, the trained anomaly detector is used to detect the time periods in the video where anomalies occur. Step 5: Construct the loss function and train the network. It measures the difference between the anomaly score output by the anomaly detector and the pseudo-label in the training data.

[0013] Furthermore, in the first step, the training set contains a mixture of video sequences of normal and abnormal events. The model (the overall model formed by the steps described in the second to fourth steps above) learns using the training set, while the test set is used to evaluate the model's performance.

[0014] Furthermore, in the second step, for a video segment feature sequence ,in It is the number of fragment features. To represent the dimension of a segment feature, first calculate the temporally adjacent segment feature pairs. Difference value : in express The norm is used to measure the magnitude of change between feature vectors of temporally adjacent segments; subsequently, by calculating the difference values ​​between feature pairs of adjacent segments across all temporal sequences, the set of video difference values ​​is obtained. : Next, the set of difference values ​​is analyzed based on the magnitude of the difference values. Sort the data from highest to lowest and select the largest value. Top- 1 difference value (Top- ), and record the time series index set of features corresponding to these high dissimilarity values, denoted as . The proportion of significant changes is considered as a hyperparameter; Definition: Only one fragment feature It participated in any significant high-dynamic change (i.e., it is a Top-) The segment containing either the antecedent or consequent of the difference pair is then labeled as a salient feature; to achieve this logic, a length of [missing information] is introduced. Significance of the allocation sequence For each time-series index , ,if ,So This indicates that the feature is significant; otherwise... This indicates that the feature is not significant.

[0015] Furthermore, the third step specifically includes: The saliency allocation sequence obtained from time-series dynamic saliency reasoning The original features are divided into non-significant sets. and significant sets ,in , , Indicates the number of non-significant fragment features. Indicates the number of significant fragment features; Then, the features are processed through the channel information interaction unit. , Perform feature enhancement along the channel dimension: in, This represents a miniature multilayer perceptron (MLP) consisting of two fully connected layers. Represents the normalized exponential function, This represents element-wise multiplication; Next, global semantic relationships are captured through the temporal information interaction unit: in, The transpose operation represents a matrix projection operation. This matrix projection operation broadcasts information from some feature branches back to the global time scale, resulting in a more accurate reconstruction. Restore to Dimension; The two enhanced branches are then linearly weighted and fused, and finally the anomaly boundary refinement features are obtained through residual connections. : in, It is a trade-off factor that balances significant and insignificant contributions. This represents the features after weighted fusion; then residual connections are performed: in, It is a hyperparameter that controls the degree of injection of global residuals to ensure the numerical stability of the training process.

[0016] Furthermore, in the fourth step, specifically: Step 4.1: Obtain pseudo-labels based on prior knowledge of normal propagation, specifically as follows: Step 4.1.1: Construct the initial normality indicator vector The first and last segments of the video are marked as normal with a value of 1, while the remaining segments in between are temporarily marked as unknown with a value of 0. in, and The value of is 1, and the value of the middle value is 0; Step 4.1.2: Calculate the feature similarity matrix of time-series modulation. : First, calculate the similarity matrix in the feature space. : In particular, when When the similarity is 0, the similarity is 0; where Represents Euclidean distance. To control the hyperparameters of similarity sensitivity, and Indicates the fragment feature index; Secondly, calculate the temporal spatial similarity matrix. Used to constrain the propagation range based on time distance: In particular, when At that time, the propagation range was 0; among which, and They represent the first The and the first Timestamp index of each segment; Finally, by multiplying the two elements element-wise and fusing them, the final temporal modulation similarity matrix is ​​obtained. : Step 4.1.3, for the matrix After performing symmetric normalization, the propagation matrix is ​​obtained. : in, It is a diagonal matrix, where each row of diagonal elements is a matrix. The sum of the elements in each row; Step 4.1.4: Finally, the normality amplitude vector is obtained. : in, It is the identity matrix. It is a hyperparameter that controls the propagation depth; Step 4.1.5: Based on the calculated normality range Statistical strategies are used to generate binarized pseudo-labels. Specifically: First, the standard deviation of the normality amplitude of all segments in the video is calculated as the video-level anomaly score. If the standard deviation is greater than a preset threshold, it indicates that the normality fluctuation within the video is large, containing abnormal events, and is therefore judged as an anomaly. For videos judged to contain anomalies, their normality amplitude vector is selected. The segments with the smallest median values ​​are marked as abnormal, and the rest are marked as normal; the final generated pseudo-label sequence The definition is as follows: in, This pseudo tag This signal will be used as a monitoring signal and input into the subsequent loss function calculation to guide the parameter optimization of the anomaly detector. Step 4.2: Calculate local temporal embedding features ; Step 4.3: Use attention units to aggregate global average pooling and max pooling information in parallel over time, and generate channel weight vectors using a multilayer perceptron (MLP). : in, The number of channels representing the feature. It is the Sigmoid activation function. This represents a one-dimensional convolutional layer. Represents a non-linear activation function. Indicates global average pooling. This indicates max pooling. This represents a multilayer perceptron used to generate channel weights; Step 4.4: Recalibrate the local temporal embedded features using the channel weight vector to obtain the enhanced features. ; Step 4.5: Based on the enhanced features... Abnormal scores for each segment The final classification head calculation yields the following results: in express At time step eigenvectors, This indicates that the multilayer perceptron used for classification is responsible for mapping the enhanced features to the probability space. Use the Sigmoid activation function; 6. The unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling according to claim 1, characterized in that, in the fifth step, the constructed binary cross-entropy loss is: .

[0017] Example: The video anomaly detection results of this invention and their comparison with other unsupervised methods are shown in Table 1. This invention uses the commonly used Area Under the Receiver Operating Characteristic (AUC) curve as the criterion for evaluating the accuracy of video anomaly detection; a larger AUC indicates higher detection accuracy. This invention uses two video anomaly detection datasets (ShanghaiTech and UCF-Crime) to verify the effectiveness of the proposed unsupervised video anomaly detection method based on temporal dynamic salience and boundary modeling. Specifically, ShanghaiTech is a dataset based on campus surveillance, and its anomaly events mainly include violations such as cycling on sidewalks, fighting, or suddenly running in crowds, totaling 437 videos. UCF-Crime is a large-scale real-world surveillance dataset containing 128 hours of unedited recordings, totaling 1900 videos. It covers 13 highly complex real-world crime or anomaly types (such as robbery, arson, and car accidents). The method of this invention is trained on a single NVIDIA-A40 graphics card using the RMSprop training optimizer. During the basic training phase, the model was trained 300 times on the ShanghaiTech and UCF-Crime datasets, with the learning rate set to 0.0001 and the momentum parameter set to 0.6.

[0018] Table 1. Accuracy Comparison of Anomaly Detection Results of the Invention

[0019] The comparison methods include OGNet (Zaheer M Z, Lee J, Astrid M, et al. Old is gold: Redefining the adversarially learned one-class classifier training paradigm[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 14183-14193.), Kim et al. (Kim J H, Kim D H, Yi S, et al. Semi-orthogonal embedding for efficient unsupervised anomaly segmentation[J]. arXiv preprint arXiv:2105.14737, 2021.), LBR-SPR (Yu G, Wang S, Cai Z, et al. Deep anomaly discovery from unlabeled videos via normality advantage and self-paced refinement[C] / / Proceedings of the IEEE / CVF Conference on computer vision and pattern recognition. 2022: 13987-13998.), GCL (Zaheer M Z, Mahmood A, Khan M H, et al. Generative cooperative learning for unsupervised video anomaly detection[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 14744-14754.), C2FPL (Al-Lahham A, Tastan N, Zaheer M Z, et al.A coarse-to-fine pseudo-labeling (c2fpl) framework forunsupervised video anomaly detection[C] / / Proceedings of the IEEE / CVF WinterConference on Applications of Computer Vision. 2024: 6793-6802.),NPU(Shi H,Wang L, Zhou S, et al. Learning Anomalies with Normality Prior forUnsupervised Video Anomaly Detection[C] / / European Conference on ComputerVision. Cham: Springer Nature Switzerland, 2024: 163-180.)。.

Claims

1. An unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling, characterized in that, Includes the following steps: The first step is data preparation: the initial videos without any editing or modification form the dataset, which is then divided into training and testing sets. No manual annotation is required for the training set; for the testing set, frame-level anomaly annotation is required for each video. All video data in the dataset (whether training or test set) are processed by ResNeXt3D's visual feature network to extract spatiotemporal segment features, which deeply integrate static appearance and short-term motion patterns. The second step is temporal dynamic saliency inference: For a video segment feature sequence, the difference value sequence is obtained by subtracting the temporally adjacent segment feature pairs, which serves as the basis for the saliency of the video's temporal dynamic changes. Then, based on the magnitude of the difference value, the segment features are divided into non-significant segment features and significant segment features. The third step is anomaly boundary enhancement: global feature enhancement is performed on the selected non-significant segment features and significant segment features respectively. The method is to pass the two types of features through the channel information interaction unit and the time series information interaction unit in turn, and then generate segment features with anomaly boundary enhancement through linear weighting operation and residual connection operation. The fourth step is the implementation of the anomaly detector: pseudo-labels are generated through graph propagation based on prior knowledge of the normality of the beginning and end of the video. A noise perception mechanism based on channel attention units is added to the anomaly detector to perform adaptive recalibration of the boundary enhancement features on the channel. During the training phase, the anomaly detector is optimized based on the pseudo-labels. During the inference phase, the trained anomaly detector is used to detect the time periods in the video where anomalies occur. Step 5: Construct the loss function and train the network. It measures the difference between the anomaly score output by the anomaly detector and the pseudo-label in the training data.

2. The unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling according to claim 1, characterized in that, In the first step, the training set contains a mixture of video sequences of normal and abnormal events. The model uses the training set to learn, while the test set is used to evaluate the model's performance.

3. The unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling according to claim 1, characterized in that, In the second step, for a video segment feature sequence ,in It is the number of fragment features. To represent the dimension of a segment feature, first calculate the temporally adjacent segment feature pairs. Difference value : in express The norm is used to measure the magnitude of change between feature vectors of temporally adjacent segments; subsequently, by calculating the difference values ​​between feature pairs of adjacent segments across all temporal sequences, the set of video difference values ​​is obtained. : Next, the set of difference values ​​is analyzed based on the magnitude of the difference values. Sort the data from highest to lowest and select the largest value. Top- 1 difference value (Top- ), and record the time series index set of features corresponding to these high dissimilarity values, denoted as . The proportion of significant changes is considered as a hyperparameter; Definition: Only one fragment feature It participated in any significant high-dynamic change (i.e., it is a Top-) The segment containing either the antecedent or consequent of the difference pair is then labeled as a salient feature; to achieve this logic, a length of [missing information] is introduced. Significance of the allocation sequence For each time-series index , ,if ,So This indicates that the feature is significant; otherwise... This indicates that the feature is not significant.

4. The unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling according to claim 1, characterized in that, The third step specifically involves: The saliency allocation sequence obtained from time-series dynamic saliency reasoning The original features are divided into non-significant sets. and significant sets ,in , , Indicates the number of non-significant fragment features. Indicates the number of significant fragment features; Then, the features are processed through the channel information interaction unit. , Perform feature enhancement along the channel dimension: in, This represents a miniature multilayer perceptron (MLP) consisting of two fully connected layers. Represents the normalized exponential function, This represents element-wise multiplication; Next, global semantic relationships are captured through the temporal information interaction unit: in, The transpose operation represents a matrix projection operation. This matrix projection operation broadcasts information from some feature branches back to the global time scale, resulting in a more accurate reconstruction. Restore to Dimension; The two enhanced branches are then linearly weighted and fused, and finally the anomaly boundary refinement features are obtained through residual connections. : in, It is a trade-off factor that balances significant and insignificant contributions. This represents the features after weighted fusion; then residual connections are performed: in, It is a hyperparameter that controls the degree of injection of global residuals to ensure the numerical stability of the training process.

5. The unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling according to claim 1, characterized in that, The fourth step specifically involves: Step 4.1: Obtain pseudo-labels based on prior knowledge of normal propagation, specifically as follows: Step 4.1.1: Construct the initial normality indicator vector The first and last segments of the video are marked as normal with a value of 1, while the remaining segments in between are temporarily marked as unknown with a value of 0. in, and The value of is 1, and the value of the middle value is 0; Step 4.1.2: Calculate the feature similarity matrix of time-series modulation. : First, calculate the similarity matrix in the feature space. : In particular, when When the similarity is 0, the similarity is 0; where Represents Euclidean distance. To control the hyperparameters of similarity sensitivity, and Indicates the fragment feature index; Secondly, calculate the temporal spatial similarity matrix. Used to constrain the propagation range based on time distance: In particular, when At that time, the propagation range was 0; among which, and They represent the first The and the first Timestamp index of each segment; Finally, by multiplying the two elements element-wise and fusing them, the final temporal modulation similarity matrix is ​​obtained. : Step 4.1.3, for the matrix After performing symmetric normalization, the propagation matrix is ​​obtained. : in, It is a diagonal matrix, where each row of diagonal elements is a matrix. The sum of the elements in each row; Step 4.1.4: Finally, the normality amplitude vector is obtained. : in, It is the identity matrix. It is a hyperparameter that controls the propagation depth; Step 4.1.5: Based on the calculated normality range Statistical strategies are used to generate binarized pseudo-labels. Specifically: First, the standard deviation of the normality amplitude of all segments in the video is calculated as the video-level anomaly score. If the standard deviation is greater than a preset threshold, it indicates that the normality fluctuation within the video is large, containing abnormal events, and is therefore judged as an anomaly. For videos judged to contain anomalies, their normality amplitude vector is selected. The segments with the smallest median values ​​are marked as abnormal, and the rest are marked as normal; the final generated pseudo-label sequence The definition is as follows: in, This pseudo tag This signal will be used as a monitoring signal and input into the subsequent loss function calculation to guide the parameter optimization of the anomaly detector. Step 4.2: Calculate local temporal embedding features ; Step 4.3: Use attention units to aggregate global average pooling and max pooling information in parallel over time, and generate channel weight vectors using a multilayer perceptron (MLP). : in, The number of channels representing the feature. It is the Sigmoid activation function. This represents a one-dimensional convolutional layer. Represents a non-linear activation function. Indicates global average pooling. This indicates max pooling. This represents a multilayer perceptron used to generate channel weights; Step 4.4: Recalibrate the local temporal embedded features using the channel weight vector to obtain the enhanced features. ; Step 4.5: Based on the enhanced features... Abnormal scores for each segment The final classification head calculation yields the following results: in express At time step eigenvectors, This indicates that the multilayer perceptron used for classification is responsible for mapping the enhanced features to the probability space. This is the Sigmoid activation function.

6. The unsupervised video anomaly detection method based on temporal dynamic saliency and boundary modeling according to claim 1, characterized in that, In step five, the constructed binary cross-entropy loss is: 。

Images