A weakly supervised video anomaly detection method based on label noise perception strategy

Abstract

This invention relates to a weakly supervised video anomaly detection method based on a label noise perception strategy, comprising the following steps: S1, video feature extraction; First, 16 consecutive frames of a video are input as a segment into a feature extractor to generate a feature map of size 32×256×512. Subsequently, the text information of the anomaly type is encoded into a 32×256×14 text embedding, wherein there are 14 anomaly types. By matching the text embedding of the anomaly type with the corresponding video frame features, and inferring pseudo-labels based on the matching similarity, this invention breaks through the limitations of the traditional self-training paradigm from the perspective of label noise learning, breaks the inherent mode of the traditional video anomaly detection self-training framework, and proposes a method for generating pseudo-labels online in an end-to-end manner, solving the problems of error accumulation and noise sensitivity in video anomaly detection pseudo-labels.

Description

Technical Field

[0001] This invention relates to the field of weakly supervised video anomaly detection, specifically a weakly supervised video anomaly detection method based on a label noise perception strategy. Background Technology

[0002] Weakly supervised video anomaly detection is a method that uses limited labeled information (such as video-level labels) to identify anomalous events in videos. Its core objective is to accurately detect anomalous events in videos through model learning. With the rapid increase in the number of surveillance videos and the continuous improvement of intelligent security requirements, traditional fully supervised methods are struggling to meet practical needs due to high annotation costs and the diversity of scenarios. Therefore, weakly supervised video anomaly detection has gradually become a research hotspot in the field of video analytics. If anomalous events cannot be detected and handled in a timely manner, they may pose public safety hazards and cause serious property damage.

[0003] In the research of weakly supervised video anomaly detection, current mainstream methods mainly fall into two categories: methods based on multi-instance learning and methods based on self-trained pseudo-labels. However, traditional methods have the following drawbacks:

[0004] (1) Error accumulation: Since the pseudo-labels generated in the first stage are of low quality, when they are directly used to train the model in the second stage, the inherent error in the pseudo-labels is further amplified, resulting in the continuous accumulation of error during the training process, which affects the model performance.

[0005] (2) Noise sensitivity: The noise contained in the pseudo-labels makes the model susceptible to interference from incorrect labels during training, thereby weakening the model's ability to distinguish real abnormal events and reducing the accuracy and robustness of detection.

[0006] Therefore, we propose a weakly supervised video anomaly detection method based on a label noise perception strategy, which breaks through the limitations of the traditional self-training paradigm from the perspective of label noise learning. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies, adapt to practical needs, and provide a weakly supervised video anomaly detection method based on a label noise perception strategy. This addresses the current technical problems where the quality of pseudo-labels generated in the first stage is low, and when directly used for model training in the second stage, the inherent errors in the pseudo-labels are further amplified, leading to the continuous accumulation of errors during training and affecting model performance. Furthermore, the noise contained in the pseudo-labels makes the model susceptible to interference from incorrect labels during training, thereby weakening the model's ability to distinguish real anomalies and reducing the accuracy and robustness of detection.

[0008] Specifically, the model first accurately identifies clean samples based on its historical prediction volatility strategy. Then, to reduce the accumulation of bias in the sample selection process, a regularization term is introduced to penalize the model's confidence output distribution. Through this regularization strategy, the model can handle clean and noisy samples more evenly, effectively avoiding overfitting to noisy data, thereby improving classification accuracy and model robustness.

[0009] To achieve the objectives of this invention, the technical solution adopted is as follows: A weakly supervised video anomaly detection method based on a label noise perception strategy is designed, comprising the following steps:

[0010] S1, Video Feature Extraction

[0011] First, 16 consecutive frames of the video are input as a segment into the feature extractor to generate a feature map of size 32×256×512. Then, the text information of the anomaly type is encoded into a text embedding of 32×256×14, where there are 14 anomaly types. The text embedding of the anomaly type is matched with the corresponding video frame features, and pseudo-labels are inferred based on the matching similarity.

[0012] S2, Online generation of pseudo-tags

[0013] To address the error accumulation in self-trained pseudo-label methods, an end-to-end online pseudo-label generation network framework is proposed. First, video features and text are input into the prediction module, whose main function is to calculate similarity matching scores and generate anomaly score maps. Next, to enhance the propagation of contextual information and improve the quality of pseudo-label generation, video features are input into the knowledge propagation module. In this module, the expectation-maximization algorithm is used to extract representative segments, and the video features are updated using a random walk algorithm. Finally, the updated video features are input into the prediction module again to generate the final pseudo-label y. The loss function is defined as follows:

[0014]

[0015] The loss function is used to compensate for the lack of fine-grained supervision information;

[0016] S3, supervised learning of networks

[0017] From the perspective of label noise learning, online generated pseudo-labels are regarded as fine-grained real labels. However, since the initial labels usually contain noise, a multi-stage training strategy network is designed. The network structure is divided into two stages: a warm-up stage and a main learning stage. In the warm-up stage, the network acquires preliminary discrimination ability through training for several epochs, and the memory bank is initialized at the same time. In the main learning stage, a fluctuation strategy is used to select clean samples for training to further improve the robustness of the model. In addition, the external memory bank module is dynamically updated in each training cycle.

[0018] Warm-up Phase: To avoid overfitting to noisy labels in the early stages of network training, we introduce a regularization term to penalize the confidence of the network output, and express the confidence penalty regularization term as follows:

[0019] R = -α(P) j )·logP j Where P j It is the second most confident class for predicting p. In the case of noisy labels, we consider class j to be the correct class for noisy instances.

[0020]

[0021] Here, Γ is a hyperparameter of the confidence threshold, and y is the category corresponding to the true label. If the network is overconfident about class y, it will be subject to a larger α(P) threshold. j )punish;

[0022] Main learning phase:

[0023] By utilizing the proposed memory module, clean samples are selected based on the strategy of whether the historical predicted categories stored in the memory are consistent in the two most recent predictions. However, directly using these samples for training may ignore the contextual relationships between video segments. Therefore, LSTM is introduced to model the temporal dependencies between video segments and is used for network training in the current epoch. Meanwhile, the memory module is dynamically updated as training progresses. Since the network is relatively weak in the early learning stages, a regularization term is designed to prevent error accumulation from causing sample selection bias. The robustness of the model is improved by penalizing the confidence output distribution of the sample (x,y).

[0024]

[0025] Where α(P) k Calculated using the formula described above, the regularization term penalizes the model's confidence output by minimizing the expected loss for each class; furthermore, For the selected samples, the loss function in the main learning phase is expressed as:

[0026]

[0027] Where λ is a hyperparameter set by cross-validation.

[0028] Preferably, the feature extractor in S1 is a CLIP image encoder.

[0029] Preferably, the feature map size in S1 is 32×256×512, and the batch size is 32.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] 1. This invention breaks the inherent pattern of traditional video anomaly detection self-training frameworks and proposes a method for generating pseudo-labels online using an end-to-end approach.

[0032] 2. This invention solves the problems of error accumulation and noise sensitivity in video anomaly detection pseudo-tags.

[0033] In summary, this invention proposes a weakly supervised video anomaly detection method based on a label noise perception strategy. Starting from the perspective of label noise learning, it breaks through the limitations of the traditional self-training paradigm. Specifically, firstly, based on the volatility strategy of the model's historical predictions, clean samples are accurately identified. Subsequently, to reduce the accumulation of bias in the sample selection process, a regularization term is introduced to penalize the model's confidence output distribution. Through this regularization strategy, the model can more balancedly handle clean and noisy samples, effectively avoiding overfitting to noisy data, thereby improving classification accuracy and model robustness. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the network framework of the present invention;

[0035] Figure 2 This is a schematic diagram of the online pseudo-label generation network of the present invention;

[0036] Figure 3 This is a schematic diagram of the knowledge dissemination module of the present invention;

[0037] Figure 4 This is a schematic diagram of the process of the present invention;

[0038] Figure 5 This is a comparative example of the process diagram of the present invention;

[0039] Figure 6 This is a schematic diagram of the process of Comparative Example 2 of the present invention;

[0040] Figure 7 This is a comparative example of the three-process diagram of the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0042] A weakly supervised video anomaly detection method based on a label noise perception strategy, see [link to relevant documentation]. Figures 1 to 3 This includes the following steps:

[0043] S1, Video Feature Extraction

[0044] First, 16 consecutive frames of the video are input as a segment into the feature extractor to generate a feature map of size 32×256×512. Then, the text information of the anomaly type is encoded into a text embedding of 32×256×14, where there are 14 anomaly types. The text embedding of the anomaly type is matched with the corresponding video frame features, and pseudo-labels are inferred based on the matching similarity.

[0045] S2, Online generation of pseudo-tags

[0046] To address the error accumulation in self-trained pseudo-label methods, an end-to-end online pseudo-label generation network framework is proposed, such as... Figure 2 As shown, firstly, video features and text are input into the prediction module, whose main function is to calculate the similarity matching degree and generate anomaly score maps. Next, to enhance the propagation of contextual information and improve the quality of pseudo-label generation, video features are input into the knowledge propagation module (e.g., Figure 3 As shown in the diagram, in this module, the expectation-maximization algorithm is used to extract representative segments, and the video features are updated using a random walk algorithm. Finally, the updated video features are input back into the prediction module to generate the final pseudo-label y. The loss function is defined as follows:

[0047]

[0048] The loss function is used to compensate for the lack of fine-grained supervision information;

[0049] S3, supervised learning of networks

[0050] From the perspective of learning from label noise, online-generated pseudo-labels are considered as fine-grained real labels. However, since the initial labels often contain noise, a multi-stage training strategy is designed for the network, such as... Figure 1 As shown, the network structure is divided into two stages: a warm-up stage and a main learning stage. In the warm-up stage, the network acquires preliminary discrimination ability through training for several epochs, and the memory bank is initialized at the same time. In the main learning stage, a fluctuation strategy is used to select clean samples for training to further improve the robustness of the model. In addition, the external memory bank module is dynamically updated in each training cycle.

[0051] Warm-up phase: To avoid overfitting to noisy labels in the early stages of network training, a regularization term is introduced to penalize the confidence of the network output, and the confidence penalty regularization term is expressed as:

[0052] R = -α(P) j )·logP j Where P j It is the class with the second highest confidence in predicting p, and it is considered that class j is likely the correct class for noisy instances when the label contains noise.

[0053]

[0054] Here, Γ is a hyperparameter of the confidence threshold, and y is the category corresponding to the true label. If the network is overconfident about class y, it will be subject to a larger α(P) threshold. j )punish;

[0055] Main learning phase:

[0056] By utilizing the proposed memory module, clean samples are selected based on the strategy of whether the historical predicted categories stored in the memory are consistent in the two most recent predictions. However, directly using these samples for training may ignore the contextual relationships between video segments. Therefore, LSTM is introduced to model the temporal dependencies between video segments and is used for network training in the current epoch. Meanwhile, the memory module is dynamically updated as training progresses. Since the network is relatively weak in the early learning stages, a regularization term is designed to prevent error accumulation from causing sample selection bias. The robustness of the model is improved by penalizing the confidence output distribution of the sample (x,y).

[0057]

[0058] Where α(P) k Calculated using the formula described above, the regularization term penalizes the model's confidence output by minimizing the expected loss for each class; furthermore, For the selected samples, the loss function in the main learning phase is expressed as:

[0059]

[0060] Where λ is a hyperparameter set by cross-validation.

[0061] Specifically, the feature extractor in S1 is the CLIP image encoder.

[0062] Furthermore, the feature map size in S1 is 32×256×512, and the batch size is 32.

[0063] Comparative Example 1: Performance Comparison between Basic Methods Based on Multi-Instance Learning (such as MGFN) and the Invention

[0064] S1. Data Preparation

[0065] To evaluate the performance of a base method based on multi-instance learning (MGFN) in anomaly detection, the UCF-Crime dataset was collected. This dataset contains 1900 video clips across 13 different types of anomalous events. To ensure the reliability of the test results, each video clip was processed using the MGFN method, generating corresponding anomaly detection results. Furthermore, features such as the distribution of anomalous events, video length, and background complexity in the dataset were ensured to be consistent with those in real-world datasets.

[0066] S2, Performance Testing Methods

[0067] During the performance testing phase, AUC (Area Under the Curve) was used as the evaluation metric to measure the model's classification ability at different thresholds. A higher AUC value indicates better classification performance.

[0068] S3. Results and Analysis

[0069] By comparing the AUC values ​​of the MGFN method and the method proposed in this invention, the performance improvement of this invention can be evaluated. Specifically, the method proposed in this invention achieves an AUC value of 87.77% on the UCF-Crime dataset, compared to 86.67% for the MGFN method, demonstrating a performance improvement. This improvement indicates that this invention possesses better classification ability and accuracy in anomaly detection tasks.

[0070] It is worth noting that although the MGFN method, as a fundamental method based on multi-instance learning, has achieved certain performance in anomaly detection tasks, this invention further improves the performance of anomaly detection by introducing more advanced strategies and techniques, such as more refined feature extraction, more accurate pseudo-label generation, and more powerful model training.

[0071] In summary, by comparing the AUC values ​​of the MGFN method and the method proposed in this invention on the UCF-Crime dataset, the advantages of this invention in anomaly detection performance can be clearly seen.

[0072] Performance evaluation:

[0073] On the UCF-Crime dataset, the MGFN method has an AUC of 86.67%.

[0074] On the XD-Violence dataset, the AP of the MGFN method is 80.11%.

[0075] Comparison points:

[0076] Compared to this invention, the MGFN method only uses video-level labels and lacks fine-grained supervision information, which limits the performance of anomaly detection.

[0077] This invention effectively improves the performance of anomaly detection by introducing textual descriptions of anomaly types and a label noise perception strategy.

[0078] Comparative Example 2: Performance Comparison between the UR-DMU Method Based on Self-Trained Pseudo-Labels and the Invention

[0079] S1, Data Preparation Stage

[0080] To evaluate the performance of the UR-DMU method in anomaly detection tasks, the UCF-Crime dataset was collected. This dataset contains videos of various types of anomalous events, each video segment of which has been preprocessed and assigned a video-level label (normal or anomalous). To ensure the accuracy and reliability of the experiments, the dataset was divided into training, validation, and test sets according to the requirements of the UR-DMU method. This facilitates the effective tuning of model parameters during training and the accurate evaluation of model performance during testing.

[0081] S2, Feature Extraction Stage

[0082] In the feature extraction stage, the feature extraction method described in the UR-DMU method was adopted. This method can capture key information in video segments, such as motion features and spatiotemporal features, and represent this information as feature vectors or feature maps. These feature vectors or feature maps will serve as inputs for subsequent model training to learn anomaly detection patterns in video segments.

[0083] S3, Pseudo-tag generation stage

[0084] Pseudo-label generation is one of the core steps of the UR-DMU method. A base model is first trained on the initial training set. Then, this base model is used to predict unlabeled video segments, generating pseudo-labels. To improve the accuracy and reliability of the pseudo-labels, multiple iterations are performed. In each iteration, the model is updated based on the current pseudo-labels, and the updated model is used to re-predict unlabeled video segments. In this way, the quality of the pseudo-labels can be gradually optimized, providing more reliable supervision information for subsequent model training.

[0085] S4, Model Training Phase

[0086] During the model training phase, generated pseudo-labels are used as supervision information to train a more complex model (such as a deep learning model). To improve the model's generalization ability, various strategies are employed, such as regularization and data augmentation. Regularization helps prevent overfitting and improves the model's generalization performance; data augmentation increases the diversity of training data, enabling the model to better adapt to various complex anomaly detection scenarios.

[0087] S5, Performance Testing and Comparative Analysis

[0088] During the performance testing phase, AUC was used as the evaluation metric to measure the anomaly detection performance of the UR-DMU method on the test set of the UCF-Crime dataset. By recording the AUC values ​​of the UR-DMU method on the test set, it can be compared and analyzed with the method proposed in this invention.

[0089] The UR-DMU method achieved an AUC of 86.97% on the UCF-Crime dataset. Compared to the method proposed in this invention, the UR-DMU method shows a certain gap in anomaly detection performance.

[0090] In summary, although the UR-DMU method has shown some performance in anomaly detection tasks, the method proposed in this invention has a greater performance advantage.

[0091] Performance evaluation:

[0092] On the UCF-Crime dataset, the UR-DMU method achieves an AUC of 86.97%.

[0093] On the XD-Violence dataset, the AP of the UR-DMU method is 81.66%.

[0094] Comparison points:

[0095] Compared with the present invention, the UR-DMU method has a slight improvement in performance, but still does not solve the problems of tag noise and error accumulation.

[0096] This invention effectively alleviates these problems and improves the robustness of the model by using end-to-end online pseudo-label generation and regularization strategies.

[0097] Comparative Example 3: Performance Comparison between the CLIP-TSA Method Based on Self-Trained Pseudo-Labels and Text Embedding and the Invention

[0098] S1. Data Preparation and Preprocessing

[0099] To evaluate the performance of the CLIP-TSA method in anomaly detection tasks, the UCF-Crime dataset was collected. This dataset contains videos of various types of anomalous events. In addition to the video data, text descriptions of the anomaly type corresponding to each video segment were also collected. These text descriptions will be used to generate text embeddings to assist in the representation and learning of video features.

[0100] S2, Feature Extraction and Text Embedding Generation

[0101] During the feature extraction phase, following the description in the CLIP-TSA method, detailed feature extraction was performed on each video segment. These features include spatiotemporal features, motion features, etc., which together constitute a complete representation of the video segment. Simultaneously, a pre-trained text encoder (such as BERT or GPT) was used to encode the text descriptions of anomaly types into text embeddings. These text embeddings will supplement the video features and be used for subsequent pseudo-label generation and model training.

[0102] S3, Pseudo-tag generation strategy

[0103] A core feature of the CLIP-TSA method is its combination of video features and text embeddings to generate pseudo-labels. In the pseudo-label generation stage, video features and text embeddings are first fused, and then the fused features are used for anomaly detection. Since there is no real label information initially, the pre-trained model is used iteratively to predict unlabeled video segments and generate pseudo-labels based on the prediction results. These pseudo-labels serve as supervision information for subsequent model training.

[0104] S4. Model Training and Optimization

[0105] During the model training phase, generated pseudo-labels are used as supervision information to train a deep learning model that fuses video features and text embeddings. To improve model performance, a multimodal information alignment and fusion strategy is employed to ensure consistency between video features and text embeddings in the feature space. Furthermore, regularization and data augmentation strategies are introduced to prevent overfitting and improve the model's generalization ability.

[0106] S5, Performance Testing and Comparative Analysis

[0107] During the performance testing phase, AUC and AP were used as evaluation metrics to measure the anomaly detection performance of the CLIP-TSA method on the test set of the UCF-Crime dataset. By recording the AUC and AP values ​​of the CLIP-TSA method on the test set, it can be compared and analyzed with the method proposed in this invention.

[0108] The CLIP-TSA method achieves an AUC of 87.58% and an AP of 82.17% on the UCF-Crime dataset. While the CLIP-TSA method performs similarly to the proposed method in anomaly detection, the proposed method still offers a certain advantage in AUC value.

[0109] Furthermore, it is worth noting that this invention also exhibits improved robustness. Due to the complexity and diversity of video data in practical applications, model robustness is crucial for anomaly detection tasks. This invention, by introducing richer features and strategies, can better address various complex anomaly detection scenarios.

[0110] In summary, while the CLIP-TSA method performs well in anomaly detection tasks, this invention still possesses certain advantages in terms of performance, generalization ability, and robustness. These advantages give this invention greater potential and value in practical applications.

[0111] Performance evaluation:

[0112] On the UCF-Crime dataset, the CLIP-TSA method achieves an AUC of 87.58%.

[0113] On the XD-Violence dataset, the CLIP-TSA method achieves an AP of 82.17%.

[0114] Comparison points:

[0115] Compared with the present invention, the CLIP-TSA method is similar in performance, but the present invention is the first to start from the perspective of tag noise learning and introduces a regularization strategy to suppress the influence of noise.

[0116] This invention improves model performance by more accurately identifying clean samples through a volatility strategy and a memory module.

[0117] Example 1: The method proposed in this invention

[0118] S1, Video Feature Extraction

[0119] First, 16 consecutive frames of the video are input as a segment into the feature extractor to generate a feature map of size 32×256×512. Then, the text information of the anomaly type is encoded into a text embedding of 32×256×14, where there are 14 anomaly types. The text embedding of the anomaly type is matched with the corresponding video frame features, and pseudo-labels are inferred based on the matching similarity.

[0120] S2, Online generation of pseudo-tags

[0121] To address the error accumulation in self-trained pseudo-label methods, an end-to-end online pseudo-label generation network framework is proposed, such as... Figure 2 As shown, firstly, video features and text are input into the prediction module, whose main function is to calculate the similarity matching degree and generate anomaly score maps. Next, to enhance the propagation of contextual information and improve the quality of pseudo-label generation, video features are input into the knowledge propagation module (e.g., Figure 3 As shown in the diagram, in this module, the expectation-maximization algorithm is used to extract representative segments, and the video features are updated using a random walk algorithm. Finally, the updated video features are input back into the prediction module to generate the final pseudo-label y. The loss function is defined as follows:

[0122]

[0123] The loss function is used to compensate for the lack of fine-grained supervision information;

[0124] S3, supervised learning of networks

[0125] From the perspective of learning from label noise, online-generated pseudo-labels are considered as fine-grained real labels. However, since the initial labels often contain noise, a multi-stage training strategy is designed for the network, such as... Figure 1 As shown, the network structure is divided into two stages: a warm-up stage and a main learning stage. In the warm-up stage, the network acquires preliminary discrimination ability through training for several epochs, and the memory bank is initialized at the same time. In the main learning stage, a fluctuation strategy is used to select clean samples for training to further improve the robustness of the model. In addition, the external memory bank module is dynamically updated in each training cycle.

[0126] Warm-up phase: To avoid overfitting to noisy labels in the early stages of network training, a regularization term is introduced to penalize the confidence of the network output, and the confidence penalty regularization term is expressed as:

[0127] R = -α(P) j )·logP j Where P j It is the class with the second highest confidence in predicting p, and it is considered that class j is likely the correct class for noisy instances when the label contains noise.

[0128]

[0129] Here, Γ is a hyperparameter of the confidence threshold, and y is the category corresponding to the true label. If the network is overconfident about class y, it will be subject to a larger α(P) threshold. j )punish;

[0130] Main learning phase:

[0131] By utilizing the proposed memory module, clean samples are selected based on the strategy of whether the historical predicted categories stored in the memory are consistent in the two most recent predictions. However, directly using these samples for training may ignore the contextual relationships between video segments. Therefore, LSTM is introduced to model the temporal dependencies between video segments and is used for network training in the current epoch. Meanwhile, the memory module is dynamically updated as training progresses. Since the network is relatively weak in the early learning stages, a regularization term is designed to prevent error accumulation from causing sample selection bias. The robustness of the model is improved by penalizing the confidence output distribution of the sample (x,y).

[0132]

[0133] Where α(P) k Calculated using the formula described above, the regularization term penalizes the model's confidence output by minimizing the expected loss for each class; furthermore, For the selected samples, the loss function in the main learning phase is expressed as:

[0134]

[0135] Where λ is a hyperparameter set by cross-validation.

[0136] Performance evaluation:

[0137] On the UCF-Crime dataset, the AUC of the method in this invention is 87.77%, which is better than the MGFN, UR-DMU and CLIP-TSA methods.

[0138] On the XD-Violence dataset, the AP of the method of this invention is 82.36%, which is also superior to other methods.

[0139]

[0140] Table 1 compares the results of this invention with different methods on two publicly available video anomaly detection datasets.

[0141] Note: The AUC (Area Under the Curve) and AP (Mean Precision) used in the table are important evaluation metrics for model classification performance. AUC reflects the overall classification ability of the model at different thresholds, and is usually the area under the ROC curve (the relationship between TruePositive Rate and False Positive Rate). The closer its value is to 1, the stronger the model's classification ability. AP represents the average precision at different recall rates, and is obtained by calculating the area under the PrecisionRecall curve. The larger the AP value, the more accurate the model's detection.

[0142] Table 1 compares the performance of different weakly supervised video anomaly detection methods. Based on the results, the following conclusions are drawn:

[0143] Overall, the method proposed in this invention outperforms current state-of-the-art methods on both publicly available datasets.

[0144] Compared to multi-instance learning-based methods (MGFN, 2023; UR-DMU, 2023), the method of this invention achieves a significant performance improvement. For example, on the UCF-Crime dataset, the method of this invention achieves an AUC of 87.77%; while on the XD-Violence dataset, the method of this invention achieves an AP of 82.36%, representing improvements of 2.25% and 0.70%, respectively. This result demonstrates that this invention effectively improves the performance of anomaly detection by generating frame-level pseudo-labels.

[0145] Compared to the self-trained pseudo-label method (CLIP-TSA, 2023), this invention achieves only a minor performance improvement, but its innovation lies in its first-time approach of learning from label noise and generating pseudo-labels in an end-to-end manner. Results show that this invention can effectively alleviate the error accumulation problem.

[0146] Summarize:

[0147] This invention effectively improves the performance of weakly supervised video anomaly detection by introducing textual descriptions of anomaly types, a label noise perception strategy, and an end-to-end online pseudo-label generation method.

[0148] Compared with other methods, the present invention achieves significant advantages in both AUC and AP metrics, demonstrating its effectiveness and robustness.

[0149] In addition, all components designed in this invention are general standard parts or components known to those skilled in the art. Their structures and principles can be learned by those skilled in the art through technical manuals or conventional experimental methods. They can be fully implemented by those skilled in the art, so there is no need to elaborate. The content protected by this invention does not involve improvements to the internal structure and methods.

[0150] The embodiments disclosed in this invention are preferred embodiments, but are not limited thereto. Those skilled in the art can easily understand the spirit of this invention based on the above embodiments and make different extensions and variations, but as long as they do not depart from the spirit of this invention, they are all within the protection scope of this invention.

Claims

1. A weakly supervised video anomaly detection method based on a label noise perception strategy, characterized in that, Includes the following steps: S1. Video feature extraction; First, 16 consecutive frames of the video are input as a segment into the feature extractor to generate a feature map. Then, anomaly-type text embeddings are embedded and matched with the corresponding video frame features. Based on the matching similarity, pseudo-labels are inferred. S2. Online generation of pseudo-tags; First, by inputting video features and text into the prediction module, similarity matching is calculated to generate an anomaly score map. Next, to enhance the propagation of contextual information and improve the quality of pseudo-label generation, video features are input into the knowledge propagation module. In this module, the expectation-maximization algorithm is used to extract representative segments, and the video features are updated using a random walk algorithm. Finally, the updated video features are input into the prediction module again to generate the final pseudo-labels. ; S3, supervised learning of networks; Design a network with a multi-stage training strategy, where training is divided into two stages: a warm-up stage and a main learning stage. Preheating phase: To avoid overfitting to noisy labels in the early stages of network training, a regularization term is introduced to penalize the confidence of the network output, and the confidence penalty regularization term R is expressed as: ， in, Representative of the sample When making predictions, the probability value corresponding to the category with the second highest confidence is output. If the label contains noise, category j is considered to be the true category of the noisy sample. ， in These are the adaptive penalty weighting coefficients for noise labels. is the hyperparameter of the confidence threshold, and y is the true label of the corresponding sample. This refers to the category predicted by the current model. When the network is confident about class y, it will be affected. punish; Main learning phase: By utilizing the proposed memory bank module, clean samples are selected based on the strategy of whether the historical prediction categories stored in the memory bank are consistent in the two most recent predictions. Directly using these samples for training may ignore the contextual relationships between video segments. Therefore, LSTM is introduced to model the temporal dependencies between video segments and is used for network training in the current epoch. At the same time, the memory bank module is dynamically updated as training progresses. A regularization term was designed. To sample The confidence output distribution is penalized to improve the robustness of the model: ， in Using the formula mentioned above The calculation is as follows: K is the number of types, and k is the number of categories predicted by the model. This represents the probability corresponding to the predicted class by the model. The regularization term penalizes the model's confidence output by minimizing the expected loss for each class. Furthermore, For the selected samples, the loss function in the main learning phase is expressed as: ， in Let cross-entropy be the loss function. This represents the final pseudo-label generated by the model. This represents the true label of the corresponding sample. This is the loss weight hyperparameter.

2. The weakly supervised video anomaly detection method based on a label noise perception strategy as described in claim 1, characterized in that, The feature extractor in S1 is a CLIP image encoder.

3. The weakly supervised video anomaly detection method based on a label noise perception strategy as described in claim 1, characterized in that, The feature map size in S1 is 32×256×512, and the batch size is 32.

4. The weakly supervised video anomaly detection method based on a label noise perception strategy as described in claim 1, characterized in that, The text information of the abnormal type in S1 is encoded as a 32×256×14 text embedding, and there are 14 abnormal types in total.

5. The weakly supervised video anomaly detection method based on a label noise perception strategy as described in claim 1, characterized in that, In the online generation step of S2 pseudo-labels, the loss function is used. To compensate for the lack of fine-grained supervision information, among which For the final pseudo-tag, The labels are processed versions of the anomaly score map generated by the prediction module.

Images