A data imbalanced weakly supervised video anomaly detection method and system

Abstract

The application discloses a kind of data imbalance weak supervision video anomaly detection method and system, including extracting the video feature of the video segment level of the video to be measured;Video segment level video feature is obtained by the segment level anomaly score of the confrontation training module;Video segment level video feature is obtained by the anomaly score of the segment level of the focusing training module;The anomaly score after fusion is obtained by fusing segment level anomaly score and anomaly score;Again compared with threshold value, all frames in the video segment greater than threshold value are regarded as anomaly, and all frames in the video segment less than threshold value are regarded as normal, to realize data imbalance weak supervision video anomaly detection and the time positioning of anomaly.The anomaly score of segment level obtained by fusing confrontation training module and focusing training module makes it easier to learn the difference between normal and abnormal video segments, and the detection of abnormal events is closer to reality, thereby improving the accuracy of video anomaly detection.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, specifically relating to a method and system for detecting anomalies in unbalanced, weakly supervised video. Background Technology

[0002] Video anomaly detection technology is applied in autonomous monitoring and is an important method for monitoring abnormal events. Video anomalies typically refer to abnormal appearances or motion properties in a video, or normal appearances or motion properties occurring at abnormal times or locations. The task of video anomaly detection is to detect temporal and spatial anomalies in the video.

[0003] Based on the presence or absence of labels on the sample data, existing technologies are generally divided into two categories. One is unsupervised video anomaly detection technology, which typically only models the distribution of normal samples and treats video frames or segments that deviate from the normal sample distribution as anomalies during testing. The other is weakly supervised video anomaly detection technology, which uses only video-level labels during training. It is usually regarded as a standard multiple instance learning problem, where each video can be regarded as a bag and each segment in the video can be regarded as an instance. If at least one anomalous frame is detected in the video, the video is marked as anomaly; otherwise, it is marked as normal.

[0004] While numerous advanced weakly supervised video anomaly detection techniques exist, these tasks implicitly rely on the imbalance between normal and anomalous frames within a video. They focus primarily on finding representative video segments or extracting temporal relationships between segments, neglecting a crucial issue: the inherent data imbalance between normal and anomalous videos. For example, the commonly used ShanghaiTech dataset contains 238 videos during training, with 175 normal and 63 anomalous—nearly one-third of the normal videos. In reality, the number of anomalous videos is far less than that of normal videos. If the ShanghaiTech dataset only slightly considers this imbalance during training, then the commonly used benchmark datasets UCF-Crime and XD-Violence set the ratio of normal to anomalous videos to approximately 1:1 during training. This also potentially leads current weakly supervised video anomaly detection tasks to ignore the inherent data imbalance between videos.

[0005] With the rapid increase in the number of surveillance cameras in recent years, anomaly detection in video surveillance has received increasing attention. Due to the difficulty of labeling massive amounts of surveillance video, obtaining video-level labels is relatively easier than obtaining frame-level labels. Therefore, weakly supervised methods that only label video-level images have become an important means of anomaly detection. However, previous weakly supervised video anomaly detection tasks implicitly rely on the imbalance between normal and abnormal frames within the video, focusing more on how to find representative video segments or how to obtain the temporal relationship between video segments, while neglecting an important issue: the inherent data imbalance between normal and abnormal videos, thus affecting the detection effect of anomalies. Summary of the Invention

[0006] To overcome the problems in the prior art, the present invention aims to provide a method and system for detecting unbalanced, weakly supervised video anomalies. This method can enhance the quantity and focus attention of abnormal videos, and fuse them to accurately achieve the detection effect of abnormal events at the video frame level.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for detecting anomalies in imbalanced, weakly supervised videos includes the following steps:

[0009] Extract video features at the video segment level from the video to be tested;

[0010] Video segment-level features are used to obtain segment-level anomaly scores through an adversarial training module;

[0011] Video segment-level features are used to obtain segment-level anomaly scores through a focused training module;

[0012] The segment-level anomaly scores obtained through the adversarial training module and the fragment-level anomaly scores obtained through the focused training module are fused to obtain the fused anomaly score.

[0013] The fused anomaly score is compared with a threshold. All frames in video segments with scores greater than the threshold are considered anomalies, while all frames in video segments with scores less than the threshold are considered normal. This enables unbalanced, weakly supervised video anomaly detection and temporal localization.

[0014] Furthermore, video segment-level features of the video under test are extracted using the I3D network.

[0015] Furthermore, extracting video segment-level features from the video to be tested specifically includes the following steps:

[0016] X each video i Divided into N consecutive and non-overlapping time segments X i={x i1 ,...,x iN}, x i1 For the first time segment, x iN For the Nth time segment, video X i All segments are categorized by video level tags. This can be formed into a positive or negative packet, where the positive packet represents video X. i There is at least one anomalous time segment in the video, and the negative packet represents video X. i All time segments are normal;

[0017] Then take each video X i As input to the I3D network, the output of the I3D network is the original spatiotemporal feature Z. i ={z i1 ,...,z iN}∈R N×D Each video X i Corresponding to an original spatiotemporal feature Z i ={z i1 ,...,z iN}∈R N×D This completes the extraction of video features; where N is the number of video segments, D is the dimension of each segment's feature, and z i1 For the first time segment x i1 The corresponding original spatiotemporal features, z iN For the Nth time segment x iN The corresponding original spatiotemporal features are represented by R, which is an N×D dimensional matrix vector.

[0018] Furthermore, the total loss of the adversarial training module is as follows:

[0019] L AT =λL clean +(1-λ)L adv (11)

[0020] In the formula, L AT Let L be the total loss function of the adversarial training module, λ be a parameter that adjusts the weights between the clean video loss and the adversarial example loss, and L be the total loss function of the adversarial training module. clean L is the total loss function for clean video. adv This is the total loss function for adversarial examples.

[0021] Furthermore, the total loss function for clean videos is as follows:

[0022]

[0023] In the formula, L clean Let be the total loss function for the clean video, and let α, β, and γ be different weight parameters. For hinge loss in clean video, For the sparse loss of clean video, The smoothness loss of a clean video.

[0024] Furthermore, the total loss function for adversarial examples is as follows:

[0025]

[0026] In the formula, L adv Let α, β, and γ be the total loss function for adversarial examples, and let α, β, and γ be different weight parameters. To combat hinge loss in samples, To combat the sparsity loss of samples, To combat sample smoothing loss To combat the cross-entropy loss of samples.

[0027] Furthermore, the loss function for the focused training module is as follows:

[0028]

[0029] In the formula, L FT To focus on the total loss function of the training module, α, β, and γ are different weight parameters, For hinge loss in clean video, For the sparsity loss of clean video, For the smooth loss of clean video, L CS For the sake of sensitive losses.

[0030] Furthermore, the cost-sensitive loss function is as follows:

[0031] L CS =-ω(1-y c ) τ ylog(y c )-(1-ω)(y c ) τ (1-y)log(1-y c (12)

[0032] In the formula, L CS For cost-sensitive loss, ω is a weighting coefficient that controls the ratio of loss between abnormal and normal videos, (1-y c ) τ Weighting factors are used to control the proportion of video loss for difficult and easy anomalies.

[0033] Furthermore, the merged outlier score is calculated using the following formula:

[0034] s fusion =sA T (if|sAT -c|≥|s FT -c|), otherwise s fusion =s FT (14)

[0035] Where c is the threshold for determining abnormal scores in adversarial training and focused training, and s fusion It is the anomalous score after fusion, s AT For video clip-level anomaly scores obtained through the adversarial training module, s FT The anomaly score at the video clip level is obtained through the focused training module.

[0036] A weakly supervised video anomaly detection system based on imbalanced data includes:

[0037] The video feature extraction module is used to extract video features at the video segment level from the video to be tested and send them to the segment-level anomaly score acquisition module and the segment-level anomaly score acquisition module.

[0038] The segment-level anomaly score acquisition module is used to obtain segment-level anomaly scores from video segment-level video features through the adversarial training module and send them to the fusion module;

[0039] The segment-level anomaly score acquisition module is used to obtain segment-level anomaly scores from video features at the segment level through the focus training module and send them to the fusion module;

[0040] The fusion module is used to fuse the segment-level anomaly scores obtained through the adversarial training module with the fragment-level anomaly scores obtained through the focus training module to obtain the fused anomaly score, which is then sent to the comparison module.

[0041] The comparison module compares the fused anomaly score with a threshold, treating all frames in video segments with scores above the threshold as anomalies and all frames in video segments with scores below the threshold as normal, thus achieving unbalanced, weakly supervised video anomaly detection and temporal localization.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] This invention expands the number of anomalous videos by generating adversarial examples from an adversarial training module. By mitigating the imbalance in the number of videos, it generates a fine-tuned decision boundary, enabling the detection model to correctly classify adversarial examples. On the other hand, it adjusts the loss cost between anomalous and normal videos through a focused training module, allowing the detection model to focus on identifying anomalous and difficult-anomaly videos (difficult-anomaly videos are those with anomalous scores close to the decision boundary, while easy-anomaly videos are those with anomalous scores but far from the decision boundary). Finally, the scores are fused, and the fused anomalous scores are compared with a threshold. All frames in video segments with scores above the threshold are considered anomalous, and all frames in video segments with scores below the threshold are considered normal. This achieves unbalanced, weakly supervised video anomaly detection and temporal localization. By fusing the segment-level anomaly scores obtained from the adversarial and focused training modules, this invention makes it easier to learn the differences between normal and anomalous video segments, resulting in more realistic anomaly detection and improved video anomaly detection accuracy.

[0044] Furthermore, this invention uses only video-level tags to automatically achieve frame-level anomaly event detection, which has high detection efficiency and overcomes the problem that traditional video anomaly detection methods usually rely on manual viewing and identification of anomaly events, resulting in inaccurate detection. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the unbalanced weakly supervised video anomaly detection method of the present invention;

[0046] Figure 2 This is a detailed diagram of the framework of the unbalanced weakly supervised video anomaly detection method of the present invention;

[0047] Figure 3 Frame detail diagrams generated for adversarial examples of anomalous videos;

[0048] Figure 4 The results are the test scores for abnormal videos in the ShanghaiTech dataset. (a) is test video 08_0158, and (b) is test video 02_0161.

[0049] Figure 5 This is a schematic diagram of a weakly supervised video anomaly detection system with unbalanced data. Detailed Implementation

[0050] The present invention will now be described in detail with reference to the accompanying drawings.

[0051] To alleviate the imbalance between the number of normal and abnormal videos in the real world, this invention proposes a weakly supervised video anomaly detection method based on imbalanced data, namely, adversarial and focused joint training. This training strategy consists of two independent modules: a data-based adversarial training module, which first generates adversarial examples of abnormal videos in the latent space, and then trains the detection model using feature sets of normal videos, abnormal videos, and their adversarial examples, thus expanding the abnormal video pool from a data perspective; and a model-based focused training module, which modifies the cross-entropy loss function by adjusting the classification costs of normal and abnormal videos and the classification costs of easy and difficult samples within abnormal videos. This allows the model to focus on identifying the fewer abnormal videos and the relatively difficult samples within those videos, thus achieving focused identification of abnormal and difficult-to-identify videos from a model perspective. Furthermore, data-based adversarial training and model-based focused training can be seen as two different ways to address the imbalance between normal and abnormal video data. Therefore, a natural approach is to fuse the two modules. This invention employs a simple and effective post-fusion strategy, which involves fusing the anomaly scores obtained from the two modules at different score levels.

[0052] See Figure 1 and Figure 2 This invention provides a data-imbalanced weakly supervised video anomaly detection method based on an adversarial and focused joint training mechanism, from... Figure 1 As can be seen, due to the scarcity of anomalous videos, the original decision boundary is prone to misclassifying some anomalous videos, which severely affects the classification performance of the detection model. To alleviate this problem, on the one hand, an adversarial training module based on data is used to generate adversarial examples of anomalous videos to expand the number of anomalous videos. By alleviating the imbalance in quantity, a fine-tuned decision boundary is generated, enabling the detection model to correctly classify adversarial examples. On the other hand, a model-based focused training module is proposed. By adjusting the loss cost between anomalous and normal videos, the detection model focuses on identifying anomalous videos and difficult-annomous videos (difficult-annomous videos are those with anomalous scores that are close to the decision boundary, while easy-annomous videos are those with anomalous scores that are far from the decision boundary). After training the complete framework, a simple and effective post-fusion strategy is adopted to combine the anomalous scores obtained from the adversarial training and focused training in the testing phase to obtain the final decision boundary, thereby achieving anomalous event detection and temporal localization. See [link to relevant documentation] Figure 2 The unbalanced, weakly supervised video anomaly detection method of the present invention comprises four parts: video feature extraction, adversarial training module, focused training module, and post-fusion strategy. The specific process is as follows:

[0053] (1) Extracting video features

[0054] Weakly supervised video anomaly detection refers to achieving anomaly detection and temporal localization solely using video-level tags. Assume this invention is given a set of training videos. and their corresponding video level tags X i For videos, i is the video number, M is the number of videos, and y is the video sequence number. i For X i The corresponding video-level tags, for each video X i First, each video X i Divide into N consecutive and non-overlapping time segments, i.e., X i ={x i1 ,...,x iN}, x i1 For the first time segment, x iN For the Nth time segment, video X i All segments are categorized by video level tags. Form a positive or negative bag, positive bag (y i =1) indicates video X i There is at least one anomalous time segment in the data, and the negative packet (y) i =0) indicates video X i All time segments are normal.

[0055] This invention uses each video X i As input to the I3D network, the output of the I3D network is the original spatiotemporal feature Z. i ={z i1 ,...,z iN}∈R N×D Each video X i Corresponding to an original spatiotemporal feature Z i ={z i1 ,...,z iN}∈R N×D This means extracting video features; where N represents the number of video segments, D represents the dimension of the features in each segment, and z i1 For the first time segment x i1 The corresponding original spatiotemporal features, z iN For the Nth time segment x iN The corresponding original spatiotemporal features are represented by R, which is an N×D dimensional matrix vector.

[0056] (2) Obtain video-level anomaly scores through the adversarial training module;

[0057] This invention combines all original spatiotemporal feature sets This is called a clean feature, where {Z} n} represents all normal video features, {Z a} represents all anomalous video features. Due to the scarcity of anomalous videos, this invention uses a latent space-based projective gradient descent method to generate adversarial examples {Z′} for anomalous videos. a Normal video features remain unchanged. Due to the adversarial examples {Z′} of anomalous videos... a The generation process of adversarial examples {Z′} is related to the network structure of the adversarial training module. The following sections will introduce the adversarial examples {Z′}. a The generation process of}.

[0058] Since the video features extracted by the pre-trained I3D network, i.e., the original spatiotemporal features, lack sufficient discriminative power, this invention proposes a multi-scale temporal network f φ The multi-scale time network f φ With the original spatiotemporal feature set Adversarial examples {Z′ from abnormal videos a As input, it outputs the corresponding multi-scale temporal feature sets with the same dimension. Multi-scale temporal feature set {T′ of adversarial examples from anomalous videos a},in Original spatiotemporal characteristics The multi-scale temporal features output after passing through the multi-scale temporal network fφ, {T′ a} represents the original spatiotemporal features {Z′} of the adversarial examples in the anomalous video. a After passing through a multi-scale time network f φ The output is a multi-scale temporal feature set. Specifically, this multi-scale temporal network f φ A self-attention mechanism and dilated convolution operation are applied to various time segments of the video to obtain a multi-scale temporal feature set. Multi-scale temporal feature set {T′ of adversarial examples from anomalous videos a}. Among them, T n For multi-scale temporal features of normal videos, T a This invention utilizes multi-scale temporal features of anomalous videos. To enhance the discriminative power of these features, it maximizes the differences between the multi-scale temporal feature sets of normal and anomalous videos, as well as the differences between the multi-scale temporal feature sets of adversarial examples of normal and anomalous videos. The hinge loss method employed in this invention is as follows:

[0059]

[0060]

[0061] Among them, y n and y a These represent the corresponding video-level labels, where m is a predefined threshold and g is a video-level label. k (Tn ), g k (T a ) and g k (T′ a These are the multi-scale temporal features T of the abnormal video. a Multi-scale temporal features T of normal videos n Multi-scale temporal features T′ of adversarial examples a The norm mean of the k segments with the largest l2-norm. and Hinge loss for clean videos and hinge loss for adversarial examples, respectively.

[0062] When the present invention obtains the multi-scale temporal feature set {T} of a normal video i} and the multi-scale temporal feature set {T′ of anomalous videos a After that, use a fragment-level classifier. The anomaly scores of regressed segments are calculated, and the anomaly scores of the top-k video segments are weighted and averaged to obtain video-level anomaly scores. These video-level anomaly scores are then compared with their corresponding video-level labels. The error between the two (i.e., the cross-entropy loss between clean videos and adversarial examples) is used as a component of the overall loss function of the adversarial training module. Training is completed by optimizing the overall loss function of the adversarial training module.

[0063] At this point, the network structure of the adversarial training module has been introduced, and the adversarial examples Z' of the abnormal videos have been presented. a The generation process is as follows Figure 3 As shown. Specifically, for the sake of formula simplicity, the multi-scale time network f is used here. φ With fully connected networks Write them together using F θ It means, that is Where Ω k Representing multi-scale time features T a The k segments with the largest l2-norm are identified. In this invention, an adversarial example Z' is generated using a latent space-based projection gradient descent method. a The formula for generating the formula is as follows:

[0064]

[0065] Among them, (Z' a ) t+1 and (Z' a ) t The adversarial examples generated in the (t+1)th and tth iterations are respectively, (Z' a ) 0 =Z a For clean anomaly features as input, L(θ,Z) a ,ya )=-y a log(F θ (Z a ))-(1-y a log(1-F) θ (Z a )), where η is the step size for updating the anti-perturbation, and y a It is an abnormal video Z a The label is θ, which is the network parameter. In this invention, (Z') has undergone t+1 iterations. a ) t+1 This is the final adversarial example Z' of the anomalous video. a .

[0066] Considering the sparsity of anomalous events, this invention uses sparse loss in both clean features and adversarial examples, as follows:

[0067]

[0068]

[0069] In the formula, and These represent the sparsity loss for clean videos and the sparsity loss for adversarial examples, respectively. ij and t' aj These are the multi-scale temporal features of clean video clips and adversarial example clips, respectively. It is a video segment-level classifier used to regress the anomaly scores of segments.

[0070] Since the outlier scores between video segments should be smooth, this invention uses smoothness loss in clean features and adversarial examples, as follows:

[0071]

[0072]

[0073] In the formula, and These represent the smoothing loss for clean videos and the smoothing loss for adversarial examples, respectively.

[0074] The classification loss is as follows:

[0075]

[0076]

[0077] In the formula, and The cross-entropy loss is calculated for clean videos and adversarial examples, respectively, where y is the true label value of the video. c and y adv These are the predicted anomaly scores for clean videos and adversarial examples, respectively.

[0078] The overall loss function for clean features is as follows:

[0079]

[0080] In the formula, L clean Let be the total loss function for the clean video, and α, β, and γ be different weight parameters.

[0081] The overall loss function for adversarial examples is as follows:

[0082]

[0083] In the formula, L adv Let α, β, and γ be the total loss function for adversarial examples, and let α, β, and γ be different weight parameters.

[0084] The overall loss of the entire adversarial training module is defined as follows:

[0085] L AT =λL clean +(1-λ)L adv (11)

[0086] In the formula, L AT Let λ be the total loss function of the adversarial training module, and λ be a parameter that adjusts the weights between the clean video loss and the adversarial example loss.

[0087] During the training phase, training is completed when the total loss function of the adversarial training module converges to a minimum, resulting in the trained adversarial training module.

[0088] During the testing phase, the video to be detected is first processed segment by segment to extract segment-level features, and then the segment features are input into the pre-trained adversarial training module to obtain the segment-level anomaly score of the video.

[0089] (3) Obtain video-level anomaly scores through the focused training module;

[0090] The focused training module, based on the cross-entropy loss function, adds a coefficient to control the weights of abnormal and normal videos, and a factor to control the weights of difficult and easy anomalies. This ensures that the penalty for misclassifying an abnormal video is greater than that for misclassifying a normal video, and that the penalty for misclassifying a difficult anomaly is greater than that for misclassifying an easy anomaly. Specifically, this invention uses the spatiotemporal features {Z} of normal videos... n} and abnormal video features {Z a} is used as input, and then fed into a multi-scale temporal network to obtain the corresponding multi-scale temporal features {T}. n} and {T a}, T n For multi-scale temporal features of normal videos, T a The multi-scale temporal features of the abnormal video are then fed into a segment-level classifier. The anomaly scores of the regressed segments are calculated, and the anomaly scores of the top-k video segments are weighted and averaged to obtain the video-level anomaly scores. The video-level anomaly scores are compared with the corresponding video-level labels, and the error between the two (i.e., cost-sensitive loss) is used as a term in the total loss function of the focused training module. The training of the focused training module is achieved by optimizing formula (13).

[0091] The cost-sensitive loss function is as follows:

[0092] L CS =-ω(1-y c ) τ ylog(y c )-(1-ω)(y c ) τ (1-y)log(1-y c (12)

[0093] In the formula, L CS For cost-sensitive loss, ω is a weighting coefficient that controls the ratio of loss between abnormal and normal videos, (1-y c ) τ Weighting factors are used to control the proportion of video loss for difficult and easy anomalies.

[0094] The overall loss of the entire focused training module is as follows:

[0095]

[0096] In the formula, L FT The total loss function for focusing the training module is defined by α, β, and γ, which are different weight parameters.

[0097] (4) Fusion is performed through a post-fusion strategy to achieve anomaly detection in unbalanced, weakly supervised videos;

[0098] Since the data-based adversarial training module in step (2) and the model-based focus training module in step (3) can be seen as two different ways to alleviate the imbalance between normal and abnormal video data, this invention proposes a simple and effective post-fusion strategy. The segment-level anomaly score of the video obtained through the adversarial training module and the fragment-level anomaly score of the video obtained through the focus training module are fused at the score level using the following formula:

[0099] s fusion =s AT (if|s AT -c|≥|s FT -c|), otherwise s fusion =s FT (14)

[0100] Where c is the threshold for determining abnormal scores in adversarial training and focused training, and s fusion It is the anomalous score after fusion, s AT For video clip-level anomaly scores obtained through the adversarial training module, s FT The anomaly score at the video clip level is obtained through the focused training module.

[0101] During the training process, the training process of the entire model is completed by optimizing equations (11) and (13). During the testing process, the video to be detected is scored by two trained modules according to the video segments. Then, the abnormal scores of the two modules are fused by equation (14) to obtain the final abnormal score, thereby realizing the detection and time localization of abnormalities.

[0102] This invention integrates data-based adversarial training and model-based focused training, which can effectively alleviate the problem of data imbalance between videos.

[0103] (5) Evaluate the performance of the trained model using test data.

[0104] This invention is tested on the ShanghaiTech anomaly detection benchmark dataset. Since this invention addresses a problem that has been neglected in the field of weakly supervised video anomaly detection, the original benchmark anomaly detection dataset is adjusted to an imbalanced data experiment setting (this invention reduces the number of anomalous videos in the original dataset to 1 / 5 of the original, while keeping the number of normal videos unchanged).

[0105] This invention was ultimately implemented in the ShanghaiTech dataset ( Figure 4 Taking videos 08_0158 and 02_0161 as examples, the frame-level accuracy reached 97.92% in the test.

[0106] To demonstrate the model's evaluation performance, a test video is input. First, the video is divided into several consecutive, non-overlapping video segments. Then, a pre-trained adversarial training module and a focus training module are used to evaluate the anomaly score of each video segment. Finally, the anomaly scores from the two modules are fused together, as shown in the following figure. Figure 4As shown in (a) and (b), the broken line represents the predicted score, and the shaded background represents the true label. The closer the broken line value corresponding to the shaded part is to 1, the higher the accuracy of the model. It can be seen that the present invention can accurately predict the true abnormal parts of the video, and the time location matches the true label, which shows the accuracy of the detection method of the present invention (the inset in the figure is a video frame randomly selected from the abnormal video to visualize the abnormal event and illustrate the accuracy of the model).

[0107] See Figure 5 A weakly supervised video anomaly detection system based on imbalanced data, comprising:

[0108] The video feature extraction module is used to extract video features at the video segment level from the video to be tested and send them to the segment-level anomaly score acquisition module and the segment-level anomaly score acquisition module.

[0109] The segment-level anomaly score acquisition module is used to obtain segment-level anomaly scores from video segment-level video features through the adversarial training module and send them to the fusion module;

[0110] The segment-level anomaly score acquisition module is used to obtain segment-level anomaly scores from video features at the segment level through the focus training module and send them to the fusion module;

[0111] The fusion module is used to fuse the segment-level anomaly scores obtained through the adversarial training module with the fragment-level anomaly scores obtained through the focus training module to obtain the fused anomaly score, which is then sent to the comparison module.

[0112] The comparison module compares the fused anomaly score with a threshold, treating all frames in video segments with scores above the threshold as anomalies and all frames in video segments with scores below the threshold as normal, thus achieving unbalanced, weakly supervised video anomaly detection and temporal localization.

[0113] This invention is the first to point out that current weakly supervised video anomaly detection tasks ignore the inherent imbalance problem between normal and anomalous video data. This invention alleviates the imbalance problem by using a data-based adversarial training module and a model-based focused training module, respectively, and then achieves high anomaly detection accuracy by fusing the two modules. Specifically, it has the following advantages:

[0114] (1) The problem of data imbalance between videos. In weakly supervised video anomaly detection tasks, since existing benchmark datasets almost all set the same number of normal and abnormal videos, this important problem is ignored. This invention points out this problem for the first time and provides a solution.

[0115] (2) High detection efficiency. Traditional video anomaly detection methods usually rely on manual viewing and identification of abnormal events. This invention uses only weak tags at the video level to automatically realize frame-level anomaly event detection, which has high detection efficiency.

[0116] (3) High detection accuracy. This invention integrates a data-based adversarial training module and a model-based focused training module, making it easier for the model to learn the differences between normal and abnormal video segments. The model's detection of abnormal events is closer to reality, thereby improving the model's accuracy.

Claims

1. A method for detecting anomalies in imbalanced, weakly supervised video data, characterized in that, Includes the following steps: Extract video features at the video segment level from the video to be tested; Video segment-level features are used to obtain segment-level anomaly scores through an adversarial training module; Video segment-level features are used to obtain segment-level anomaly scores through a focused training module; The focused training module is based on the cross-entropy loss function, by adding a coefficient to control the weights of abnormal and normal videos, and adding a factor to control the weights of difficult and easy-to-abnormal videos. With normal video spatiotemporal characteristics and abnormal video features As input, it is then fed into a multi-scale temporal network to obtain the corresponding multi-scale temporal features. and , Multi-scale temporal features of normal video. The multi-scale temporal features of the abnormal video are then fed into a segment-level classifier. The anomaly scores of the regressed segments are calculated, and the anomaly scores of the top-k video segments are weighted and averaged to obtain the video-level anomaly scores. The segment-level anomaly scores obtained through the adversarial training module and the fragment-level anomaly scores obtained through the focused training module are fused to obtain the fused anomaly score. The fused anomaly score is compared with a threshold. All frames in video segments with scores greater than the threshold are considered anomalies, while all frames in video segments with scores less than the threshold are considered normal. This enables unbalanced, weakly supervised video anomaly detection and temporal localization.

2. The method for detecting anomalies in unbalanced, weakly supervised video data according to claim 1, characterized in that, Video features at the video segment level are extracted from the video to be tested using the I3D network.

3. The method for detecting anomalies in unbalanced, weakly supervised video data according to claim 1, characterized in that, Extracting video segment-level features from the video to be tested specifically includes the following steps: Each video Divided into A series of non-overlapping time segments , For the first time segment, For the Nth time segment, the video All segments are categorized by video level tags. This can be formed into a positive or negative packet, where the positive packet represents the video. There is at least one anomalous time segment in the video; the negative packet indicates that the video contains an anomalous time segment. All time segments are normal; Then for each video As input to the I3D network, the output of the I3D network is the original spatiotemporal features. Each video Corresponding to an original spatiotemporal feature This completes the extraction of video features; among them, Number of video clips The dimension of each fragment feature, For the first time segment The corresponding original spatiotemporal characteristics, For the Nth time segment The corresponding original spatiotemporal features, R = N A D-dimensional matrix vector.

4. The method for detecting anomalies in unbalanced, weakly supervised video according to claim 1, characterized in that, The total loss of the adversarial training module is as follows: （11） In the formula, The total loss function for the adversarial training module. To adjust the parameters of the weights between the clean video loss and the adversarial example loss, Let the total loss function be the clean video. This is the total loss function for adversarial examples.

5. The method for detecting anomalies in unbalanced, weakly supervised video according to claim 4, characterized in that, The total loss function for clean video is as follows: （9） In the formula, Let the total loss function be the clean video. , and For different weight parameters, For hinge loss in clean video, For the sparse loss of clean video, The smoothness loss of a clean video.

6. The method for detecting anomalies in unbalanced, weakly supervised video according to claim 4, characterized in that, The total loss function for adversarial examples is as follows: （10） In the formula, The total loss function for adversarial examples, , and For different weight parameters, To combat hinge loss in samples, To combat the sparsity loss of samples, To combat sample smoothing loss To combat the cross-entropy loss of samples.

7. The method for detecting anomalies in unbalanced, weakly supervised video according to claim 4, characterized in that, The loss function for the focused training module is as follows: （13） In the formula, To focus on the total loss function of the training module, , and For different weight parameters, For hinge loss in clean video, For the sparsity loss of clean video, For the smooth loss of clean video, For the sake of sensitive losses.

8. The method for detecting anomalies in unbalanced, weakly supervised video according to claim 7, characterized in that, The cost-sensitive loss function is as follows: （12） In the formula, For the sake of sensitive losses, To control the weighting coefficients for the loss ratio of abnormal and normal videos, Weighting factors are used to control the proportion of video loss for difficult and easy anomalies.

9. The method for detecting anomalies in unbalanced, weakly supervised video according to claim 1, characterized in that, The merged outlier score is calculated using the following formula. ,otherwise (14) Where c is the threshold for determining abnormal scores in adversarial training and focused training. These are the anomalous scores after fusion. The video clip-level anomaly score is obtained through the adversarial training module. The anomaly score at the video clip level is obtained through the focused training module.

10. A weakly supervised video anomaly detection system based on imbalanced data, characterized in that, include: The video feature extraction module is used to extract video features at the video segment level from the video to be tested and send them to the segment-level anomaly score acquisition module and the segment-level anomaly score acquisition module. The segment-level anomaly score acquisition module is used to obtain segment-level anomaly scores from video segment-level video features through the adversarial training module and send them to the fusion module; The segment-level anomaly score acquisition module is used to obtain segment-level anomaly scores from video features at the segment level through the focus training module and send them to the fusion module; The fusion module is used to fuse the segment-level anomaly scores obtained through the adversarial training module with the fragment-level anomaly scores obtained through the focus training module to obtain the fused anomaly score, which is then sent to the comparison module. The focused training module, based on the cross-entropy loss function, adds a coefficient to control the weights of abnormal and normal videos, and adds a factor to control the weights of videos with difficult and easy anomalies; using the spatiotemporal features of normal videos... and abnormal video features As input, it is then fed into a multi-scale temporal network to obtain the corresponding multi-scale temporal features. and , Multi-scale temporal features of normal video. The multi-scale temporal features of the abnormal video are then fed into a segment-level classifier. The anomaly scores of the regressed segments are calculated, and the anomaly scores of the top-k video segments are weighted and averaged to obtain the video-level anomaly scores. The comparison module compares the fused anomaly score with a threshold, treating all frames in video segments with scores above the threshold as anomalies and all frames in video segments with scores below the threshold as normal, thus achieving unbalanced, weakly supervised video anomaly detection and temporal localization.

Images