A Weakly Supervised Video Violence Detection Method Based on Temporal Augmentation and Contrastive Learning

By employing temporal augmentation and contrastive learning methods, the problems of weak temporal learning ability and label mismatch in video violence detection are addressed, thereby improving the feature representation and detection accuracy of video frames and achieving better generalization ability.

CN118015507BActive Publication Date: 2026-11-10XIAN UNIV OF TECH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410081573.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-19
Publication Date
2026-11-10
Estimated Expiration
2044-01-19

AI Technical Summary

Technical Problem

Existing video violence detection methods suffer from weak temporal learning capabilities and mismatch between video labels and frame labels in weakly supervised learning, resulting in insufficient generalization ability.

Method used

We employ a method based on temporal augmentation and contrastive learning. By enhancing temporal features, we improve the ability to learn temporal context. Combined with contrastive learning with feature amplification and self-distillation optimization training, we enhance the feature representation and discriminativeness of video frames and optimize the frame classifier.

Benefits of technology

It improves the accuracy and generalization ability of video violence detection, enhances the feature representation of video frames, improves the matching degree between video labels and frame labels, and improves the detection effect of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118015507B_ABST
    Figure CN118015507B_ABST
Patent Text Reader

Abstract

This invention discloses a weakly supervised video violence detection method based on temporal enhancement and contrastive learning, specifically including a video feature extraction module, a video feature fusion module, a temporal feature enhancement module, a contrastive learning module based on feature amplification, and a self-distillation training module. This invention addresses the problems of weak temporal learning capability and mismatch between video labels and frame labels in weakly supervised learning in existing technologies, thereby improving detection performance. This invention uses a temporal feature enhancement method to strengthen the learning of video frames near and far from the target frame, improving the feature representation of video frames. Simultaneously, it employs a contrastive learning method based on feature amplification to enhance the discriminativeness of video frames and introduces a self-distillation learning strategy to optimize the model training process, thereby improving the video violence detection effect and generalization ability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and video understanding technology, and relates to a weakly supervised video violence detection method based on temporal augmentation and contrastive learning. Background Technology

[0002] Video violence detection aims to determine whether a given video contains violent scenes and to pinpoint the start and end times of these scenes. This technology can be used not only for content moderation of internet videos but also for video surveillance in sensitive locations within intelligent security systems, possessing significant theoretical and practical value. Due to the difficulty and time-consuming nature of video frame annotation, unsupervised and weakly supervised methods are commonly used in violence detection research.

[0003] Unsupervised video violence detection methods typically use training sets containing only normal, non-violent videos, omitting anomalous violent videos. They learn a model paradigm of normal behavior from non-violent videos and then classify video frames that deviate significantly from this paradigm as exhibiting anomalous violence. However, real-world non-violent scenarios are diverse and difficult to fully cover in the training set. This leads to the tendency to misclassify newly appearing non-violent video frames as anomalous, resulting in weak generalization ability and hindering practical applications.

[0004] The training set for weakly supervised video violence detection methods includes not only normal, non-violent videos but also anomalous violent videos, along with category labels indicating whether the videos contain violent behavior. Compared to strongly supervised violence detection algorithms, weakly supervised methods do not require labeling every frame of the video, reducing labeling costs. Furthermore, compared to unsupervised violence detection methods, weakly supervised methods can achieve better category differentiation by incorporating label knowledge. Therefore, video violence detection methods based on weakly supervised learning have gained increasing attention.

[0005] With the development of deep learning, weakly supervised video violence detection using deep neural networks has made significant progress, but some problems still exist: 1) Weak temporal learning ability. Videos are ordered continuous frames composed of a series of images, and there is a strong temporal relationship between video frames. Understanding the content of a video frame depends not only on the main information of the current frame, but also on the changes in adjacent frames and the background information provided by other frames that are far away. Focusing only on the content of a single target frame may not be able to obtain an accurate description of the video frame. 2) Mismatch between video labels and frame labels. The labeling method of weak supervision is that videos containing violent scenes are violent videos, and otherwise are non-violent videos, with only video labels and no frame labels. However, as can be seen from real-world scenarios, violent segments in violent videos often appear for a short time, so the number of non-violent video frames is much greater than the number of violent video frames, making it difficult to establish a relationship between video labels and video frames. Therefore, to address the above problems, this invention provides a weakly supervised video violence detection method based on temporal enhancement and contrastive learning. The temporal feature enhancement method strengthens the learning ability of temporal context and improves the feature representation of video frames. At the same time, the contrastive learning method based on feature amplification increases the distinguishability of different categories of video frames, thereby improving the video violence detection effect. Summary of the Invention

[0006] The purpose of this invention is to provide a weakly supervised video violence detection method based on temporal enhancement and contrastive learning, which solves the problems of weak temporal context learning ability and mismatch between video labels and frame labels in the existing technology.

[0007] The technical solution adopted in this invention is a weakly supervised video brute-force detection method based on temporal enhancement and contrastive learning. Specifically, the video to be detected is subjected to video feature extraction, video feature fusion, temporal feature enhancement, contrastive learning based on feature amplification, and self-distillation optimization training to complete the brute-force detection of the video.

[0008] The invention is further characterized by:

[0009] The weakly supervised video brute-force detection method based on temporal augmentation and contrastive learning is implemented according to the following steps:

[0010] Step 1: Obtain the visual and auditory features of the video to be detected;

[0011] Step 2: Fuse the visual and auditory features obtained in Step 1 to obtain multimodal fusion features;

[0012] Step 3, temporal feature enhancement, strengthens the learning of local and global contexts from the time dimension, and obtains the temporally enhanced feature vector;

[0013] Step 4, based on feature magnification contrastive learning, firstly, the feature magnification method is used to reduce the interference caused by camera movement, and during the training process, the most discriminative video frame in the video is selected as the feature representation of the video. Then, contrastive learning is performed on the video frames with labels to optimize the frame classifier.

[0014] Step 5, Self-distillation to optimize training: The self-distillation module is used to optimize the model training process. A two-stage distillation method is used to allow the student network of the current cycle to learn from the teacher network of the previous cycle. At the same time, label smoothing is used to maintain the consistency of training and improve the generalization ability of the model.

[0015] Step 1 specifically involves:

[0016] Step 1.1: Extract visual features using the I3D network. Select ResNet50 as the backbone network of the I3D model to obtain high-level visual feature vectors X. v ;

[0017] Step 1.2: Using the VGGish model as the auditory feature extractor, the auditory signal is preprocessed to obtain log-Mel spectrum features that conform to human perception, and then the auditory feature vector X of the video is obtained. a ;

[0018] Step 2 specifically involves:

[0019] Step 2.1: The visual and auditory features obtained in Step 1 are fused using an attention-based feature fusion method to obtain the attention-fused feature X. att ;

[0020] Step 2.2: Based on attention fusion, an adaptive feature fusion method is adopted, allowing the network to adaptively learn weight coefficients. The auditory, visual, and attention fusion features are weighted and summed to obtain the multimodal adaptive fusion feature X. ada ;

[0021] Step 3 specifically involves:

[0022] Step 3.1: A non-local structure is used for global temporal feature enhancement. The correlation between video frame i and any other arbitrary video frame j is obtained by learning the similarity between them. The feature vectors of each video frame are weighted, summed, and normalized to obtain the feature vector X after global temporal feature enhancement. v-n ;

[0023] Step 3.2: A multi-layer dilated convolutional network is used for local temporal feature enhancement. Different scales of dilated convolutions are set to increase the receptive field to nearby frames, thereby achieving short-range temporal feature enhancement and obtaining the feature vector X after local temporal feature enhancement. v-d ;

[0024] Step 3.3 involves concatenating the feature vectors enhanced by global and local temporal features, and then generating d through a one-dimensional convolution mapping. v The fused feature vector of dimension is used as the feature vector after temporal enhancement.

[0025] Step 4 specifically involves:

[0026] Step 4.1: Use the feature amplification method to learn a feature regularization term for the feature vector of each video frame, making the feature representation of the video frame smooth, and obtain the amplified feature vector F. v .

[0027] Step 4.2: Use a three-layer MLP as the video frame classifier. The input to the classifier is the feature vector F after feature amplification. v The output is the prediction score p of the video frame;

[0028] Step 4.3: In each training process, select the t video frames with the highest prediction scores as video feature representations, and use the t video frames and corresponding video labels to train the frame classifier;

[0029] Step 4.4: Construct the loss function and train the algorithm. The training loss for the feature-amplified contrastive learning method is composed of the classification loss L. CE Feature contrast loss L c Smoothness loss L m and sparsity loss L s It consists of four parts, as shown in the following formula:

[0030] L m =L CE +λ1L c +λ2L m +λ3L s (1)

[0031] In the formula, λ1, λ2, and λ3 represent the weight coefficients of feature contrast loss, smoothness loss, and sparsity loss, respectively, indicating their importance;

[0032] The loss function in step 4.4 is as follows:

[0033] Step 4.4.1: In each training process, select the t video frames with the highest predicted scores as video feature representations, train a frame classifier using the t video frames and their corresponding video labels, and calculate the classification loss L using the cross-entropy loss function. CE ;

[0034] L CE =-(y·log(p) i )+(1-y)·log(1-p i(2)

[0035] In the formula, y represents the true label of the current video, and y = 1 indicates that the current video is a violent video and is a positive sample; p i 1-p represents the probability that the model predicts video frame i to be a positive sample. i The probability of it being a negative sample;

[0036] Step 4.4.2, use feature contrast loss L c Learn the differences between features of different categories, reduce the distance between features of the same category, and increase the distance between features of different categories;

[0037]

[0038] In the formula, B represents the batch size, and B / 2 violent videos and B / 2 non-violent videos are selected from the training set each time. p and q represent the indices of the positive sample videos, m and n are the indices of the negative sample videos, c is an indicator parameter, when c=1 indicates that the two randomly selected feature vectors come from the same category, that is, both are positive samples or both are negative samples, c=0 indicates that the two feature vectors come from different categories, and d(.) represents the distance between the feature vectors;

[0039] Step 4.4.3: The video is a continuous sequence of frames in the time dimension. The violent scenes appear continuously, and adjacent video frames are highly similar with close scores. Therefore, a smoothness loss L is introduced. m To avoid the randomness of video frame prediction scores;

[0040]

[0041] In the formula, k represents the number of video frames, and s i s represents the prediction score of video frame i in the video. i+1 The prediction score is the next video frame i+1 after video frame i.

[0042] Step 4.4.4: Violent scenes in violent videos are a minority of scenes, appearing for a short time, resulting in a low overall prediction score for the video. Therefore, a sparsity loss L is introduced. s Constrain the overall score of all video frames in the video:

[0043]

[0044] Step 5 specifically involves:

[0045] Step 5.1: Use the self-distillation algorithm to use the previous cycle as the teacher network to assist the current student network. At the end of each training cycle, save the input data and predicted score of the current cycle, and then input them in the next cycle. In this way, the student network learns the knowledge of the teacher network under the same data, so that the positive and negative sample scores have a certain degree of differentiation.

[0046] Step 5.2: Use JS divergence to calculate the similarity of the distribution between the current model's predicted score and the predicted score of the same data in the previous cycle's model. The closer the distribution, the smaller the KL value and the smaller the loss. Establish a self-distillation loss function, introduce a temperature coefficient τ to control the smoothness of the labels, and calculate the self-distillation loss L. JSD ;

[0047] Step 5.3: Optimize the model loss function, using a contrastive learning method based on feature amplification for the loss function L. m and self-distillation loss L JSD It consists of two parts, which improves the model's generalization ability;

[0048] L = L m +λ4L JSD (6)

[0049] In the formula, λ4 is the weighting coefficient for self-distillation loss;

[0050] The specific steps in step 5.2 for establishing the self-distillation loss function are as follows:

[0051]

[0052] At period t, the feature vector is used to obtain a prediction score through a contrastive learning algorithm. in The predicted score of the feature vector from the previous period in the model for this period. Save the predicted score of the current period's feature vector. The soft label is used for the teacher network in the next cycle. In the next cycle, the feature vector and the predicted score are input at the same time, so that the student network can learn the knowledge of the teacher network under the same data. The distribution similarity between the current model's predicted score and the predicted score of the same data in the previous cycle is calculated. The closer the distribution is, the smaller the KL value and the smaller the loss. The temperature coefficient τ controls the smoothness of the label. The higher the temperature, the more uniform the distribution. The lower the temperature, the steeper the distribution.

[0053] The beneficial effects of this invention are:

[0054] The present invention provides a weakly supervised video brute-force detection method based on temporal enhancement and contrastive learning. This method primarily addresses the weaknesses in temporal learning capabilities and the mismatch between video labels and frame labels in existing technologies. The invention employs temporal feature enhancement to strengthen the learning of video frames near and far from the target frame, thereby improving the feature representation of video frames. During training, a contrastive learning method based on feature amplification is used to enhance the discriminative power of video frames and optimize the frame classifier. Furthermore, a self-distillation learning strategy is introduced to learn during different training cycles of the model, achieving label smoothing and improving the model's brute-force detection performance and generalization ability. Attached Figure Description

[0055] Figure 1 This is a diagram illustrating the overall model framework of the weakly supervised video violence detection method based on temporal augmentation and contrastive learning of the present invention.

[0056] Figure 2 This is a structural diagram of the temporal feature enhancement module of the weakly supervised video violence detection method based on temporal enhancement and contrastive learning of the present invention;

[0057] Figure 3 This is a structural diagram of the self-distillation training module of the weakly supervised video brute-force detection method based on temporal augmentation and contrastive learning of the present invention. Detailed Implementation

[0058] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0059] This invention provides a weakly supervised video violence detection method based on temporal augmentation and contrastive learning, as shown in the following embodiments:

[0060] Example 1: The design concept of the present invention is as follows: In view of the problems of weak temporal learning ability and mismatch between video labels and frame labels in existing weakly supervised detection methods, the present invention provides a weakly supervised video brute-force detection method based on temporal enhancement and contrastive learning. The method specifically includes the following modules: video feature extraction module, video feature fusion module, temporal feature enhancement module, contrastive learning module based on feature amplification, and self-distillation training module.

[0061] The specific process and effects are shown in the following examples:

[0062] Example 2, as Figure 1 As shown, the weakly supervised video brute-force detection method based on temporal augmentation and contrastive learning is implemented according to the following steps:

[0063] Step 1, Video Feature Extraction: The I3D (Inflated 3D ConvNet) network and VGGish model are used to extract the visual and auditory features of the video to be detected, respectively. This includes the following steps:

[0064] Step 1.1: Use the I3D network to extract visual features. Select ResNet50 as the backbone network of the I3D model, which can effectively capture the spatiotemporal information of vision and obtain high-level visual feature vectors. Where k is the number of video frames, d v =1024 indicates the dimension of the video feature vector;

[0065] Step 1.2: Using the VGGish model as the auditory feature extractor, the auditory signal is preprocessed to obtain log-Mel spectrum features that conform to human perception, and then the auditory feature vector of the video is obtained. Where d a =128 is the dimension of the auditory feature vector;

[0066] Step 2, video feature fusion: An adaptive feature fusion method based on an attention mechanism is used to fuse the visual and auditory features obtained in Step 1 to obtain multimodal fused features; specifically, it includes the following steps:

[0067] Step 2.1: The visual and auditory features obtained in Step 1 are fused using an attention-based feature fusion method to obtain attention-fused features.

[0068] X att =LN((G+L)V v (8)

[0069] In the formula, G is the learned global feature matrix, representing the global correlation between visual and auditory features; L is the learned local feature matrix, which considers the influence of temporal location on the correlation of audio and video features, and assigns greater weight to frames near the target frame; V v The linear transformation of the visual feature vectors allows the visual features to learn global and local knowledge of the auditory features. Then, the layer standardization LN(·) operation is performed to stabilize the model training process, thereby obtaining the attention fusion features.

[0070] Step 2.2: Based on attention fusion, an adaptive feature fusion method is adopted, allowing the network to adaptively learn weight coefficients. The auditory, visual, and attention fusion features are weighted and summed to obtain multimodal adaptive fusion features.

[0071] X ada =α·f(X) a )+β·X v +γ·X att (9)

[0072] In the formula, α, β, and γ are the parameters learned by the model, representing the coefficients of auditory features, visual features, and attention fusion features, respectively, indicating the importance of the three modal features. The feature vectors of visual features, attention fusion features, and adaptive fusion features all have the same dimension of 1024, while the auditory feature vector has a dimension of 128. Therefore, f(x) is used. a The auditory features are mapped to 1024-dimensional feature vectors, and then the three are summed.

[0073] Step 3, temporal feature enhancement: Video is a continuous sequence of frames in the time dimension. Any frame is related to frames at other time points. Therefore, it is necessary to extract the content of the target frame while simultaneously grasping the length and distance information of the entire video; thus, as... Figure 2 As shown, non-local operations and multi-layer dilated convolutional networks are used to enhance the learning of local and global contexts in the temporal dimension, respectively, to obtain temporally enhanced feature vectors, thereby improving the feature representation of video frames, as detailed below:

[0074] Step 3.1: A non-local structure is used for global temporal feature enhancement. The correlation between video frame i and any other arbitrary video frame j is obtained by learning the similarity between them. The feature vectors of each video frame are weighted, summed, and normalized to obtain the feature vector X after global temporal feature enhancement. v-n ;

[0075]

[0076] In the formula, y i ∈X v-n Let x represent the feature vector after global temporal feature enhancement for video frame i. i and x j Let f(·) represent the adaptive fusion feature vectors of video frame i and video frame j, respectively. The function f(·) represents the similarity between video frame i and video frame j, and the function g(·) represents the feature mapping of video frame j. C(x) is the normalization function. The nonlocal operation module can capture the contextual information and dependencies in the temporal dimension, learn temporal features, and enhance the feature expressiveness of video frames. While calculating the feature representation of any video frame, it takes into account the feature information of all other video frames in the video, and obtains different response calculations according to different video frame relationships. It pays more attention to features that are highly correlated with the target video frame and learns more information, while paying less attention to video frames with low similarity. Global feature learning is obtained through weight calculation.

[0077] Step 3.2: A multi-layer dilated convolutional network is used for local temporal feature enhancement. Different scales of dilated convolutions are set to increase the receptive field to nearby frames, thereby achieving short-range temporal feature enhancement and obtaining the feature vector X after local temporal feature enhancement.v-d ;

[0078] X v-d =σ(M1)·M1+σ(M2)·M2+σ(M4)·M4 (11)

[0079] In the formula, M i To represent features under different dilated convolution dilation rates i = (1, 2, 4), dilated convolution is a special form of standard convolution. It involves taking corresponding positions in every i columns or i rows of the feature map and performing convolution with the kernel. A larger dilation rate results in a larger receptive field, but excessive dilation can lead to local information loss. Different scales can be set to obtain video feature vectors under different receptive fields. However, the expressive power of feature vectors varies depending on the receptive field. Therefore, weight coefficients σ(M1), σ(M2), and σ(M4) are introduced to perform a weighted summation, resulting in a feature vector that integrates features from different receptive fields. The weight coefficients are obtained through the activation function σ(·). The feature vectors under different dilation rates are processed by the activation function to obtain expert scores for the feature, indicating the importance of the feature vector.

[0080] Step 3.3 involves concatenating the feature vectors enhanced by global and local temporal features, and then generating d through a one-dimensional convolution mapping. v The fused feature vector of dimension is used as the feature vector after temporal enhancement.

[0081]

[0082] Step 4, based on feature magnification contrastive learning, firstly, the feature magnification method is used to reduce the interference caused by camera movement, and during the training process, the most discriminative video frame in the video is selected as the feature representation of the video. Then, contrastive learning is performed on the video frames with labels to optimize the frame classifier. The specific implementation is as follows:

[0083] Step 4.1: Use the feature amplification method to learn a feature regularization term for the feature vector of each video frame, making the feature representation of the video frame smooth, and obtain the amplified feature vector F. v :

[0084]

[0085] f i =z i +γConv1d(n i (14)

[0086] In the formula, f i ∈F v This represents the magnified feature vector of video frame i. This is the feature vector after temporal feature enhancement, γ is the weight parameter, and Conv1d(·) represents a one-dimensional convolution operation, mapping one-dimensional data to d. v 3D feature vector;

[0087] Step 4.2: Use a three-layer MLP as the video frame classifier. The input to the classifier is the feature vector F after feature amplification. v The output is the prediction score p of the video frame;

[0088] Step 4.3: The video frame with the highest prediction score is the most discriminative video frame. In each training process, the t video frames with the highest prediction scores are selected as video feature representations, and the frame classifier is trained using the t video frames and their corresponding video labels.

[0089] Step 4.4: Construct the loss function and train the algorithm. The training loss for the feature-amplified contrastive learning method is composed of the classification loss L. CE Feature contrast loss L c Smoothness loss L m and sparsity loss L s It consists of four parts, as shown in the following formula:

[0090] L m =L CE +λ1L c +λ2L m +λ3L s (1)

[0091] In the formula, λ1, λ2, and λ3 represent the weight coefficients of feature contrast loss, smoothness loss, and sparsity loss, respectively, indicating their importance;

[0092] The loss function is established as follows:

[0093] Step 4.4.1: In each training process, select the t video frames with the highest predicted scores as video feature representations, train a frame classifier using the t video frames and their corresponding video labels, and calculate the classification loss L using the cross-entropy loss function. CE ;

[0094] L CE =-(y·log(p) i )+(1-y)·log(1-p i (2)

[0095] In the formula, y represents the true label of the current video, and y = 1 indicates that the current video is a violent video and is a positive sample; p i 1-p represents the probability that the model predicts video frame i to be a positive sample. i The probability of it being a negative sample;

[0096] Step 4.4.2, use feature contrast loss L c Learn the differences between features of different categories, reduce the distance between features of the same category, and increase the distance between features of different categories;

[0097]

[0098] In the formula, B represents the batch size, where B / 2 of the violent videos and B / 2 of the non-violent videos are selected from the training set during each training session. p and q represent the indices of the positive sample videos, m and n represent the indices of the negative sample videos, and c is an indicator parameter. When c = 1, it indicates that the two randomly selected feature vectors come from the same category, i.e., both are positive samples or both are negative samples. When c = 0, it indicates that the two feature vectors come from different categories. d(·) represents the distance between the feature vectors. When the two features come from the same sample, the greater the distance, the greater the loss. Through loss constraints, the distance between feature vectors of the same sample is reduced. When the two features come from different samples, through the third loss constraint, the distance between the two feature vectors is increased, and the greater the distance, the smaller the loss.

[0099] Step 4.4.3: The video is a continuous sequence of frames in the time dimension. The violent scenes appear continuously, and adjacent video frames are highly similar with close scores. Therefore, a smoothness loss L is introduced. m To avoid the randomness of video frame prediction scores;

[0100]

[0101] In the formula, k represents the number of video frames, and s i s represents the prediction score of video frame i in the video. i+1 The prediction score is the next video frame i+1 after video frame i.

[0102] Step 4.4.4: Violent scenes in violent videos are a minority of scenes, appearing for a short time, resulting in a low overall prediction score for the video. Therefore, a sparsity loss L is introduced. s Constrain the overall score of all video frames in the video:

[0103]

[0104] Step 5, self-distillation optimization training: Due to the difficulty of video frame annotation, and the fact that brute-force detection algorithms are based on small datasets, overfitting is prone to occur during model training, leading to poor generalization. Therefore, as... Figure 3 As shown, a self-distillation module is used to optimize the model training process. A two-stage distillation method is used to allow the student network of the current cycle to learn from the teacher network of the previous cycle. At the same time, label smoothing is used to maintain the consistency of training and improve the generalization ability of the model. The specific implementation is as follows:

[0105] Step 5.1: Use the previous cycle as the teacher network to assist the current student network, and save the input data and predicted score of the current cycle at the end of each training cycle. in The feature vector representing video frame i in the current period t. and the true label of the corresponding feature The predicted score for this network is then calculated for period t. This is followed by input in the next period, allowing the student network to learn from the teacher network using the same data, thus making the scores of positive and negative samples more discriminative.

[0106] Step 5.2: Use JS divergence to calculate the similarity of the distribution between the current model's predicted score and the predicted score of the same data in the previous cycle's model. The closer the distribution, the smaller the KL value and the smaller the loss. Establish a self-distillation loss function, introduce a temperature coefficient τ to control the smoothness of the labels, and calculate the self-distillation loss L. JSD ;

[0107]

[0108] At period t, the feature vector is used to obtain a prediction score through a contrastive learning algorithm. in The predicted score of the feature vector from the previous period in the model for this period. Save the predicted score of the current period's feature vector. The soft label is used for the teacher network in the next cycle. In the next cycle, the feature vector and the predicted score are input at the same time, so that the student network can learn the knowledge of the teacher network under the same data. The distribution similarity between the current model's predicted score and the predicted score of the same data in the previous cycle is calculated. The closer the distribution is, the smaller the KL value and the smaller the loss. The temperature coefficient τ controls the smoothness of the label. When the temperature is higher, the distribution is more uniform. When the temperature is lower, the distribution is steeper.

[0109] Step 5.3: Optimize the model loss function, using a contrastive learning method based on feature amplification for the loss function L. m and self-distillation loss L JSD It consists of two parts, which improves the model's generalization ability;

[0110] L = L m +λ4L JSD (6)

[0111] In the formula, λ4 is the weighting coefficient for self-distillation loss;

[0112] Example 3 verifies the weakly supervised video brute-force detection method based on temporal augmentation and contrastive learning of the present invention: Load the trained detection model and use the test set of a public dataset to perform brute-force detection classification verification;

[0113] This invention uses two publicly available violence detection datasets for experimental verification: the unimodal UCF-Crime dataset and the multimodal XD-Violence dataset. The UCF-Crime dataset contains 1900 videos from real surveillance equipment and indoor surveillance equipment; the XD-Violence dataset contains 4754 videos from movies, YouTube, handheld devices, surveillance videos, etc. This experiment is implemented using the PyTorch framework, and the specific model parameter settings are shown in Table 1.

[0114] Table 1. Main parameter settings for the model

[0115] Training parameters Parameter meaning Parameter values Learning_rate Initial learning rate 4e-4 Batch_size Number of samples per training batch 128 Epochs Training rounds 50 Seed Random Seed 9 IoU Classification threshold 0.5 γ Regularization coefficient 0.1 t Select the t highest-scoring frames 3 λ1 Comparison of loss coefficients 0.001 λ2 Smoothness loss coefficient 1 λ3 sparsity loss coefficient 1 λ4 Self-distillation loss coefficient 1 τ Self-distillation temperature coefficient 0.1

[0116] The following tables illustrate the detection performance of this invention on the UCF-Crime and XD-Violence public datasets, respectively, and are evaluated using AUC and AP metrics:

[0117] Table 2 Comparison of detection performance on the UCF-Crime dataset

[0118] Model Modal AUC(%) MIL C3D-RGB 75.40 DTED C3D-RGB 79.49 MIST C3D-RGB 81.40 MIST I3D-RGB 82.30 CTR-FD I3D-RGB 82.44 Our method I3D-RGB 85.97

[0119] Table 3 Comparison of detection performance on the XD-Violence dataset.

[0120] Model Modal AP (%) Our method Visual features 79.12 Our method Fusion features 83.23

[0121] Table 2 lists the following existing methods for comparison: Waqas Sultani et al. Real-World Anomaly Detection in Surveillance Videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, pp. 6479-6488;

[0122] Ammar Mansoor Kamoona et al. Multiple instance-based video anomalydetection using deep temporal encoding–decoding. "Expert Systems with Applications". 2023, Volume 214;

[0123] Jia-Chang Feng et al. MIST: Multiple instance self-training framework for video anomaly detection. "Proceedings of the IEEE / CVF conference on computer vision and pattern recognition". 2021, pages 14009-14018;

[0124] Peng Wu et al.Learning causal temporal relation and featurediscrimination for anomaly detection. "IEEE Transactions on Image Processing". 2021, Volume 30, Pages 3513-3527;

[0125] As can be seen from Tables 2 and 3, the detection performance of the weakly supervised brute-force detection method based on temporal enhancement and contrastive learning in this invention is significantly improved compared to the contrastive method. Furthermore, the experimental results using multimodal fusion features are superior to the method using only video features, demonstrating the effectiveness and superiority of the proposed method. The contrastive learning method based on feature amplification proposed in this invention belongs to supervised contrastive learning. It uses a max-pooling strategy to convert video labels into video frame labels. The label-based contrastive learning method effectively reduces the feature distance between video frames of the same category and increases the feature distance between video frames of different categories, thereby achieving better detection results. Simultaneously, different visual feature extraction methods yield different detection results under the same method. The I3D feature extraction method is superior to the C3D feature extraction method, with an improvement of nearly 1%. Therefore, this invention uses the I3D feature extraction method.

Claims

1. A weakly supervised video brute-force detection method based on temporal augmentation and contrastive learning, characterized in that, Specifically, the video to be detected is subjected to video feature extraction, video feature fusion, temporal feature enhancement, feature-amplified contrastive learning, and self-distillation optimization training to complete brute-force detection of the video; The specific steps are as follows: Step 1: Obtain the visual and auditory features of the video to be detected; Step 2: Fuse the visual and auditory features obtained in Step 1 to obtain multimodal fusion features; Step 3, temporal feature enhancement, strengthens the learning of local and global contexts from the time dimension, and obtains the temporally enhanced feature vector; Step 4, based on feature magnification contrastive learning, firstly, the feature magnification method is used to reduce the interference caused by camera movement, and during the training process, the most discriminative video frame in the video is selected as the feature representation of the video. Then, contrastive learning is performed on the video frames with labels to optimize the frame classifier. Step 5, Self-distillation to optimize training: The self-distillation module is used to optimize the model training process. A two-stage distillation method is used to allow the student network of the current cycle to learn from the teacher network of the previous cycle. At the same time, label smoothing is used to maintain the consistency of training and improve the generalization ability of the model. Step 4 specifically involves: Step 4.1: Use the feature amplification method to learn a feature regularization term for the feature vector of each video frame, making the feature representation of the video frame smoother, and obtain the amplified feature vector. ; Step 4.2: Use a three-layer MLP as the video frame classifier. The input to the classifier is the feature vector after feature amplification. The output is the prediction score of the video frame. ; Step 4.3: Select the highest predicted score during each training session. t Each video frame is used as a video feature representation, using t Train a frame classifier using one video frame and its corresponding video label; Step 4.4: Construct the loss function and train the algorithm. The training loss for the feature-amplified contrastive learning method consists of the classification loss. Feature contrast loss Smoothness loss and sparsity loss It consists of four parts, as shown in the following formula: (1) In the formula, , , These represent the weight coefficients of the feature contrast loss, smoothness loss, and sparsity loss, respectively. The loss function in step 4.4 is as follows: Step 4.4.1: Select the highest predicted score in each training session. t Each video frame is used as a video feature representation, using t A frame classifier is trained using one video frame and its corresponding video label, and the classification loss is calculated using the cross-entropy loss function. ; (2) In the formula, Indicates the true tag of the current video. This indicates that the current video is a violent video and is a positive sample; The model predicts video frames. i The probability of being a positive sample. The probability of it being a negative sample; Step 4.4.2, use feature contrast loss Learn the differences between features of different categories, reduce the distance between features of the same category, and increase the distance between features of different categories; (3) In the formula, B This indicates the batch size, which is selected from the training set each time training is performed. B / 2 violent videos and B / 2 non-violent videos, and These represent the indices of the positive sample videos. and For the index of negative sample videos, To indicate parameters, when This indicates that the two randomly selected feature vectors originate from the same category, i.e., both are positive samples or both are negative samples. This indicates that the two feature vectors originate from different categories. This represents the distance between feature vectors; Step 4.4.3: The video is a continuous sequence of frames in the time dimension. The violent scenes appear continuously, and adjacent video frames are highly similar with close scores. Therefore, a smoothness loss is introduced. To avoid the randomness of video frame prediction scores; (4) In the formula, Indicates the number of video frames. Represents video frames in a video The predicted score, For video frames The next adjacent video frames The predicted score; Step 4.4.4: Violent scenes are a minority in violent videos, appearing for a short time, resulting in a low overall prediction score for the video. Therefore, sparsity loss is introduced. Constrain the overall score of all video frames in the video: (5)。 2. The weakly supervised video violence detection method based on temporal augmentation and contrastive learning according to claim 1, characterized in that, Step 1 specifically involves: Step 1.1: Extract visual features using the I3D network. Select ResNet50 as the backbone network of the I3D model to obtain high-level visual feature vectors. ; Step 1.2: Using the VGGish model as the auditory feature extractor, the auditory signal is preprocessed to obtain log-Mel spectrum features that conform to human perception, and then the auditory feature vector of the video is obtained. .

3. The weakly supervised video violence detection method based on temporal augmentation and contrastive learning according to claim 1, characterized in that, Step 2 specifically involves: Step 2.1: The visual and auditory features obtained in Step 1 are fused using an attention-based feature fusion method to obtain attention-fused features. ; Step 2.2: Based on attention fusion, an adaptive feature fusion method is adopted, allowing the network to adaptively learn weight coefficients. The auditory, visual, and attention fusion features are weighted and summed to obtain multimodal adaptive fusion features. .

4. The weakly supervised video violence detection method based on temporal augmentation and contrastive learning according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Global temporal feature enhancement is performed using a non-local structure by learning video frames. With other arbitrary video frames The similarity between the two is used to determine their correlation. The feature vectors of each video frame are weighted, summed, and normalized to obtain the feature vector after global temporal feature enhancement. ; Step 3.2: A multi-layer dilated convolutional network is used for local temporal feature enhancement. Different scales of dilated convolutions are set to increase the receptive field to nearby frames, thereby achieving short-range temporal feature enhancement and obtaining the feature vector after local temporal feature enhancement. ; Step 3.3 involves concatenating the feature vectors enhanced by global and local temporal features, and then generating a new feature vector through a one-dimensional convolution mapping. The fused feature vectors of dimension 1 are used as the feature vectors after temporal enhancement. .

5. The weakly supervised video violence detection method based on temporal augmentation and contrastive learning according to claim 1, characterized in that, Step 5 specifically involves: Step 5.1: Use the self-distillation algorithm to use the previous cycle as the teacher network to assist the current student network. At the end of each training cycle, save the input data and predicted score of the current cycle, and then input them in the next cycle. In this way, the student network learns the knowledge of the teacher network under the same data, so that the positive and negative sample scores have a certain degree of differentiation. Step 5.2: Use JS divergence to calculate the similarity in distribution between the current model's predicted score and the predicted score of the same data in the previous cycle's model. The closer the distribution, the better. The smaller the value, the smaller the loss; establish a self-distillation loss function and introduce a temperature coefficient. By controlling the smoothness of the label, the loss from self-distillation is calculated. ; Step 5.3: Optimize the model loss function using a contrastive learning method based on feature amplification. and self-distillation loss It consists of two parts, which improves the model's generalization ability; (6) In the formula, This is the weighting coefficient for self-distillation loss.

6. The weakly supervised video violence detection method based on temporal augmentation and contrastive learning according to claim 5, characterized in that, The self-distillation loss function established in step 5.2 is specifically as follows: (7) In the cycle t At that time, the feature vectors are used to obtain the predicted score through a contrastive learning algorithm. ,in The predicted score of the feature vector from the previous period in the model for this period. Save the predicted score of the current period's feature vector. This serves as a soft label for the teacher network in the next cycle. In the next cycle, both the feature vector and the predicted score are input simultaneously, allowing the student network to learn from the teacher network using the same data. The similarity between the current model's predicted score and the predicted scores in the previous cycle with the same data is calculated; the closer the distribution, the better. The smaller the value, the smaller the loss; temperature coefficient The smoothness of the labels is controlled; the higher the temperature, the more uniform the distribution, and the lower the temperature, the steeper the distribution.

Citation Information

Patent Citations

  • Semi-supervised video classification method and system based on neighbor consistency and comparative learning

    CN115311605A