Real-time online weak supervision video anomaly detection method

Abstract

The invention discloses a real-time online weak supervision video anomaly detection method. The method comprises the following steps: acquiring a training sample of video clip data; constructing a teacher network and a student network, wherein the teacher network and the student network are similar deep learning models; constructing a feature extraction module, and performing training and knowledge distillation on the teacher network, the student network and the feature extraction module through training samples; an optimized feature extraction module is obtained; and performing anomaly detection on the current video data through the feature extraction module to obtain a video anomaly detection result.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, and in particular relates to a real-time online weakly supervised video anomaly detection method. Background Technology

[0002] Video anomaly detection is widely used in security, surveillance, industrial safety, traffic management, retail loss prevention, and power facility monitoring. Based on models and algorithms, it can analyze video data in real time to detect various abnormal behaviors and events, such as intrusions, accidents, and equipment failures, thereby providing a more efficient and accurate monitoring and early warning system.

[0003] However, traditional video anomaly detection methods typically analyze entire video segments, lacking real-time capabilities and the ability to immediately detect and identify abnormal behaviors or events within the video. This prevents relevant personnel from taking timely and necessary actions to mitigate potential losses or risks. Therefore, developing real-time video anomaly detection is of great significance for security monitoring and early warning systems across various fields, helping to improve security, reduce risks, save costs, and increase work efficiency. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a real-time online weakly supervised video anomaly detection method to solve the problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides a real-time online weakly supervised video anomaly detection method, comprising:

[0006] Obtain training samples from video clip data;

[0007] Construct teacher networks and student networks, where the teacher networks and student networks are of the same type of deep learning model;

[0008] A feature extraction module is constructed, and the teacher network, student network, and feature extraction module are trained and knowledge distilled using training samples to obtain an optimized feature extraction module.

[0009] The feature extraction module performs anomaly detection on the current video data to obtain video anomaly detection results.

[0010] Optionally, the process of training and knowledge distillation of the teacher network, student network, and feature extraction module includes:

[0011] Feature extraction is performed on the training samples to obtain the original features, wherein the training samples include normal samples and abnormal samples;

[0012] The original features are extracted using a teacher network, a student network, and a feature extraction module to obtain a first feature, a second feature, and a third feature. The total loss is calculated based on the similarity between the first and second features, the similarity between the second and third features, and the multi-instance loss. The feature extraction model is then optimized based on the total loss to obtain an optimized feature extraction module.

[0013] Optionally, the teacher network includes a global attention mechanism and a first anomaly detection head;

[0014] The first anomaly detection head includes several fully connected layers and activation functions;

[0015] The data processing procedure for the global attention mechanism is as follows:

[0016]

[0017] Where f1 is the first feature, f represents the original feature, and d k Let Q be the dimension of the original feature f, and Q and K be different mappings of the original feature f. The superscript T indicates transpose.

[0018] Optionally, the student network includes a masked attention mechanism and a second anomaly detection head;

[0019] The second anomaly detection head includes several fully connected layers and activation functions;

[0020] The data processing procedure for the attention-blocking mechanism is as follows:

[0021]

[0022] Where f2 is the second feature, d k Let f be the dimension of the original feature f, where f represents the original feature, Q and K represent different mappings of the original feature f, and the superscript T indicates transpose.

[0023] Optionally, the feature extraction module includes a memory module and a second anomaly detection head;

[0024] The data processing of the memory module includes: compressing the original features using an encoder; storing the compressed features; acquiring new original features in real time and compressing them; and updating the stored features in real time based on the new compressed original features.

[0025] The third feature is obtained by performing cross-attention calculation on the stored features and the original features.

[0026] Optionally, the process of obtaining the original features includes:

[0027] The training samples are encoded using a pre-trained I3D model to obtain the original features.

[0028] Optionally, the calculation process for the total loss value includes:

[0029] loss=loss_mil-αloss_dir1-βloss_dir2

[0030] Where loss represents the total loss value, loss_mil represents the multi-instance loss, loss_dir1 represents the first knowledge distillation loss, loss_dir2 represents the second knowledge distillation loss, and α and β represent different weighting factors.

[0031] loss_mil=max(1-S a +S n )

[0032] Among them, S a S represents the output of abnormal samples. n This represents the output of a normal sample;

[0033] loss_dir1 = sim(f1, f2)

[0034] Where sim represents the dot product operation, f1 is the first feature, and f2 is the second feature;

[0035] loss_dir2 = sim(f3, f2)

[0036] Among them, f3 is the third feature.

[0037] On the other hand, the present invention provides a real-time online weakly supervised video anomaly detection system for performing the above-described method.

[0038] Compared with the prior art, the present invention has the following advantages and technical effects:

[0039] This invention provides a real-time online weakly supervised video anomaly detection method. Addressing the time-consuming nature of traditional weakly supervised deep learning methods, it proposes a lightweight detection model based on a distillation model. To overcome the limitations of traditional anomaly detection methods that utilize entire video segments for modeling, this invention proposes a historical modeling method that stores both short-term and long-term historical features, using current and past features for modeling. This invention offers real-time detection capabilities with good performance, can be deployed on edge devices, and has high practical value. Attached Figure Description

[0040] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0041] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram illustrating the detection effect of an embodiment of the present invention. Detailed Implementation

[0043] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0045] like Figure 1 As shown, this invention provides a real-time online weakly supervised video anomaly detection method, the specific scheme of which is as follows:

[0046] (1) Perform I3D encoding on the video features of all normal and abnormal samples, and encode each 16-frame video segment into a one-dimensional vector to obtain the original feature f;

[0047] (2) In order to better capture the temporal relationship of features in different frames, the original I3D features are enhanced based on the global attention temporal feature modeling method to obtain the first feature f1;

[0048] (3) Design a masked attention feature modeling method (i.e. use past features and current features to perform attention calculation) to enhance the original I3D features and obtain the second feature f2;

[0049] (4) Two anomaly detection heads are designed respectively, namely the first anomaly detection head s1 and the second anomaly detection head s2. Each anomaly detection head consists of 3 fully connected layers and a ReLU activation function. The last activation function is Sigmoid to ensure that the output anomaly score is between 0 and 1.

[0050] (5) Construct an offline detection model, connect the first feature f1 and the first anomaly detection head s1 in series as a teacher network, and predict the score output of the abnormal segment;

[0051] (6) Construct an online detection model, connect the second feature f2 and the second anomaly detection head s2 in series as a student network, and predict the score output of the anomaly segment;

[0052] (7) Perform knowledge distillation on the first feature f1 of the teacher network and the second feature f2 of the student network to better express the second feature f2;

[0053] (8) Design a pair of memory modules to compress features from past time points, reducing the computational complexity of the model, denoted as . and The memory module is cross-attentioned with the original feature f to obtain the third feature f3. The student feature, i.e. the second feature f2, is then subjected to knowledge distillation with the third feature f3.

[0054] (9) During the training phase, abnormal samples and normal samples are compared and learned to obtain the predicted scores of the teacher network and the student network. The output of the abnormal detection head is controlled by multi-instance loss, smoothing constraint and feature distillation loss respectively.

[0055] (10) During the inference phase, the video coding features are transmitted through the memory module. and Combined with the second anomaly detection head s2, the anomaly score is output.

[0056] The above technical solution is described in detail:

[0057] 1. To initially extract video features, this invention uses the I3D model for feature extraction, with the following steps:

[0058] (1) First, the video data of each training sample and test sample needs to be segmented into a continuous sequence of frames. Each video frame is processed as input. Both training and test samples include normal samples and abnormal samples, and both training and test samples are video data.

[0059] (2) After dividing each video data into a continuous frame sequence, each 16-frame video segment is encoded as a unit. This is because short video segments, i.e., frame sequences, can provide rich motion information and spatial features, which helps to better capture the content of the video.

[0060] (3) For each 16-frame video segment, a pre-trained I3D model is used to encode it. The encoding process involves inputting the video segment into the I3D model and extracting the output of the last global pooling layer as the feature representation of the video segment. The output of this global pooling layer is a fixed-length vector used to represent the features of the video segment.

[0061] (4) For each 16-frame video segment, the feature representation extracted by the I3D model is further transformed into a one-dimensional vector representation. This can be achieved by flattening the feature representation of the video segment, transforming it into a one-dimensional vector to obtain the original feature f.

[0062] 2. The design of the global attention temporal feature f1 is as follows:

[0063] Global feature modeling refers to modeling and adjusting features at different time steps under the guidance of a global attention mechanism. By modeling the original features at different time points in a video sequence, the model can better understand the temporal evolution of the video and capture its important features.

[0064] The formula for global attention is as follows:

[0065]

[0066] Where Q and K are both mappings to the original I3D feature f, obtained through linear mapping, d k Let f be the dimension of the original feature f, where f represents the original feature. Based on this formula, the enhanced feature f1 is obtained.

[0067] 3. Attention-masking feature modeling method:

[0068] To achieve real-time detection, the current features cannot contain information from future moments. Real-time online video anomaly detection tasks cannot incorporate segments after the current frame during computation. Therefore, during the enhancement of the current features, features after the current frame must be masked. The mask matrix formula is as follows:

[0069]

[0070] In this matrix, i represents the i-th video segment, j represents the j-th video segment, and inf represents infinity. That is, by performing the Softmax operation using the above formula, the segments after the current frame can be masked.

[0071] The expression for feature f2 is as follows:

[0072]

[0073] Where Q and K are both mappings to the original feature f, d k Let f be the dimension of the original feature f.

[0074] 4. Design of the memory module

[0075] (1) Feature compression: The original input feature sequence is compressed to reduce the dimensionality and complexity of the original features. The encoder maps the high-dimensional features of the input to a low-dimensional space, retaining the most important information. Specifically, the encoder here adopts an LSTM structure.

[0076] (2) Memory Storage: The compressed features are stored as memory for use in the current or future time. Specifically, the key K corresponding to the features at time t is stored... t Value V t Extended to obtain the memory module from time tM to time t and This module consists of multiple vector sets representing the long- and short-term features near the current segment. Specifically, and The expression is as follows:

[0077]

[0078] Among them, f v with f k It is a long short-term feature update memory operator based on LSTM, namely the encoder mentioned in (1), where M represents the time interval.

[0079] (3) Memory Update: When a new original feature sequence is input, it is updated using a stack. Specifically, when the latest feature is added to the memory module, the module... and The earliest feature f k (K(tM)) and f v (V(tM)) was removed.

[0080] (4) Feature enhancement: Combine the original feature f with the memory module and Cross-attention calculation is performed to obtain the enhanced third feature f3.

[0081] The calculation formula is as follows:

[0082]

[0083] 5. Anomaly detection head design.

[0084] The design includes an anomaly detection head, comprising a first anomaly detection head and a second anomaly detection head. Each anomaly detection head consists of three fully connected layers and a ReLU activation function. The last activation function is a sigmoid function, ensuring that the output anomaly score is between 0 and 1. Finally, anomalies are identified through multi-instance loss. a S represents the output of abnormal samples. nRepresenting normal sample output, the multi-instance loss is denoted as:

[0085] loss_mil=max(1-S a +S n )

[0086] 6. Knowledge Distillation Method

[0087] Algorithm flowchart as follows Figure 1 As shown, knowledge distillation is performed between the first feature f1 of the teacher network and the second feature f2 of the student network to better represent feature f2;

[0088] Distillation loss is denoted as:

[0089] loss_dir1 = sim(f1, f2)

[0090] Where sim is the dot product of f1 and f2, used to represent feature similarity. Based on this loss, f2 is made closer to f1, and feature f1 is learned better.

[0091] Furthermore, to further reduce the computational burden on the model, a memory module mechanism is used to compress historical features. To better approximate historical features with f2 and reduce the error caused by compression, knowledge distillation loss is also used to constrain the third feature f3 to be close to the second feature f2, as shown in the following formula:

[0092] loss_dir2 = sim(f3, f2)

[0093] Based on the above distillation loss, the online model can learn the features of the offline model, reducing the accuracy loss caused by feature compression.

[0094] 6. Model Training and Inference

[0095] In summary, during the training phase, abnormal samples and normal samples are compared and learned. The output of the anomaly detection head is controlled through multi-instance loss, smoothing constraints, and distillation loss. The final loss is denoted as:

[0096] loss=loss_mil-αloss_dir1-βloss_dir2

[0097] By adjusting different weights of the loss function, as well as parameters such as the model's learning rate, batch size, number of iterations, convolutional kernel size, and network optimization function, the learning performance and effectiveness of the model can be improved. During training, the model is run for 200 epochs with a batch size of 32 on a single GPU. The Adam optimizer is used for optimization during training, with a learning rate of 0.01 for the first 10 epochs, 0.001 for 10-50 epochs, and 0.0001 for 50-200 epochs.

[0098] During the inference phase, the video coding features are processed through the memory module P and the s2 detector head to obtain the anomaly score output.

[0099] Ultimately, as Figure 2 As shown, this method solves the limitations of traditional offline model inference in terms of real-time performance and efficiency, as well as the memory and computational bottlenecks faced by real-time online detection, such as the inability to obtain information about future segments and the modeling of temporal relationships. This online detection method is completely comparable to the performance of offline detection.

[0100] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A real-time online weakly supervised video anomaly detection method, characterized in that, include: Obtain training samples from video clip data; Construct teacher networks and student networks, where the teacher networks and student networks are of the same type of deep learning model; A feature extraction module is constructed, and the teacher network, student network, and feature extraction module are trained and knowledge distilled using training samples to obtain an optimized feature extraction module. The feature extraction module performs anomaly detection on the current video data to obtain video anomaly detection results.

2. The method according to claim 1, characterized in that, The process of training and knowledge distillation of the teacher network, student network, and feature extraction module includes: Feature extraction is performed on the training samples to obtain the original features, wherein the training samples include normal samples and abnormal samples; The original features are extracted using a teacher network, a student network, and a feature extraction module to obtain a first feature, a second feature, and a third feature. The total loss is calculated based on the similarity between the first and second features, the similarity between the second and third features, and the multi-instance loss. The feature extraction model is then optimized based on the total loss to obtain an optimized feature extraction module.

3. The method according to claim 2, characterized in that, The teacher network includes a global attention mechanism and a first anomaly detection head; The first anomaly detection head includes several fully connected layers and activation functions; The data processing procedure for the global attention mechanism is as follows: Where f1 is the first feature, d k Let f be the dimension of the original feature f, where f represents the original feature, Q and K are different mappings of the original feature f, and the superscript T indicates transpose.

4. The method according to claim 2, characterized in that, The student network includes an attention-blocking mechanism and a second anomaly detection head; The second anomaly detection head includes several fully connected layers and activation functions; The data processing procedure for the attention-blocking mechanism is as follows: Where f2 is the second feature, d k Let f be the dimension of the original feature f, where f represents the original feature, Q and K represent different mappings of the original feature f, and the superscript T indicates transpose.

5. The method according to claim 2, characterized in that, The feature extraction module includes a memory module and a second anomaly detection head; The data processing of the memory module includes: compressing the original features using an encoder; storing the compressed features; acquiring new original features in real time and compressing them; and updating the stored features in real time based on the new compressed original features. The third feature is obtained by performing cross-attention calculation on the stored features and the original features.

6. The method according to claim 2, characterized in that, The process of obtaining the original features includes: The training samples are encoded using a pre-trained I3D model to obtain the original features.

7. The method according to claim 2, characterized in that, The calculation process for the total loss value includes: loss=loss_mil-αloss_dir1-βloss_dir2 Where loss represents the total loss value, loss_mil represents the multi-instance loss, loss_dir1 represents the first knowledge distillation loss, loss_dir2 represents the second knowledge distillation loss, and α and β represent different weighting factors. loss_mil=max(1-S a +S n ) Among them, S a S represents the output of abnormal samples. n This represents the output of a normal sample; loss_dir1 = sim(f1, f2) Where sim represents the dot product operation, f1 is the first feature, and f2 is the second feature; loss_dir2 = sim(f3, f2) Among them, f3 is the third feature.

8. A real-time online weakly supervised video anomaly detection system, characterized in that, Used to perform the method described in any one of claims 1-7.

Images