Video anomaly detection method and system fusing key frame and multi-task hybrid model
Keyframes are extracted through segmented clustering and inter-frame differential, and combined with a skeleton-based dual-branch hybrid model, fusion reconstruction and prediction methods, the video anomaly detection is achieved efficient and accurate, solving the low efficiency and error in the existing technology The problem of missed inspection.
Patent Information
- Application Number
- CN202411222192.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2025-01-03
AI Technical Summary
The existing video abnormality detection methods are inefficient due to large data volume, high memory consumption and redundancy in information when processing high-definition videos, and the problems of misjudgment and missed detection are prone to the reconstruction and prediction.
Using inter-frame differential keyframe extraction technology with segmented clustering, combining the two-branch hybrid model based on bones, fusion reconstruction and prediction methods, and using historical frames and future frames to reconstruct and predict current frames through multi-task learning, improving detection efficiency and accuracy.
By extracting keyframes and fusing multitask learning hybrid models, memory consumption is reduced, the efficiency and accuracy of video anomaly detection is improved, misjudgment and missed detection are reduced, and the robustness and generalization ability of detection are enhanced.
Smart Images
Figure CN119251723A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a video anomaly detection method and system that integrates key frames and a multi-task hybrid model. Background Art
[0002] Video anomaly detection refers to the use of algorithms to detect behaviors or events in videos that do not meet people's expectations, such as crowds, arson, robbery, driving in the wrong direction, jaywalking, etc. With the rapid development of society and the deployment of a large number of high-definition cameras, video anomaly detection has become a hot research area at home and abroad. With the increase in data volume, traditional machine learning algorithms are unable to capture the complex structure between data samples, and their performance drops significantly when detecting outliers. The deep learning-based method has great advantages in video anomaly detection. It can not only use the constructed deep convolutional neural network model to autonomously learn the hierarchical discriminant features between data samples, but also has excellent adaptive and data processing capabilities.
[0003] Video can be viewed as many consecutive images superimposed in chronological order. Since surveillance video contains not only behavioral action information but also a large number of repeated frames, this leads to a large amount of memory consumption and information redundancy, increases the computational complexity of the model, and reduces the efficiency and recognition rate of video anomaly detection.
[0004] At present, video anomaly detection mainly adopts reconstruction and prediction-based methods. In the reconstruction method, autoencoders are mainly used to complete encoding and decoding. However, their powerful encoding and decoding capabilities lead to good reconstruction of abnormal data samples, resulting in a high misjudgment rate of abnormal data. The prediction-based method mainly predicts future frames by learning historical frames and compares them with real future frames. However, this method is easily affected by data changes and environmental noise, especially for the processing of real-time video streams, which will lead to model instability and increased errors, affecting detection performance.
[0005] On the other hand, the current mainstream video anomaly detection methods based on reconstruction and prediction mostly reconstruct and predict future frames by learning historical frames, but scene changes and occlusions can lead to inaccurate predictions, and this method can lead to missed detection of abnormal behaviors in past time periods. Therefore, methods based on future frame reconstruction and prediction still have certain defects. Summary of the invention
[0006] In view of the above problems, the present invention proposes a video anomaly detection method and system that integrates key frames and multi-task hybrid models. The method first uses the segmented clustering inter-frame differential key frame extraction technology and uses the extracted key frames as model input, which saves storage space and improves detection efficiency. Then, based on the human skeleton feature extraction, the reconstruction-based method and the prediction-based method are integrated to construct a hybrid model. In terms of multi-task learning, historical frames are used to reconstruct and predict future frames (Ftr), future frames are used to reconstruct and predict historical frames (Pst), and historical frames and future frames are used to reconstruct and predict current frames (Prs).
[0007] In order to achieve the above object, the present invention adopts the following technical solutions:
[0008] On the one hand, the present invention proposes a video anomaly detection method integrating key frames and multi-task hybrid models, comprising:
[0009] Step 1: Get the video dataset;
[0010] Step 2: construct a hybrid model integrating key frames and multi-task learning; the hybrid model includes a key frame extraction module and a skeleton-based dual-branch hybrid model; the key frame extraction module is used to extract key frames of the video; the skeleton-based dual-branch hybrid model includes two branches, reconstruction and prediction, each branch includes a human skeleton extraction module, an encoder, a memory module, a memory enhancement module, a decoder and a convolutional attention module;
[0011] Step 3: Divide the extracted key frames into historical frames, current frames, and future frames for multi-task learning;
[0012] Step 4: Based on the constructed dual-branch hybrid model, reconstruct and predict future frames using historical frames, reconstruct and predict historical frames using future frames, and reconstruct and predict current frames using historical frames and future frames;
[0013] Step 5: Train the dual-branch hybrid model of each task based on the acquired video dataset to obtain a hybrid model of trained fusion key frames and multi-task learning;
[0014] Step 6: Perform video anomaly detection based on the trained hybrid model of fused key frames and multi-task learning.
[0015] Furthermore, the key frame extraction module is specifically used to segment and extract features from the original video according to requirements, and then use the K-means clustering method to cluster the feature vectors into k clusters, where each cluster is represented by a representative center point, and then differentiate the images, calculate the absolute difference in frame pixel intensity, and select key frames based on a threshold.
[0016] Furthermore, the human skeleton extraction module is specifically used to extract human skeleton data from the input video frame using a human posture estimation algorithm.
[0017] Furthermore, the gradient loss of the encoder is:
[0018]
[0019] Where L B is the gradient loss of the encoder, t i , Indicates the time position, z, i represent the reconstructed position, the real position and the time series respectively, is a positive sample pair, is a pair of soft negative samples, is a pair of hard negative samples, and γ is a boundary hyperparameter.
[0020] Furthermore, in the memory module, a softmax function is used to calculate the memory addressing weight:
[0021]
[0022] where r i represents the memory addressing weight of the i-th row of the memory matrix, s i is the memory storage of the i-th row of the memory matrix, N is the number of rows of the memory matrix, y is the encoded feature vector, and d(·) is an approximate value measurement method.
[0023] Furthermore, the memory enhancement module includes a self-attention module and a feature projection module, and the feature projection module is used to map the input data to another space.
[0024] Furthermore, when the input is a historical frame, the loss of the decoder is:
[0025]
[0026] Where L D represents the loss of the decoder, T represents the number of time steps, t i , t represents the time position, and Represent the predicted future frame and the real future frame respectively.
[0027] Furthermore, the final loss of the multi-task learning is:
[0028] L=L B +λL D
[0029] Among them, L is the final loss of multi-task learning, L Bis the gradient loss of the encoder, L D is the decoder loss and λ is a hyperparameter adjusted based on the skeleton joint points and bounding box corner tracks.
[0030] Another aspect of the present invention provides a video anomaly detection system integrating key frames and multi-task hybrid models, comprising:
[0031] A video data set acquisition unit, used to acquire a video data set;
[0032] A hybrid model construction unit, used to construct a hybrid model integrating key frames and multi-task learning; the hybrid model includes a key frame extraction module and a skeleton-based dual-branch hybrid model; the key frame extraction module is used to extract key frames of the video; the skeleton-based dual-branch hybrid model includes two branches, reconstruction and prediction, each branch includes a human skeleton extraction module, an encoder, a memory module, a memory enhancement module, a decoder and a convolutional attention module;
[0033] A key frame division unit is used to divide the extracted key frames into historical frames, current frames and future frames for multi-task learning;
[0034] A multi-task learning unit, used to reconstruct and predict future frames using historical frames, reconstruct and predict historical frames using future frames, and reconstruct and predict current frames using historical frames and future frames based on the constructed dual-branch hybrid model;
[0035] A hybrid model training unit, used to train a dual-branch hybrid model of each task based on the acquired video data set, so as to obtain a hybrid model of trained fusion key frames and multi-task learning;
[0036] The video anomaly detection unit is used to perform video anomaly detection based on the hybrid model of trained fusion key frames and multi-task learning.
[0037] Compared with the prior art, the present invention has the following beneficial effects:
[0038] 1. Use the inter-frame differential key frame extraction technology based on K-means segmented clustering. This method extracts key frames by calculating the absolute difference of frame pixel intensity through segmented clustering and differential images. It has the advantages of less redundant information, better representation and generalization, improved robustness, and high computational efficiency in human recognition. Therefore, using this method to extract key frames from the original video frames and use them as the input of the model can reduce memory consumption in practical applications and improve the efficiency of video anomaly detection.
[0039] 2. Based on the key frame extraction, human skeleton data is extracted, and a hybrid model based on the reconstruction method and the prediction method is constructed, which effectively solves the problems of high misjudgment rate of abnormal data and susceptibility to environmental noise in the reconstruction method and the prediction method. The hybrid model comprehensively utilizes multi-source information and combines the advantages of the two methods to improve the practicability and accuracy of the model.
[0040] 3. A multi-task anomaly detection method based on a hybrid model, that is, comprehensively using historical frames to reconstruct and predict future frames, using future frames to reconstruct and predict historical frames, and using historical frames and future frames to reconstruct and predict current frames. The multi-task learning method is used to mine rich time series information, making up for the problems of missed detection and low accuracy of the previous single time series segment detection method, comprehensively capturing abnormal signals, and enhancing the prediction ability of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 A flowchart of a video anomaly detection method integrating key frames and a multi-task hybrid model according to an embodiment of the present invention;
[0042] Figure 2 A flow chart of key frame extraction based on segment clustering and inter-frame difference provided in an embodiment of the present invention;
[0043] Figure 3 A schematic diagram of the structure of a skeleton-based dual-branch hybrid model provided by an embodiment of the present invention;
[0044] Figure 4 A calculation flow chart of a memory enhancement module provided in an embodiment of the present invention;
[0045] Figure 5 The present invention is a schematic diagram of a video anomaly detection system architecture that integrates key frames and a multi-task hybrid model. DETAILED DESCRIPTION
[0046] For ease of understanding, some of the terms that appear in the specific embodiments of the present invention are explained as follows:
[0047] Key frame: A frame that is representative and contains important information in a video sequence. It can effectively summarize the video content and save storage space, while improving video retrieval efficiency. It is often used in applications such as video summarization, compression, and retrieval.
[0048] Unsupervised learning: is a machine learning paradigm whose goal is to discover patterns, structures, and relationships from unlabeled data without the use of labels or human intervention. In unsupervised learning, the algorithm attempts to infer the hidden structure or distribution of the data from the data without the need for pre-labeled outputs.
[0049] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:
[0050] like Figure 1 As shown, a video anomaly detection method integrating key frames and multi-task hybrid models includes:
[0051] Step 1: Obtain a video dataset, including a training dataset and a test dataset; specifically, download the HR-ShanghaiTech dataset. ShanghaiTech Campus is a challenging large-scale dataset containing 330 untrimmed training videos (all normal events) and 107 untrimmed test videos, with a total video frame number of 317k. These photos were taken by surveillance cameras at 13 different heights and angles on the campus of ShanghaiTech University, mainly reflecting the activities of pedestrians. The test set includes 130 abnormal events such as fighting, jumping, riding, and vehicles passing by. Since events such as vehicles passing by are not visual objects based on skeletal abnormal behavior detection, Morais et al. deleted 6 test videos containing abnormal events that are not related to humans, forming a subset called Human-Relevance (HR) ShanghaiTech. The present invention uses the HR-ShanghaiTech dataset sorted by Morais et al.
[0052] Step 2: Build a hybrid model that integrates keyframes and multi-task learning. The overall network structure is as follows: Figure 1 As shown in Figure 1, it mainly includes two modules: key frame extraction module and skeleton-based dual-branch hybrid model. Figure 2 As shown in the figure, in the key frame extraction module, the original video is first segmented and feature extracted according to the requirements, and then clustered using the K-means clustering method. The K-means algorithm divides the feature vector into k clusters, where each cluster is represented by a representative center point. After that, the image is differentially processed, the absolute difference of the frame pixel intensity is calculated, and the key frame is selected according to the threshold.
[0053] Step 3: Divide the extracted key frames into historical (video) frames, current (video) frames, and future (video) frames for multi-task learning. Taking historical frames as an example, a skeleton-based dual-branch hybrid model is constructed, such as Figure 3As shown, it includes two branches, reconstruction and prediction, each of which includes a human skeleton extraction module, an encoder, a memory module, a memory enhancement module, a decoder and a convolutional attention module. The human skeleton extraction module extracts human skeleton data from the input video frame through the open source human pose estimation algorithm OpenPose. It uses deep learning technology, especially convolutional neural network (CNN), to achieve accurate multi-human key point detection and pose estimation. Its steps include feature extraction, key point detection, pose estimation, etc. The extracted data is then input into the encoder, and the encoder performs convolution and pooling operations to implement related operations on the feature vector dimension and size to obtain the initial feature vector representation. The encoder gradient loss is defined as L B :
[0054]
[0055] where t i , Indicates the time position, z, i represent the reconstructed position, the real position and the time series respectively, is a positive sample pair (Positive Pair), is a soft negative sample pair (Soft NegativePair), is a hard negative sample pair (Hard Negative Pair), and γ is a boundary hyperparameter.
[0056] The obtained feature vector is then mapped to the memory module as the input of the memory module. The memory module can effectively increase the loss function value of the abnormal data sample during the training process, making the abnormal data sample easier to detect. The present invention uses the softmax function to calculate the memory addressing weight, which is defined as:
[0057]
[0058] where r i represents the memory addressing weight of the memory matrix in row i, s is the memory storage, s i is the memory storage of the i-th row of the memory matrix, N is the number of rows of the memory matrix, y is the encoded feature vector, and d(·) is an approximate value measurement method, which is defined as:
[0059]
[0060] Then through the memory enhancement module (such as Figure 4As shown in the figure, it is combined with the feature map to form a memory-enhanced feature map, which can help the neural network better utilize historical information and intermediate states to improve the performance and generalization ability of the model, so as to make the reconstruction and prediction effects better. Among them, the self-attention module introduces the self-attention mechanism, so that the model can better capture global dependencies when processing sequence data, thereby improving the expression ability of the model. The feature projection module refers to the operation of mapping the input data to another space, so as to better perform feature extraction and representation learning, so as to achieve the purpose of data dimensionality reduction, feature conversion, and feature enhancement. The encoded potential representation is then remapped back to the space of the original data through the decoder, and the convolutional attention module is used to enable the model to obtain a richer feature vector. The convolutional attention module is a common module in deep learning. By combining convolution operations and attention mechanisms, it can better capture feature information when processing images or sequence data, and improve the representation and generalization capabilities of the model. The decoder loss L D :
[0061]
[0062] Where T represents the number of time steps, t i , t represents the time position, and Represent the predicted future (video) frames and the real future (video) frames respectively.
[0063] It is worth noting that when the input data is different, the meanings of the parameters in the above formula are different. For example, when the input data is a future frame, in formula (4), and Represent the predicted historical frame and the real historical frame respectively.
[0064] This model is based on an unsupervised video anomaly detection learning method and is trained using reconstruction error and prediction error to jointly constrain the model.
[0065] Step 4: Based on the above skeleton-based dual-branch hybrid model, multi-task learning is performed, such as Figure 1 As shown, historical frames are used to reconstruct and predict future frames (Ftr), future frames are used to reconstruct and predict historical frames (Pst), and historical frames and future frames are used to reconstruct and predict current frames (Prs). Combining the above encoder and decoder losses, the final loss of multi-task learning is defined as:
[0066] L=L B +λL D #(5)
[0067] where λ is a hyperparameter and is adjusted based on the skeleton joint points and bounding box corner tracks.
[0068] Step 5: Use the prepared HR-ShanghaiTech training set as the input of step 2 to train the skeleton-based two-branch hybrid model for multiple tasks, set the hyperparameters, and train the entire network model for 70 epochs to learn the optimal parameters of the network model, thereby obtaining a trained hybrid model of fusion key frames and multi-task learning.
[0069] Step 6: Input the test dataset into the trained hybrid model of fusion key frames and multi-task learning for video anomaly detection.
[0070] Based on the above embodiments, Figure 5 As shown, the present invention also proposes a video anomaly detection system integrating key frames and multi-task hybrid models, comprising:
[0071] A video data set acquisition unit, used to acquire a video data set;
[0072] A hybrid model construction unit, used to construct a hybrid model integrating key frames and multi-task learning; the hybrid model includes a key frame extraction module and a skeleton-based dual-branch hybrid model; the key frame extraction module is used to extract key frames of the video; the skeleton-based dual-branch hybrid model includes two branches, reconstruction and prediction, each branch includes a human skeleton extraction module, an encoder, a memory module, a memory enhancement module, a decoder and a convolutional attention module;
[0073] A key frame division unit is used to divide the extracted key frames into historical frames, current frames and future frames for multi-task learning;
[0074] A multi-task learning unit, used to reconstruct and predict future frames using historical frames, reconstruct and predict historical frames using future frames, and reconstruct and predict current frames using historical frames and future frames based on the constructed dual-branch hybrid model;
[0075] A hybrid model training unit, used to train a dual-branch hybrid model of each task based on the acquired video data set, so as to obtain a hybrid model of trained fusion key frames and multi-task learning;
[0076] The video anomaly detection unit is used to perform video anomaly detection based on the hybrid model of trained fusion key frames and multi-task learning.
[0077] In summary, the present invention uses the segmented clustering inter-frame differential key frame extraction technology to remove duplicate frames with high similarity while highly retaining the original video information, obtain key frames, and use the extracted key frames as model input, saving storage space and solving the problems of large memory consumption and low detection efficiency in practical applications. On the basis of extracting human skeletal features, the present invention integrates a hybrid model by fusing a reconstruction-based method and a prediction-based method, thereby making up for the shortcomings of the reconstruction method's high misjudgment rate of abnormal data and the prediction method's susceptibility to data changes and environmental noise. The present invention jointly learns video frames of different time sequences to distinguish anomalies that occur in different time periods, thereby making up for the shortcomings of the future frame prediction method's easy omission of historical frame anomalies.
[0078] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.
Claims
1. A video anomaly detection method integrating key frames and multi-task hybrid models, characterized in that: include: Step 1: Get the video dataset; Step 2: construct a hybrid model integrating key frames and multi-task learning; the hybrid model includes a key frame extraction module and a skeleton-based dual-branch hybrid model; the key frame extraction module is used to extract key frames of the video; the skeleton-based dual-branch hybrid model includes two branches, reconstruction and prediction, each branch includes a human skeleton extraction module, an encoder, a memory module, a memory enhancement module, a decoder and a convolutional attention module; Step 3: Divide the extracted key frames into historical frames, current frames, and future frames for multi-task learning; Step 4: Based on the constructed dual-branch hybrid model, reconstruct and predict future frames using historical frames, reconstruct and predict historical frames using future frames, and reconstruct and predict current frames using historical frames and future frames; Step 5: Train the dual-branch hybrid model of each task based on the acquired video dataset to obtain a hybrid model of trained fusion key frames and multi-task learning; Step 6: Perform video anomaly detection based on the trained hybrid model of fused key frames and multi-task learning.
2. The video anomaly detection method integrating key frames and multi-task hybrid models according to claim 1, characterized in that: The key frame extraction module is specifically used to segment and extract features from the original video according to requirements, and then use the K-means clustering method to cluster the feature vectors into k clusters, where each cluster is represented by a representative center point, and then differentiate the images, calculate the absolute difference of frame pixel intensity, and select key frames according to the threshold.
3. The video anomaly detection method integrating key frames and multi-task hybrid models according to claim 1, characterized in that: The human skeleton extraction module is specifically used to extract human skeleton data from input video frames using a human posture estimation algorithm.
4. The video anomaly detection method integrating key frames and multi-task hybrid models according to claim 1, characterized in that: The gradient loss of the encoder is: Where L B is the gradient loss of the encoder, t i , Indicates the time position, z, i represent the reconstructed position, the real position and the time series respectively, is a positive sample pair, is a pair of soft negative samples, is a pair of hard negative samples, and γ is a boundary hyperparameter.
5. The video anomaly detection method integrating key frames and multi-task hybrid models according to claim 1, characterized in that: In the memory module, the softmax function is used to calculate the memory addressing weight: where r i represents the memory addressing weight of the i-th row of the memory matrix, s i is the memory storage of the i-th row of the memory matrix, N is the number of rows of the memory matrix, y is the encoded feature vector, and d(·) is an approximate value measurement method.
6. The video anomaly detection method integrating key frames and multi-task hybrid models according to claim 1, characterized in that: The memory enhancement module includes a self-attention module and a feature projection module, and the feature projection module is used to map input data to another space.
7. The video anomaly detection method integrating key frames and multi-task hybrid models according to claim 1, characterized in that: When the input is a historical frame, the loss of the decoder is: Where L D represents the loss of the decoder, T represents the number of time steps, t i , t represents the time position, and Represent the predicted future frame and the real future frame respectively.
8. The video anomaly detection method integrating key frames and multi-task hybrid models according to claim 1, characterized in that: The final loss of the multi-task learning is: L=L B +λL D Among them, L is the final loss of multi-task learning, L B is the gradient loss of the encoder, L D is the decoder loss and λ is a hyperparameter adjusted based on the skeleton joint points and bounding box corner tracks.
9. A video anomaly detection system integrating key frames and multi-task hybrid models, characterized in that: include: A video data set acquisition unit, used to acquire a video data set; A hybrid model construction unit, used to construct a hybrid model integrating key frames and multi-task learning; the hybrid model includes a key frame extraction module and a skeleton-based dual-branch hybrid model; the key frame extraction module is used to extract key frames of the video; the skeleton-based dual-branch hybrid model includes two branches, reconstruction and prediction, each branch includes a human skeleton extraction module, an encoder, a memory module, a memory enhancement module, a decoder and a convolutional attention module; A key frame division unit is used to divide the extracted key frames into historical frames, current frames and future frames for multi-task learning; A multi-task learning unit, used to reconstruct and predict future frames using historical frames, reconstruct and predict historical frames using future frames, and reconstruct and predict current frames using historical frames and future frames based on the constructed dual-branch hybrid model; A hybrid model training unit, used to train a dual-branch hybrid model of each task based on the acquired video data set, so as to obtain a hybrid model of trained fusion key frames and multi-task learning; The video anomaly detection unit is used to perform video anomaly detection based on the hybrid model of trained fusion key frames and multi-task learning.