Video abnormal event self-adaptive detection system and method based on time sequence modeling

Abstract

The invention discloses a video abnormal event self-adaptive detection system and method based on time sequence modeling, and relates to the technical field of video security, and the system comprises a data preprocessing module for multi-modal feature extraction and data cleaning enhancement; the spatial-temporal feature modeling module is used for constructing an auto-encoder and embedding abnormal features; the anomaly detection and classification module is used for anomaly detection and anomaly classification; the adaptive learning module is used for online incremental learning and feedback adjustment; the event positioning segmentation module is used for event positioning and event segmentation abstract generation; the real-time processing response module is used for performing real-time processing and abnormal alarm; the data management and storage module is used for constructing a video database and compressing and encrypting data; and the system monitoring and evaluation module is used for system performance monitoring and effect evaluation. According to the invention, through online incremental learning and a feedback adjustment mechanism, new abnormal events can be automatically learned and adapted, modeling is carried out in combination with multi-modal features and spatial-temporal features, the system can more accurately capture abnormal behaviors in a video, and the detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of video security technology, and more specifically, to a system and method for adaptively detecting abnormal video events based on time series modeling. Background Art

[0002] Time series modeling refers to the use of time series analysis technology to model and predict data that changes over time. In the field of video processing, this means extracting and analyzing the spatiotemporal features in video frame sequences to capture dynamic changes and patterns in the video. Through time series modeling, video data can be understood and processed more effectively, and useful information can be extracted for subsequent analysis and application. Video abnormal events refer to events that are inconsistent with normal or expected behaviors in video surveillance scenarios. These abnormal events may include intrusions, thefts, fights, fires, etc. They are usually sudden and unpredictable. The detection of video abnormal events is of great significance to security monitoring, intelligent transportation, public safety management and other fields.

[0003] Existing common methods for detecting abnormal events in videos mainly rely on established rules and thresholds, or are based on simple statistical models. These methods usually have poor generalization capabilities and are designed for specific scenarios and abnormal types. They are difficult to adapt to abnormal events in different scenarios and types. Since manually set rules and thresholds often cannot accurately capture the complex dynamic changes in the video, traditional methods have low accuracy and usually require a lot of computing resources and time to process video data, making it difficult to achieve real-time detection.

[0004] Therefore, traditional methods cannot automatically learn and adapt to new abnormal events, and require manual updating of rules and models. The noise and redundant information in video data can easily interfere with traditional methods and affect the detection effect. They only focus on the features of a single modality (such as image features) and it is difficult to effectively integrate multimodal features for anomaly detection.

[0005] To this end, some people explored the application of deep learning in video time series event detection (VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection). Initially, it was based on the Transformer-based AnomalyBERT model, which was trained through a data degradation scheme for time series anomaly detection, including abnormal event detection in video data. After a period of development, scholars proposed a new solution for the above anomaly detection scheme (AnomalyBERT: Self-Supervised Transformer for Time Series Anomaly Detection using Data Degradation Scheme), although VadCLIP proposed a new paradigm of applying visual language models (such as CLIP) to weakly supervised video anomaly detection (WSVAD), this method fully utilizes the powerful ability of CLIP in image-text alignment, and realizes coarse-grained and fine-grained detection of video anomaly events through visual classification and language-visual alignment dual-branch networks. However, due to the use of a dual-branch network and the processing of visual and text features, the computational complexity of VadCLIP is relatively high, which poses certain challenges for application scenarios with high real-time requirements. Anomaly BERT adopts a self-supervised learning method to generate abnormal samples through a data degradation scheme, and can be trained without a large amount of labeled data. Although it reduces the dependence on data annotation, the performance depends to a large extent on the choice of data degradation scheme. Different degradation schemes may have a significant impact on the anomaly detection effect, so it needs to be carefully designed and selected for specific tasks.

[0006] In summary, it is necessary to propose a video abnormal event adaptive detection system and method based on time series modeling to solve the above problems. Summary of the invention

[0007] The purpose of the present invention is to provide a system and method for adaptively detecting abnormal video events based on time series modeling to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The video abnormal event adaptive detection system based on time series modeling includes a data preprocessing module, a spatiotemporal feature modeling module, an abnormality detection classification module, an adaptive learning module, an event location segmentation module, a real-time processing response module, a data management storage module, and a system monitoring and evaluation module;

[0010] The data preprocessing module is used for video data multimodal feature extraction and video data cleaning and enhancement to obtain preprocessed video data;

[0011] The spatiotemporal feature modeling module is used to receive the preprocessed video data, and construct a spatiotemporal autoencoder to compress and reconstruct the spatiotemporal features in the video data, embed abnormal features according to the spatiotemporal features, and generate a feature embedding vector of the abnormal event;

[0012] The anomaly detection and classification module is used to receive the feature embedding vector to perform explainable anomaly detection and context-aware anomaly classification to obtain detection and classification data;

[0013] The adaptive learning module is used to receive the detection classification data and the evaluation result of the system monitoring and evaluation module, and perform online incremental learning and feedback adjustment on the system model;

[0014] The event location and segmentation module is used to receive the detection classification data, accurately locate the abnormal event and generate an event segmentation summary;

[0015] The real-time processing response module is used to perform real-time processing and immediate abnormal alarm according to the result of the precise event positioning and the event segmentation summary;

[0016] The data management storage module is used to store and manage video data and data generated by various modules in the system, build a distributed video database, and compress and encrypt video data;

[0017] The system monitoring and evaluation module is used to monitor in real time, collect performance data of other modules in the entire system, evaluate the effect of the system model and feed back the evaluation results to the adaptive learning module.

[0018] Preferably, the data preprocessing module includes a multimodal feature extraction unit and a data cleaning and enhancement unit;

[0019] The multimodal feature extraction unit is used to extract image, optical flow and audio multimodal features in the video by combining convolutional neural network CNN and Farneback optical flow algorithm with audio feature extraction technology to fully capture video information;

[0020] The data cleaning and enhancement unit uses Gaussian filtering to perform image denoising on the video information extracted by the multimodal feature extraction unit, and removes noise and redundant information in the video information through random cropping, rotation, and scaling data enhancement methods, thereby improving the system's generalization ability for video data processing.

[0021] Preferably, the spatiotemporal feature modeling module comprises a spatiotemporal autoencoder unit and an abnormal feature embedding unit;

[0022] The spatiotemporal autoencoder unit uses a convolutional long short-term memory network ConvLSTM to construct the spatiotemporal autoencoder, and trains the system model by minimizing the reconstruction error to compress and reconstruct the spatiotemporal features in the video and learn the pattern of normal behavior;

[0023] The abnormal feature embedding unit combines the attention mechanism and the Transformer structure to encode the spatiotemporal features and generate a feature embedding vector of the abnormal event to facilitate subsequent anomaly detection and classification.

[0024] Preferably, the anomaly detection and classification module comprises an explainable anomaly detection unit and a context-aware anomaly classification unit;

[0025] The explainable anomaly detection unit adopts the Holmes-VAD framework, combines video multimodality to fine-tune the system model, and based on the feature embedding vector of the abnormal event, uses precise timing supervision and rich multimodal instructions to achieve accurate abnormal event detection and provide explanations for abnormal events;

[0026] The context-aware anomaly classification unit combines the graph neural network GNN and the recurrent neural network RNN ​​to construct a context-aware classifier, and classifies the detected abnormal events by utilizing the spatiotemporal dependencies between video frames.

[0027] Preferably, the adaptive learning module includes an online incremental learning unit and a feedback adjustment unit;

[0028] The online incremental learning unit uses an incremental support vector machine to implement online learning of new abnormal events by the system model without retraining the entire system model;

[0029] The feedback adjustment unit designs a reinforcement learning mechanism based on user feedback, and optimizes system performance and improves user satisfaction by adjusting the anomaly detection threshold of the anomaly detection classification module and the parameters of the system model.

[0030] Preferably, the event location and segmentation module includes an accurate event location unit and an event segmentation summary generation unit;

[0031] The precise event positioning unit adopts a temporal action detection method, combined with boundary regression and action classification, to achieve precise positioning of the detected abnormal event in the video;

[0032] The event segmentation summary generation unit uses the non-maximum suppression (NMS) method to remove redundant video clips according to the abnormal event location, and combines the video summarization technology to generate a summary of the abnormal event for subsequent analysis and viewing.

[0033] Preferably, the real-time processing response module includes a real-time processing unit and an immediate abnormal alarm unit;

[0034] The real-time processing unit adopts a distributed processing architecture and GPU acceleration technology to improve the processing speed and efficiency of video data, ensuring that the system can process large-scale video data in real time;

[0035] The instant abnormal alarm unit integrates a message push system. When the abnormal detection and classification module detects the abnormal event, it immediately sends an alarm message to relevant personnel in the form of text messages, emails or instant messages.

[0036] Preferably, the data management storage module includes a distributed video database and a video data compression and encryption unit;

[0037] The distributed video database uses the HBase distributed database system combined with video indexing technology to achieve storage, management and distributed access of video data in the system;

[0038] The video data compression and encryption unit uses the H.265 / HEVC video compression algorithm and the AES encryption algorithm to compress and encrypt the video data, thereby reducing storage space occupancy and protecting data security.

[0039] Preferably, the system monitoring and evaluation module includes a system performance monitoring unit and a model effect evaluation unit;

[0040] The system performance monitoring unit integrates Prometheus and Grafana performance monitoring tools to monitor the system's processing speed and accuracy performance indicators in real time and display the system's operating status;

[0041] The model effect evaluation unit adopts cross-validation and A / B testing methods to regularly evaluate the detection effect of the system model, and sends the evaluation results to the adaptive learning module to adjust the system model parameters or update the system model to ensure continuous optimization and improvement of the system.

[0042] To achieve the purpose of the invention, based on the adaptive detection system for abnormal video events based on time series modeling described in the above technical solution and applied to the detection system, the present invention also proposes an adaptive detection method for abnormal video events based on time series modeling, the method comprising the following steps:

[0043] S1. Data preprocessing: using convolutional neural network (CNN) and Farneback algorithm combined with audio feature extraction technology to extract image, optical flow and audio multimodal features from the video, then using Gaussian filtering to perform image denoising on the video information extracted by the multimodal feature extraction unit, and enhancing the video information by random cropping, rotation and scaling methods to obtain preprocessed video data;

[0044] S2. Spatiotemporal feature modeling: A convolutional long short-term memory network ConvLSTM is used to construct a spatiotemporal autoencoder to compress and reconstruct the spatiotemporal features in the preprocessed video data. The spatiotemporal features are encoded by combining the attention mechanism and the Transformer structure to generate a feature embedding vector of the abnormal event.

[0045] S3. Anomaly detection and classification: The Holmes-VAD framework is used to fine-tune the system model in combination with video multimodality, and the abnormal event detection is realized based on the feature embedding vector of the abnormal event. At the same time, the context-aware classifier is constructed using the graph neural network GNN and the recurrent neural network RNN ​​to classify the detected abnormal events;

[0046] S4. Adaptive learning: Use incremental support vector machines to implement the system model's ability to learn new abnormal events online, and design a reinforcement learning mechanism based on user feedback to adjust the anomaly detection threshold and system model parameters of the anomaly detection classification module;

[0047] S5. Event location and segmentation: Using the temporal action detection method, combined with boundary regression and action classification, the detected abnormal event is located in the video, and the non-maximum suppression (NMS) and video summarization technology are used to generate a summary of the abnormal event;

[0048] S6. Real-time processing response: using distributed processing architecture and GPU acceleration technology to improve the video data processing speed, integrating message push system, and sending alarm information when the anomaly detection and classification module detects an anomaly;

[0049] S7. Data management and storage: Use the HBase distributed database system and video indexing technology to store and manage the system's video data and other data generated by the system, and use the H.265 / HEVC video compression algorithm and AES encryption algorithm to compress and encrypt the video data;

[0050] S8. System monitoring and evaluation: Integrate Prometheus and Grafana performance monitoring tools to monitor system performance in real time, use cross-validation and A / B testing methods to regularly evaluate the system model effect, and send the evaluation results to the adaptive learning module to adjust the system model parameters or update the system model.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] Through online incremental learning and feedback adjustment mechanism, the system of the present invention can automatically learn and adapt to new abnormal events without human intervention. Combined with multimodal feature extraction and spatiotemporal feature modeling technology, the system can more accurately capture abnormal behaviors in videos and improve detection accuracy. By adopting distributed processing architecture and GPU acceleration technology, the system can process large-scale video data in real time to meet practical application needs. By providing explanations and context-aware classification of abnormal events, the system can enhance users' understanding and trust in detection results. Through video data compression encryption and distributed database storage technology, the system can protect the security and privacy of video data. In summary, the present invention can effectively solve the problems existing in traditional methods, improve the accuracy, real-time and adaptability of abnormality detection, and provide strong technical support for the development of the video surveillance field. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A topological diagram of a video abnormal event adaptive detection system based on time series modeling according to the present invention;

[0054] Figure 2 This is a flow chart of the method for adaptively detecting abnormal video events based on time series modeling of the present invention. DETAILED DESCRIPTION

[0055] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0057] Example 1

[0058] See also Figure 1 ,This embodiment 1 proposes a video abnormal event adaptive detection system based on time series modeling, the system includes a data preprocessing module, a spatiotemporal feature modeling module, an abnormality detection classification module, an adaptive learning module, an event location segmentation module, a real-time processing response module, a data management storage module, and a system monitoring and evaluation module;

[0059] Among them, the data preprocessing module is used for multimodal feature extraction and data cleaning and enhancement, the spatiotemporal feature modeling module of this system is used to build spatiotemporal autoencoders and abnormal feature embedding, the anomaly detection and classification module of this system is used for explainable anomaly detection and context-aware anomaly classification, the adaptive learning module of this system is used for online incremental learning and feedback adjustment, the event location and segmentation module of this system is used for accurate event location and event segmentation summary generation, the real-time processing response module of this system is used for real-time processing and immediate abnormal alarm, the data management and storage module of this system is used to build a distributed video database and video data compression and encryption, and the system monitoring and evaluation module of this system is used for system performance monitoring and model effect evaluation.

[0060] In this embodiment 1, the data preprocessing module includes a multimodal feature extraction unit and a data cleaning and enhancement unit. The multimodal feature extraction unit combines audio feature extraction technology through convolutional neural network (CNN) and Farneback algorithm to extract image, optical flow and audio multimodal features in the video to comprehensively capture video information. The data cleaning and enhancement unit uses Gaussian filtering to perform image denoising on the video information extracted by the multimodal feature extraction unit, and removes noise and redundant information in the video through random cropping, rotation, and scaling data enhancement methods, thereby improving the generalization ability of the system model for video data processing.

[0061] The data preprocessing module is unidirectionally connected to the spatiotemporal feature modeling module to provide the latter with preprocessed video data. The data preprocessing module is used to extract image, optical flow, and audio multimodal features from the original video and clean and enhance them to eliminate noise and redundant information and improve data quality. The preprocessed video data is passed to the spatiotemporal feature modeling module for further analysis.

[0062] In this embodiment 1, the spatiotemporal feature modeling module includes a spatiotemporal autoencoder unit and an abnormal feature embedding unit. The spatiotemporal autoencoder unit adopts a convolutional long short-term memory network (ConvLSTM) to construct a spatiotemporal autoencoder, and trains the system model by minimizing the reconstruction error. It is used to compress and reconstruct the spatiotemporal features in the video and learn the pattern of normal behavior. The abnormal feature embedding unit combines the attention mechanism and the Transformer structure to encode the spatiotemporal features and generate a feature embedding vector of the abnormal event to facilitate subsequent anomaly detection and classification.

[0063] The spatiotemporal feature modeling module is unidirectionally connected to the anomaly detection and classification module. The spatiotemporal feature modeling module receives the preprocessed video data from the data preprocessing module, models and encodes the spatiotemporal features in the video data, embeds the anomaly features, generates feature embedding vectors of abnormal events, and then passes these vectors to the anomaly detection and classification module.

[0064] In this embodiment 1, the anomaly detection and classification module includes an explainable anomaly detection unit and a context-aware anomaly classification unit. The explainable anomaly detection unit adopts the Holmes-VAD framework and fine-tunes the system model in combination with video multimodality. Based on the feature embedding vector of the abnormal event, it uses precise timing supervision and rich multimodal instructions to achieve accurate anomaly detection and provide explanations for the abnormal events. The context-aware anomaly classification unit combines the graph neural network (GNN) and the recurrent neural network (RNN) to construct a context-aware classifier, and uses the spatiotemporal dependency between video frames to accurately classify the detected abnormal events.

[0065] The anomaly detection and classification module is unidirectionally connected to the event location and segmentation module to provide detection and classification effect data.

[0066] The anomaly detection and classification module is bidirectionally connected to the adaptive learning module for model updating and parameter adjustment.

[0067] The anomaly detection and classification module is unidirectionally connected to the system monitoring and evaluation module to provide detection and classification effect data.

[0068] The anomaly detection and classification module receives the output of the spatiotemporal feature modeling module to perform anomaly detection and classification. At the same time, the anomaly detection and classification module also communicates bidirectionally with the adaptive learning module to perform online learning and model adjustment based on new data. The anomaly detection and classification module also provides detection and classification effect data to the system monitoring and evaluation module for evaluating system performance.

[0069] In this embodiment 1, the adaptive learning module includes an online incremental learning unit and a feedback adjustment unit. The online incremental learning unit uses an incremental support vector machine to realize the system's online learning of new abnormal events without retraining the entire system model. The feedback adjustment unit designs a reinforcement learning mechanism based on user feedback, which optimizes system performance and improves user satisfaction by adjusting the anomaly detection threshold and system model parameters of the anomaly detection classification module.

[0070] The adaptive learning module is bidirectionally connected to the anomaly detection and classification module to receive detection data and provide model updates.

[0071] The adaptive learning module is bidirectionally connected to the system monitoring and evaluation module, and adjusts the learning parameters according to the evaluation results of the system monitoring and evaluation module.

[0072] The adaptive learning module updates the system model by learning new abnormal events online, and communicates bidirectionally with the anomaly detection and classification module and the system monitoring and evaluation module to optimize system performance and improve user satisfaction.

[0073] In the present embodiment 1, the event location and segmentation module includes an accurate event location unit and an event segmentation summary generation unit. The accurate event location unit adopts a temporal action detection method, combined with boundary regression and action classification, to achieve accurate location of abnormal events in the video. The event segmentation summary generation unit uses a non-maximum suppression (NMS) method to remove redundant video clips based on the location of abnormal events in the video, and combines video summarization technology to generate a summary of abnormal events for subsequent analysis and viewing.

[0074] The event location and segmentation module is unidirectionally connected to the real-time processing response module. The event location and segmentation module receives the output of the anomaly detection and classification module, locates and segments the abnormal events, then generates a summary of the abnormal events and passes this information to the real-time processing response module.

[0075] In this embodiment 1, the real-time processing response module includes a real-time processing unit and an immediate abnormal alarm unit. The real-time processing unit adopts a distributed processing architecture and GPU acceleration technology to improve the processing speed and efficiency of video data, ensuring that the system can process large-scale video data in real time. The immediate abnormal alarm unit integrates a message push system to ensure that when the abnormal detection and classification module detects an abnormal event, an alarm information is immediately sent to relevant personnel in the form of SMS, email or instant message.

[0076] The real-time processing response module is unidirectionally connected to the data management storage module to store alarm information. The real-time processing response module receives the output of the event location segmentation module and performs real-time processing and immediate abnormal alarm. The alarm information can also be stored in the data management storage module.

[0077] In this embodiment 1, the data management and storage module includes a distributed video database and a video data compression and encryption unit. The distributed video database adopts the HBase distributed database system combined with video indexing technology to realize the storage, management and distributed access of massive video data in the system. The video data compression and encryption unit adopts the H.265 / HEVC video compression algorithm and the AES encryption algorithm to compress and encrypt the video data, reduce storage space occupancy and protect data security.

[0078] The data management storage module is bidirectionally connected to the system monitoring and evaluation module to provide storage data and performance monitoring data. The data management storage module is responsible for storing and managing video data and other data generated by the system (such as alarm information), and also communicates bidirectionally with the system monitoring and evaluation module to provide storage data and performance monitoring data.

[0079] In this embodiment 1, the system monitoring and evaluation module includes a system performance monitoring unit and a model effect evaluation unit. The system performance monitoring unit integrates Prometheus and Grafana performance monitoring tools to monitor and collect the system's processing speed and accuracy performance indicators in real time, and display the system operation status. The model effect evaluation unit adopts cross-validation and A / B testing methods to regularly evaluate the detection effect of the system model, and sends the evaluation results to the adaptive learning module to adjust the system model parameters or update the system model to ensure continuous optimization and improvement of the system.

[0080] The system monitoring and evaluation module is unidirectionally connected to all other modules to monitor and collect system operation, performance data and evaluation results. The system monitoring and evaluation module is responsible for real-time monitoring of system performance, evaluating the detection effect of the system model, providing adjustment suggestions based on the evaluation results, and sending the evaluation results to the adaptive learning module to adjust the system model parameters or update the system model.

[0081] Example 2

[0082] See also Figure 2 Based on the system of Example 1, the present invention further proposes a method for adaptively detecting abnormal video events based on time series modeling, which specifically includes the following steps in practical application:

[0083] S1: Data preprocessing:

[0084] Firstly, the convolutional neural network (CNN) and Farneback algorithm are combined with audio feature extraction technology to extract multimodal features such as image, optical flow and audio from the video.

[0085] These features can fully capture the information in the video and provide a basis for subsequent analysis. Then, the extracted features are cleaned and enhanced;

[0086] Use Gaussian filtering to reduce image noise and remove noise from the video;

[0087] At the same time, the generalization ability of the system model is improved through data enhancement methods such as random cropping, rotation, and scaling.

[0088] S2: Spatiotemporal feature modeling:

[0089] A convolutional long short-term memory network (ConvLSTM) is used to construct a spatiotemporal autoencoder, and the system model is trained by minimizing the reconstruction error;

[0090] This step aims to compress and reconstruct the spatiotemporal features in the video, learn the pattern of normal behavior, combine the attention mechanism and Transformer structure, encode the spatiotemporal features, and generate the feature embedding vector of abnormal events;

[0091] These vectors will be used for subsequent anomaly detection classification.

[0092] S3: Anomaly Detection Classification:

[0093] The Holmes-VAD framework is used to fine-tune the system model in combination with video multimodality. Based on the feature embedding vector of abnormal events, precise temporal supervision and rich multimodal instructions are used to achieve accurate abnormal event detection and provide explanations for abnormal events. A context-aware classifier is constructed by combining graph neural network (GNN) and recurrent neural network (RNN).

[0094] The spatiotemporal dependencies between video frames are exploited to accurately classify detected abnormal events.

[0095] S4: Adaptive Learning:

[0096] Incremental support vector machine is used to enable the system to learn new abnormal events online;

[0097] Through this step, there is no need to retrain the entire spatiotemporal autoencoder model, and the model knowledge can be updated efficiently, and a reinforcement learning mechanism based on user feedback can be designed;

[0098] By adjusting the anomaly detection threshold and system model parameters of the anomaly detection classification module, the system performance can be optimized and user satisfaction can be improved.

[0099] S5: Event location segmentation:

[0100] The Temporal Action Detection method is used, combined with boundary regression and action classification, to accurately locate abnormal events in the video. The Non-Maximum Suppression (NMS) method is used to remove redundant video clips. Combined with video summarization technology, a summary of abnormal events is generated, which is helpful for subsequent analysis and viewing.

[0101] S6: Real-time processing response:

[0102] Adopt distributed processing architecture and GPU acceleration technology to improve the processing speed and efficiency of video data;

[0103] Ensure that the system can process large-scale video data in real time, integrate the message push system, and immediately send alarm information to relevant personnel in the form of SMS, email or instant message when an abnormal event is detected.

[0104] S7: Data management storage:

[0105] The HBase distributed database system is used in combination with video indexing technology to achieve storage, management and distributed access of massive video data. The H.265 / HEVC video compression algorithm and AES encryption algorithm are used to compress and encrypt video data.

[0106] This step helps reduce storage space usage and protect data security.

[0107] S8: System Monitoring Assessment:

[0108] Integrate Prometheus and Grafana performance monitoring tools to monitor the system's processing speed, accuracy and other performance indicators in real time, display the system's operating status, use cross-validation and A / B testing methods to regularly evaluate the detection effect of the system model and send the evaluation results to the adaptive learning module to adjust the system model parameters or update the system model;

[0109] This step ensures continuous optimization and improvement of the system.

[0110] Example 3

[0111] Based on Example 2, this Example 3 further illustrates steps S2, S3, S4, and S5 in Example 2, as follows:

[0112] S2: Spatiotemporal feature modeling

[0113] S21: Feature Compression and Reconstruction:

[0114] A spatiotemporal autoencoder network is constructed using ConvLSTM units, where the encoder part is responsible for compressing the input spatiotemporal features into a low-dimensional representation, and the decoder part attempts to reconstruct the original features based on the low-dimensional representation.

[0115] The system model parameters are optimized by minimizing the reconstruction error (such as mean square error MSE), so that the system model can effectively learn the spatiotemporal characteristic patterns of normal behavior.

[0116] S22: Application of attention mechanism:

[0117] In the process of spatiotemporal feature encoding, an attention mechanism is introduced to highlight important features and suppress irrelevant features. After the output of ConvLSTM, an attention layer is added to calculate the weight of each feature point, and the weighted sum is used to obtain the final spatiotemporal feature representation, as shown in equations (1), (2) and (3):

[0118] Z=Encoder(X) (1)

[0119] In the formula, the encoder receives high-dimensional spatiotemporal features X and compresses them into low-dimensional representation Z through a convolutional long short-term memory network (ConvLSTM).

[0120] X^=Decoder(Z) (2)

[0121] Here, the encoder receives a low-dimensional representation Z and attempts to reconstruct the original spatiotemporal features X^.

[0122]

[0123] In the formula, the system model parameters are optimized by minimizing the reconstruction error, so that the system model can effectively learn the spatiotemporal characteristic patterns of normal behavior.

[0124] S23: Transformer structure encoding:

[0125] The spatiotemporal features are further encoded at a high level to generate feature embedding vectors of abnormal events.

[0126] The features output by the attention mechanism are used as the input of the Transformer. Through the multi-head self-attention mechanism and feedforward neural network layer, the complex relationship between the features is learned and the feature embedding vector is output.

[0127] S24: Feature embedding vector generation:

[0128] The output of Transformer is the feature embedding vector of the abnormal event.

[0129] These vectors will serve as input to the subsequent anomaly detection classifier to distinguish normal behaviors from abnormal events.

[0130] S25: Model training and optimization:

[0131] Training data: Use video data labeled as normal to train the model.

[0132] Optimization algorithm: Select Adam optimizer and set appropriate learning rate and batch size.

[0133] Stop condition: Stop training when the reconstruction error converges to a preset threshold or reaches the maximum number of training rounds.

[0134] S26: Model evaluation and validation:

[0135] Evaluation Metrics: The mean and standard deviation of the reconstruction error are used to evaluate the performance of the system model.

[0136] Validation data: Use an independent validation dataset to verify the generalization ability of the system model and ensure that the system model can accurately reconstruct the spatiotemporal characteristics of normal behavior.

[0137] S3: Anomaly Detection Classification

[0138] S31: Holmes-VAD framework application:

[0139] The system model is fine-tuned in combination with video multimodality to achieve accurate anomaly detection.

[0140] Implementation method: Input the spatiotemporal feature embedding vector into the Holmes-VAD framework and use the classifier in the framework to perform anomaly detection.

[0141] S32: Multimodal instruction fusion:

[0142] Combine multimodal features such as image, optical flow, and audio to improve the accuracy of anomaly detection.

[0143] Implementation method: Concatenate or weightedly fuse the feature embedding vectors of different modalities as the input of the Holmes-VAD framework.

[0144] S33: Precise Timing Supervision:

[0145] Using precise timing supervision information, the system model is trained to accurately identify the time points of abnormal events.

[0146] Implementation: During the training process, use labeled data containing timestamps of abnormal events to optimize the system model's sensitivity to abnormal timing.

[0147] S34: Context-aware classifier construction:

[0148] By using the spatiotemporal dependency between video frames, abnormal events are accurately classified. The spatiotemporal feature embedding vector is input into the classifier composed of GNN and RNN. The category of abnormal events is learned through the graph structure and time series information, as shown in formula (4):

[0149] y=C(E) (4);

[0150] In the formula, the classifier C is composed of a graph neural network (GNN) and a recurrent neural network (RNN). It receives the spatiotemporal feature embedding vector E and outputs the category label y of the abnormal event. The GNN is responsible for capturing the graph structure information, and the RNN is responsible for capturing the timing information.

[0151] S35: Classification result output:

[0152] Output: The output of the classifier is the category label of the abnormal event.

[0153] Purpose: Used for subsequent abnormal event processing and alarm.

[0154] S36: Model tuning and verification:

[0155] Tuning method: Adjust model parameters such as learning rate, batch size, etc. according to the classification results.

[0156] Verification method: Use cross-validation method to evaluate the classification performance of the system model to ensure that the system model can achieve good results on different data sets.

[0157] S4: Adaptive Learning

[0158] S41: Incremental Support Vector Machine Application:

[0159] Enable the system model to learn new abnormal events online without retraining the entire system model.

[0160] Implementation method: When a new abnormal event is detected, its feature embedding vector is used as a new sample to update the support vector set of the incremental support vector machine.

[0161] S42: Model knowledge update:

[0162] Enables system models to adapt to changing types of abnormal events.

[0163] S43: User feedback mechanism design:

[0164] Optimize system performance and improve user satisfaction through user feedback.

[0165] Design a user feedback interface to collect users’ evaluation and suggestions on detection results, and adjust anomaly detection thresholds and system model parameters based on feedback.

[0166] S44: Threshold adjustment and optimization:

[0167] Dynamically adjust the anomaly detection threshold based on the accuracy and missed detection rate provided by users.

[0168] While ensuring detection accuracy, reduce false positives and missed positives.

[0169] S45: Model parameter tuning:

[0170] Use grid search or random search method to find the optimal parameter combination in the preset parameter space to improve the generalization ability and detection performance of the system model.

[0171] S46: System performance evaluation:

[0172] Select accuracy, recall rate, F1 score, etc. as evaluation indicators, use independent test data sets, regularly evaluate the detection performance of the system, and adjust the system model parameters or update the system model based on the evaluation results.

[0173] S5: Event localization segmentation

[0174] S51: Construction of temporal action detection network:

[0175] Based on the combination of convolutional neural network (CNN) and recurrent neural network (RNN), a temporal action detection network is constructed to process the feature embedding vector output by the spatiotemporal feature modeling module, and output the time boundary and category of abnormal events in the video;

[0176] S52: Boundary Regression and Action Classification:

[0177] Use the temporal action detection network to perform boundary regression on the input feature embedding vector to predict the start and end time of the abnormal event, and perform action classification to determine the specific category of the abnormal event (including intrusion, fire);

[0178] S53: Candidate event generation:

[0179] Based on the results of boundary regression, candidate abnormal event segments are generated. Each segment contains a possible abnormal event. These candidate event segments may overlap and need further processing to remove redundancy.

[0180] S54: Non-maximum suppression (NMS):

[0181] The non-maximum suppression (NMS) method is used to screen and remove duplicate candidate event segments. NMS compares the confidence scores and overlap levels of candidate event segments, retains segments with high confidence and little overlap, and removes redundant and repeated segments.

[0182] S55: Video summary generation:

[0183] The abnormal event clips filtered by NMS are further processed to generate abnormal event summaries, which include key frames, short video clips or text description information of the abnormal events, to facilitate subsequent analysis and review;

[0184] S56: Result output and storage:

[0185] The time boundary, category and summary results of abnormal events segmented by event location are output to the real-time processing response module for triggering alarms and responses, and the results are stored in the data management storage module for subsequent query and analysis.

[0186] The other steps or technical details of this embodiment 3 are the same as those of embodiment 2 and will not be repeated here.

[0187] Through online incremental learning and feedback adjustment mechanism, the system of the present invention can automatically learn and adapt to new abnormal events without human intervention. Combined with multimodal feature extraction and spatiotemporal feature modeling technology, the system can more accurately capture abnormal behaviors in videos and improve detection accuracy. By adopting distributed processing architecture and GPU acceleration technology, the system can process large-scale video data in real time to meet practical application needs. By providing explanations and context-aware classification of abnormal events, the system can enhance users' understanding and trust in detection results. Through video data compression and encryption and distributed database storage technology, the system can protect the security and privacy of video data.

[0188] In summary, the adaptive detection system and method for video abnormal events based on time series modeling have significant advantages and broad application prospects. They can effectively solve the problems existing in traditional methods, improve the accuracy, real-time and adaptability of anomaly detection, and provide technical support for the development of the video surveillance field.

[0189] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. The adaptive video abnormal event detection system based on time series modeling is characterized by: It includes data preprocessing module, spatiotemporal feature modeling module, anomaly detection and classification module, adaptive learning module, event location and segmentation module, real-time processing and response module, data management and storage module, and system monitoring and evaluation module; The data preprocessing module is used for video data multimodal feature extraction and video data cleaning and enhancement to obtain preprocessed video data; The spatiotemporal feature modeling module is used to receive the preprocessed video data, and construct a spatiotemporal autoencoder to compress and reconstruct the spatiotemporal features in the video data, embed abnormal features according to the spatiotemporal features, and generate a feature embedding vector of the abnormal event; The anomaly detection and classification module is used to receive the feature embedding vector to perform explainable anomaly detection and context-aware anomaly classification to obtain detection and classification data; The adaptive learning module is used to receive the detection classification data and the evaluation result of the system monitoring and evaluation module, and perform online incremental learning and feedback adjustment on the system model; The event location and segmentation module is used to receive the detection classification data, accurately locate the abnormal event and generate an event segmentation summary; The real-time processing response module is used to perform real-time processing and immediate abnormal alarm according to the result of the precise event positioning and the event segmentation summary; The data management storage module is used to store and manage video data and data generated by various modules in the system, build a distributed video database, and compress and encrypt video data; The system monitoring and evaluation module is used to monitor in real time, collect performance data of other modules in the entire system, evaluate the effect of the system model and feed back the evaluation results to the adaptive learning module.

2. The video abnormal event adaptive detection system based on time series modeling according to claim 1 is characterized in that: The data preprocessing module includes a multimodal feature extraction unit and a data cleaning and enhancement unit; The multimodal feature extraction unit is used to extract image, optical flow and audio multimodal features in the video by combining convolutional neural network CNN and Farneback optical flow algorithm with audio feature extraction technology to fully capture video information; The data cleaning and enhancement unit uses Gaussian filtering to perform image denoising on the video information extracted by the multimodal feature extraction unit, and removes noise and redundant information in the video information through random cropping, rotation, and scaling data enhancement methods, thereby improving the system's generalization ability for video data processing.

3. The video abnormal event adaptive detection system based on time series modeling according to claim 1 is characterized in that: The spatiotemporal feature modeling module includes a spatiotemporal autoencoder unit and an abnormal feature embedding unit; The spatiotemporal autoencoder unit uses a convolutional long short-term memory network ConvLSTM to construct the spatiotemporal autoencoder, and trains the system model by minimizing the reconstruction error to compress and reconstruct the spatiotemporal features in the video and learn the pattern of normal behavior; The abnormal feature embedding unit combines the attention mechanism and the Transformer structure to encode the spatiotemporal features and generate a feature embedding vector of the abnormal event to facilitate subsequent anomaly detection and classification.

4. The video abnormal event adaptive detection system based on time series modeling according to claim 3 is characterized in that: The anomaly detection and classification module includes an explainable anomaly detection unit and a context-aware anomaly classification unit; The explainable anomaly detection unit adopts the Holmes-VAD framework, combines video multimodality to fine-tune the system model, and based on the feature embedding vector of the abnormal event, uses precise timing supervision and rich multimodal instructions to achieve accurate abnormal event detection and provide explanations for abnormal events; The context-aware anomaly classification unit combines the graph neural network GNN and the recurrent neural network RNN ​​to construct a context-aware classifier, and classifies the detected abnormal events by utilizing the spatiotemporal dependencies between video frames.

5. The video abnormal event adaptive detection system based on time series modeling according to claim 4 is characterized in that: The adaptive learning module includes an online incremental learning unit and a feedback adjustment unit; The online incremental learning unit uses an incremental support vector machine to implement online learning of new abnormal events by the system model without retraining the entire system model; The feedback adjustment unit designs a reinforcement learning mechanism based on user feedback, and optimizes system performance and improves user satisfaction by adjusting the anomaly detection threshold of the anomaly detection classification module and the parameters of the system model.

6. The video abnormal event adaptive detection system based on time series modeling according to claim 4 is characterized in that: The event location and segmentation module includes an accurate event location unit and an event segmentation summary generation unit; The precise event positioning unit adopts a temporal action detection method, combined with boundary regression and action classification, to achieve precise positioning of the detected abnormal event in the video; The event segmentation summary generation unit uses the non-maximum suppression (NMS) method to remove redundant video clips according to the abnormal event location, and combines the video summarization technology to generate a summary of the abnormal event for subsequent analysis and viewing.

7. The video abnormal event adaptive detection system based on time series modeling according to claim 4 is characterized in that: The real-time processing response module includes a real-time processing unit and an immediate abnormal alarm unit; The real-time processing unit adopts a distributed processing architecture and GPU acceleration technology to improve the processing speed and efficiency of video data, ensuring that the system can process large-scale video data in real time; The instant abnormal alarm unit integrates a message push system. When the abnormal detection and classification module detects the abnormal event, it immediately sends an alarm message to relevant personnel in the form of text messages, emails or instant messages.

8. The video abnormal event adaptive detection system based on time series modeling according to claim 1 is characterized in that: The data management storage module includes a distributed video database and a video data compression and encryption unit; The distributed video database uses the HBase distributed database system combined with video indexing technology to achieve storage, management and distributed access of video data in the system; The video data compression and encryption unit uses the H.265 / HEVC video compression algorithm and the AES encryption algorithm to compress and encrypt the video data, thereby reducing storage space occupancy and protecting data security.

9. The video abnormal event adaptive detection system based on time series modeling according to claim 1 is characterized in that: The system monitoring and evaluation module includes a system performance monitoring unit and a model effect evaluation unit; The system performance monitoring unit integrates Prometheus and Grafana performance monitoring tools to monitor the system's processing speed and accuracy performance indicators in real time and display the system's operating status; The model effect evaluation unit adopts cross-validation and A / B testing methods to regularly evaluate the detection effect of the system model, and sends the evaluation results to the adaptive learning module to adjust the system model parameters or update the system model to ensure continuous optimization and improvement of the system.

10. A method for adaptively detecting abnormal video events based on time series modeling, applied to a system for adaptively detecting abnormal video events based on time series modeling as claimed in any one of claims 1 to 9, characterized in that: The method comprises the following steps: S1. Data preprocessing: using convolutional neural network (CNN) and Farneback algorithm combined with audio feature extraction technology to extract image, optical flow and audio multimodal features from the video, then using Gaussian filtering to perform image denoising on the video information extracted by the multimodal feature extraction unit, and enhancing the video information by random cropping, rotation and scaling methods to obtain preprocessed video data; S2. Spatiotemporal feature modeling: A convolutional long short-term memory network ConvLSTM is used to construct a spatiotemporal autoencoder to compress and reconstruct the spatiotemporal features in the preprocessed video data. The spatiotemporal features are encoded by combining the attention mechanism and the Transformer structure to generate a feature embedding vector of the abnormal event. S3. Anomaly detection and classification: The Holmes-VAD framework is used to fine-tune the system model in combination with video multimodality, and the abnormal event detection is realized based on the feature embedding vector of the abnormal event. At the same time, the context-aware classifier is constructed using the graph neural network GNN and the recurrent neural network RNN ​​to classify the detected abnormal events; S4. Adaptive learning: Use incremental support vector machines to implement the system model's ability to learn new abnormal events online, and design a reinforcement learning mechanism based on user feedback to adjust the anomaly detection threshold and system model parameters of the anomaly detection classification module; S5. Event location and segmentation: Using the temporal action detection method, combined with boundary regression and action classification, the detected abnormal event is located in the video, and the non-maximum suppression (NMS) and video summarization technology are used to generate a summary of the abnormal event; S6. Real-time processing response: using distributed processing architecture and GPU acceleration technology to improve the video data processing speed, integrating message push system, and sending alarm information when the anomaly detection and classification module detects an anomaly; S7. Data management and storage: Use the HBase distributed database system and video indexing technology to store and manage the system's video data and other data generated by the system, and use the H.265 / HEVC video compression algorithm and AES encryption algorithm to compress and encrypt the video data; S8. System monitoring and evaluation: Integrate Prometheus and Grafana performance monitoring tools to monitor system performance in real time, use cross-validation and A / B testing methods to regularly evaluate the system model effect, and send the evaluation results to the adaptive learning module to adjust the system model parameters or update the system model.

Images