Video abnormal event detection method
By constructing a target spatiotemporal cube using YOLOv7 and dense optical flow, and combining a teacher-student network and a multimodal temporal Transformer architecture, the problems of high annotation cost and high false alarm rate in video anomaly detection are solved, achieving refined modeling and adaptive detection.
Patent Information
- Application Number
- CN202511508234.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-22
- Publication Date
- 2025-11-21
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing video anomaly detection solutions heavily rely on a large number of accurately labeled anomaly samples, which are costly to label. Multimodal analysis frameworks struggle to integrate information from multiple sources, and traditional temporal modeling has limited ability to capture long-distance dependencies, resulting in high false alarm rates and insensitivity to progressive anomaly detection.
We employ YOLOv7 and dense optical flow-based spatiotemporal cube modeling of targets, combined with multi-level feature distillation of teacher-student networks and a multimodal temporal Transformer architecture. Through unsupervised anomaly detection and dynamic threshold decision-making, we achieve target localization, feature fusion, and temporal analysis.
It improves the accuracy and practicality of video anomaly detection, realizes refined behavior modeling, long-term time-series analysis and scene adaptation capabilities, and reduces the false alarm rate.
Smart Images
Figure CN120997745A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, specifically to a method for detecting abnormal events in videos. Background Technology
[0002] Existing video anomaly detection solutions heavily rely on supervised training with a large number of precisely labeled anomaly samples, while anomalies are rare and labeling is costly in real-world scenarios. Secondly, they often employ single-modal analysis frameworks or focus only on appearance or motion features, making it difficult to effectively integrate multi-source information to cope with complex scene changes. Furthermore, traditional temporal modeling methods have limited ability to capture long-distance dependencies and often use fixed threshold judgment mechanisms, resulting in insensitivity to progressive anomaly detection and a high false alarm rate, which limits the accuracy and practicality of existing technical solutions in real-world scenarios. Summary of the Invention
[0003] To address the aforementioned technical problems, a video anomaly event detection method is provided, and this technical solution resolves the problems described above.
[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A video anomaly event detection method, comprising: S1. Based on the big data set of original videos of known abnormal events, the pre-trained target detector locates and marks the confidence of targets in the big data frames of original videos of known abnormal events, extracts the target motion cue feature vector, and establishes spatiotemporal cubes of each target in the original videos of known abnormal events. S2. Based on the big data set of original videos of known abnormal events, a teacher-student network architecture is pre-trained to perform anomaly perception distillation on targets in the spatiotemporal cube of each target in the original videos of known abnormal events, extract the action feature vector of the target's abnormal motion behavior, and establish a sample set of abnormal events of each target in the original videos of known abnormal events. S3. Based on the multimodal temporal Transformer architecture, the teacher-student network architecture is used as the visual feature encoder and the target detector is used as the motion feature encoder. Feature fusion is performed on the spatiotemporal cubes of each target and the sample sets of each target abnormal event in the original video of known abnormal events. The abnormal scores under the unit time frame in the big data of the original video of known abnormal events are verified, and it is determined whether each target in the big data of the original video of known abnormal events is in an abnormal state.
[0005] Preferably, step S1 specifically includes: The backbone network CSPDarknet based on YOLOv7 is pre-trained using original videos of known anomalous events and the COCO dataset. It utilizes the bounding boxes and class labels of targets in large data frames of original videos of known anomalous events to adjust the weights of the backbone network relative to each specified scene for the single-stage target detector at the anchor point, thus obtaining the single-stage target detector with anchor point. Based on the single-stage target detector of the anchor point, the original video frames of the original video big data set of known abnormal events are input one by one to generate the bounding box coordinates and category labels of the original video frames. Substitute them into the sigmoid function to obtain the matching degree between the bounding box of the original video frame and the bounding box of the real object. Based on the bounding box coordinates and class labels of the original video frame, calculate the area of the intersection region of the predicted box and the ground truth box and the area of their union region to determine the probability of object presence at the bounding box coordinates of the original video frame. Based on the matching degree between the bounding box of the original video frame and the bounding box of the real object, and the probability of object existence at the bounding box coordinates of the original video frame, the confidence of the bounding box coordinates and class label of the original video frame is calculated, and a sequence of bounding box coordinates and class label of the original video frame is constructed.
[0006] Preferably, step S1 further includes: Based on a large dataset of original videos containing known anomalous events, and following the dense optical flow algorithm, the mean and standard deviation of the optical flow motion vector between each adjacent frame in the sequence of bounding box coordinates and category labels of the original video frames are marked. The optical flow amplitude of each pixel in the bounding box of the original video frame is calculated. The bounding box of the original video frame is divided into several intervals according to the magnitude of the amplitude, and the histogram of the optical flow amplitude of the bounding box of the original video frame is obtained. The mean and standard deviation of the optical flow motion vector between each adjacent frame in the bounding box of the original video frame are concatenated with the histogram of the optical flow amplitude in the bounding box of the original video frame to obtain the motion cue feature vector of each target in the original video frame. Based on the multi-target tracking algorithm, the motion cue feature vectors of the same target in the original video frame are associated to generate the trajectory vector of the target in the original video frame. The bounding box and category label of each target in the original video big data set of known abnormal events are extracted to obtain the three-dimensional data block of the bounding box and category label of each target in the original video big data set of known abnormal events. Based on the bounding box and category label 3D data block of each target in the original video big data set of known abnormal events, the trajectory vector image regions of the targets in all the captured original video frames are unified to the same spatial size, resulting in a spatiotemporal cube set of each target in the original video of known abnormal events.
[0007] Preferably, step S2 specifically includes: Based on Bayesian prior distribution, the original video big data set of known abnormal events is filtered and divided to establish a reference knowledge base of abnormal behavior action feature vectors of known abnormal events in each specified scenario. Based on the Teacher-Student deep learning network architecture, and following the ResNet-50 deep residual network, a teacher network is established. The abnormal behavior action feature vectors of known abnormal events in various specified scenarios are used as a reference to the knowledge base to pre-train the teacher network. Based on the spatiotemporal cube sets of each target in the original video of known anomalous events, the spatiotemporal cube sets of each target are sorted and filtered according to the confidence of the bounding box coordinates and category labels of the original video frames to construct a high-confidence sample dataset of each target anomalous event in the original video of known anomalous events. Based on the Teacher-Student deep learning network architecture, and using the MobileNetV2 lightweight neural network, a student network is established. The high-confidence sample dataset of each target abnormal event in the original video of known abnormal events is used as the input to the teacher network and the student network. The multi-level abnormal features output by the teacher network and the corresponding level of abnormal features output by the student network are obtained.
[0008] Preferably, step S2 further includes: For the abnormal features corresponding to the student network output level, an adaptive pooling layer is used for convolution processing; Based on the multi-level anomaly features output by the teacher network and the corresponding level anomaly features output by the student network from the set of spatiotemporal cubes of each target in the original video of the known anomaly event, the anomaly feature vectors of the teacher network level and the anomaly feature vectors of the student network level of each target spatiotemporal cube in the original video of the known anomaly event are constructed by the difference splicing method. Based on the multi-level perceptual distillation loss function, the sum of feature differences at each level between the abnormal feature vectors of each target spatiotemporal cube teacher network level and the abnormal feature vectors of each target spatiotemporal cube student network level in the original video of known abnormal events is calculated. Then, a weighted regression is performed on the loss of each level to update the intermediate layer weights of the abnormal feature vectors of each target spatiotemporal cube student network level in the student network, thereby generating a sample set of each target abnormal event in the original video of known abnormal events.
[0009] Preferably, step S3 specifically includes: Based on the Teacher-Student deep learning network architecture, the set of spatiotemporal cubes of each target in the original video of the known anomalous event is used as input, and the sequence of visual anomalous feature vectors of each spatiotemporal cube of the target in the original video of the known anomalous event is used as output. Based on the set of spatiotemporal cubes of each target in the original video of the known abnormal event and the motion cue feature vectors of each target in the original video frame, the motion feature vector sequence of each spatiotemporal cube of the target in the YOLOv7 backbone network CSPDarknet is generated. Based on a sliding window, the visual anomaly feature vector sequence of each target spatiotemporal cube in the original video of the known anomaly event is concatenated with the motion feature vector sequence of each target spatiotemporal cube in the original video of the known anomaly event and substituted into the linear projection layer according to a unit time, so as to obtain the fused multimodal anomaly feature sequence of each target spatiotemporal cube in the original video of the known anomaly event.
[0010] Preferably, step S3 further includes: Based on the fused multimodal anomaly feature sequence of each target spatiotemporal cube in the original video of the known anomaly event, the multimodal temporal Transformer architecture is substituted into it. According to the self-attention mechanism, the correlation strength between the fused multimodal anomaly features at any two unit time points in the fused multimodal anomaly feature sequence is calculated to obtain the fused multimodal anomaly context-aware feature sequence of each target spatiotemporal cube in the original video of the known anomaly event. Based on the fused multimodal anomaly context-aware feature sequence of each target spatiotemporal cube in the original video of known anomaly events and the big data set of original videos of known anomaly events, the spatial distance between the fused multimodal anomaly context-aware feature of each target spatiotemporal cube and the corresponding anomaly event feature in the big data set of original videos of known anomaly events is calculated using the Euclidean distance formula, and then normalized to assign an anomaly score in the big data set of original videos of known anomaly events per unit time frame. Based on the Gaussian distribution, the mean and standard deviation of the abnormal score distribution in the original video big data of known abnormal events per unit time frame are verified. The Sigma rule threshold function is established to obtain the adaptive abnormal discrimination threshold in the original video big data of known abnormal events per unit time frame. Based on the anomaly adaptive discrimination threshold in the original video big data of known abnormal events per unit time frame, determine whether the anomaly score in the original video big data of known abnormal events per unit time frame exceeds the threshold. If yes, determine that the current target in the original video big data of known abnormal events is in an abnormal state. If no, determine that the current target in the original video big data of known abnormal events is not in an abnormal state.
[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a video anomaly detection scheme. Through a three-tiered architecture—target localization and spatiotemporal modeling, unsupervised anomaly perception distillation, and multimodal temporal fusion decision-making—it significantly improves the accuracy and practicality of video anomaly detection. First, a refined behavioral modeling is achieved using a target spatiotemporal cube constructed based on YOLOv7 and optical flow analysis. Second, a multi-level feature distillation mechanism using a teacher-student network effectively captures anomaly pattern features without anomaly labeling. Finally, by fusing visual and motion features using a Transformer architecture and combining it with a dynamic threshold decision-making mechanism, the system possesses long-term temporal analysis capabilities and scene adaptability. This solves the problems of high false alarm rates, reliance on manual labeling, and difficulty in capturing temporal anomalies in traditional methods in complex scenarios, providing automated solutions for fields such as intelligent security and industrial inspection. Attached Figure Description
[0012] Figure 1 This is a flowchart of a video anomaly event detection method. Detailed Implementation
[0013] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.
[0014] Reference Figure 1 As shown, a video anomaly event detection method includes: S1. Based on the big data set of original videos of known abnormal events, the pre-trained target detector locates and marks the confidence of targets in the big data frames of original videos of known abnormal events, extracts the target motion cue feature vector, and establishes spatiotemporal cubes of each target in the original videos of known abnormal events. Step S1 specifically includes: The backbone network CSPDarknet based on YOLOv7 is pre-trained using original videos of known anomalous events and the COCO dataset. It utilizes the bounding boxes and class labels of targets in large data frames of original videos of known anomalous events to adjust the weights of the backbone network relative to each specified scene for the single-stage target detector at the anchor point, thus obtaining the single-stage target detector with anchor point. Based on the single-stage target detector of the anchor point, the original video frames of the original video big data set of known abnormal events are input one by one to generate the bounding box coordinates and category labels of the original video frames. Substitute them into the sigmoid function to obtain the matching degree between the bounding box of the original video frame and the bounding box of the real object. Based on the bounding box coordinates and class labels of the original video frame, calculate the area of the intersection region of the predicted box and the ground truth box and the area of their union region to determine the probability of object presence at the bounding box coordinates of the original video frame. Based on the matching degree between the bounding box of the original video frame and the bounding box of the real object, and the probability of the object's existence at the bounding box coordinates of the original video frame, the confidence of the bounding box coordinates and class label of the original video frame is calculated, and a sequence of the bounding box coordinates and class label of the original video frame is constructed. Step S1 also includes: Based on a large dataset of original videos containing known anomalous events, and following the dense optical flow algorithm, the mean and standard deviation of the optical flow motion vector between each adjacent frame in the sequence of bounding box coordinates and category labels of the original video frames are marked. The optical flow amplitude of each pixel in the bounding box of the original video frame is calculated. The bounding box of the original video frame is divided into several intervals according to the magnitude of the amplitude, and the histogram of the optical flow amplitude of the bounding box of the original video frame is obtained. The mean and standard deviation of the optical flow motion vector between each adjacent frame in the bounding box of the original video frame are concatenated with the histogram of the optical flow amplitude in the bounding box of the original video frame to obtain the motion cue feature vector of each target in the original video frame. Based on the multi-target tracking algorithm, the motion cue feature vectors of the same target in the original video frame are associated to generate the trajectory vector of the target in the original video frame. The bounding box and category label of each target in the original video big data set of known abnormal events are extracted to obtain the three-dimensional data block of the bounding box and category label of each target in the original video big data set of known abnormal events. Based on the bounding box and category label 3D data block of each target in the original video big data set of known abnormal events, the trajectory vector image regions of the targets in all the captured original video frames are unified to the same spatial size, resulting in a spatiotemporal cube set of each target in the original video of known abnormal events.
[0015] When using it, please refer to the steps outlined above: As a further step, a spatiotemporal data structure centered on independent targets is constructed by fusing target appearance features and motion cues. Using a YOLOv7 detector based on CSPDarknet, precise target localization and classification are achieved through domain-adaptive fine-tuning. The Sigmoid function is then used to process bounding box prediction, and intersection-over-union (IoU) calculation ensures detection accuracy. Subsequently, the Farneback dense optical flow algorithm is employed to extract motion features. By statistically analyzing the mean, standard deviation, and amplitude histogram of the optical flow vectors, feature vectors representing target motion patterns are formed. Finally, a multi-target tracking algorithm is used to correlate target trajectories, normalizing the spatiotemporal slices of targets into standard-sized spatiotemporal cubes. This forms a complete processing chain from raw video to structured spatiotemporal data units, providing a high-quality, multimodal input foundation for subsequent anomaly detection distillation.
[0016] S2. Based on the big data set of original videos of known abnormal events, a teacher-student network architecture is pre-trained to perform anomaly perception distillation on targets in the spatiotemporal cube of each target in the original videos of known abnormal events, extract the action feature vector of the target's abnormal motion behavior, and establish a sample set of abnormal events of each target in the original videos of known abnormal events. Step S2 specifically includes: Based on Bayesian prior distribution, the original video big data set of known abnormal events is filtered and divided to establish a reference knowledge base of abnormal behavior action feature vectors of known abnormal events in each specified scenario. Based on the Teacher-Student deep learning network architecture, and following the ResNet-50 deep residual network, a teacher network is established. The abnormal behavior action feature vectors of known abnormal events in various specified scenarios are used as a reference to the knowledge base to pre-train the teacher network. Based on the spatiotemporal cube sets of each target in the original video of known anomalous events, the spatiotemporal cube sets of each target are sorted and filtered according to the confidence of the bounding box coordinates and category labels of the original video frames to construct a high-confidence sample dataset of each target anomalous event in the original video of known anomalous events. Based on the Teacher-Student deep learning network architecture, and using the MobileNetV2 lightweight neural network, a student network is established. The high-confidence sample dataset of each target abnormal event in the original video of known abnormal events is used as the input to the teacher network and the student network. The multi-level abnormal features output by the teacher network and the corresponding level of abnormal features output by the student network are obtained.
[0017] Step S2 also includes: For the abnormal features corresponding to the student network output level, an adaptive pooling layer is used for convolution processing; Based on the multi-level anomaly features output by the teacher network and the corresponding level anomaly features output by the student network from the set of spatiotemporal cubes of each target in the original video of the known anomaly event, the anomaly feature vectors of the teacher network level and the anomaly feature vectors of the student network level of each target spatiotemporal cube in the original video of the known anomaly event are constructed by the difference splicing method. Based on the multi-level perceptual distillation loss function, the sum of feature differences at each level between the abnormal feature vectors of each target spatiotemporal cube teacher network level and the abnormal feature vectors of each target spatiotemporal cube student network level in the original video of known abnormal events is calculated. Then, a weighted regression is performed on the loss of each level to update the intermediate layer weights of the abnormal feature vectors of each target spatiotemporal cube student network level in the student network, thereby generating a sample set of each target abnormal event in the original video of known abnormal events.
[0018] When using it, please refer to the steps outlined above: As a further development, an unsupervised anomaly detection mechanism was constructed based on a knowledge distillation framework. An anomalous behavior feature reference knowledge base was established through a Bayesian prior distribution, providing the model with scene-specific prior knowledge. ResNet-50 was used as the teacher network, leveraging its powerful feature extraction capabilities to learn the feature representation of normal patterns. Simultaneously, MobileNetV2 was used as a lightweight student network, with high-confidence samples used to ensure training data quality. The advantage lies in multi-level feature difference learning. Adaptive pooling layers align the feature spaces of the teacher and student networks, and a hierarchical anomalous feature vector is constructed using a difference concatenation method. A weighted multi-level perceptual distillation loss function is used to optimize the student network, allowing it to expose differences from normal patterns while imitating the teacher network. These difference features constitute the anomalous event sample set. The degree of difference between the teacher and student networks in the feature space is used as the anomaly criterion, enabling rapid annotation of anomalous features automatically distilled from normal patterns, providing a novel technical path for video anomaly detection.
[0019] S3. Based on the multimodal temporal Transformer architecture, the teacher-student network architecture is used as the visual feature encoder and the target detector is used as the motion feature encoder. Feature fusion is performed on the spatiotemporal cubes of each target and the sample set of each target abnormal event in the original video of known abnormal events. The abnormal score under the unit time frame in the big data of the original video of known abnormal events is verified, and it is determined whether each target in the big data of the original video of known abnormal events is in an abnormal state. Step S3 specifically includes: Based on the Teacher-Student deep learning network architecture, the set of spatiotemporal cubes of each target in the original video of the known anomalous event is used as input, and the sequence of visual anomalous feature vectors of each spatiotemporal cube of the target in the original video of the known anomalous event is used as output. Based on the set of spatiotemporal cubes of each target in the original video of the known abnormal event and the motion cue feature vectors of each target in the original video frame, the motion feature vector sequence of each spatiotemporal cube of the target in the YOLOv7 backbone network CSPDarknet is generated. Based on the sliding window, the visual anomaly feature vector sequence of each target spatiotemporal cube in the original video of the known anomaly event is concatenated with the motion feature vector sequence of each target spatiotemporal cube in the original video of the known anomaly event according to the unit time and then substituted into the linear projection layer to obtain the fused multimodal anomaly feature sequence of each target spatiotemporal cube in the original video of the known anomaly event. Step S3 also includes: Based on the fused multimodal anomaly feature sequence of each target spatiotemporal cube in the original video of the known anomaly event, the multimodal temporal Transformer architecture is substituted into it. According to the self-attention mechanism, the correlation strength between the fused multimodal anomaly features at any two unit time points in the fused multimodal anomaly feature sequence is calculated to obtain the fused multimodal anomaly context-aware feature sequence of each target spatiotemporal cube in the original video of the known anomaly event. Based on the fused multimodal anomaly context-aware feature sequence of each target spatiotemporal cube in the original video of known anomaly events and the big data set of original videos of known anomaly events, the spatial distance between the fused multimodal anomaly context-aware feature of each target spatiotemporal cube and the corresponding anomaly event feature in the big data set of original videos of known anomaly events is calculated using the Euclidean distance formula, and then normalized to assign an anomaly score in the big data set of original videos of known anomaly events per unit time frame. Based on the Gaussian distribution, the mean and standard deviation of the abnormal score distribution in the original video big data of known abnormal events per unit time frame are verified. The Sigma rule threshold function is established to obtain the adaptive abnormal discrimination threshold in the original video big data of known abnormal events per unit time frame. Based on the anomaly adaptive discrimination threshold in the original video big data of known abnormal events per unit time frame, determine whether the anomaly score in the original video big data of known abnormal events per unit time frame exceeds the threshold. If yes, determine that the current target in the original video big data of known abnormal events is in an abnormal state. If no, determine that the current target in the original video big data of known abnormal events is not in an abnormal state.
[0020] When using it, please refer to the steps outlined above: As a further development, based on a multimodal temporal fusion decision framework, deep fusion and temporal modeling of visual and motion features are achieved through the Transformer architecture. The visual anomaly feature vector generated by S2 is concatenated and linearly projected with the motion feature vector extracted by S1 to form a unified multimodal feature sequence. The advantage lies in utilizing the self-attention mechanism of Transformer to calculate the correlation strength between features at any two time points in the sequence, capturing long-distance temporal dependencies, generating feature representations rich in contextual information, and using Euclidean distance to calculate the deviation of the current feature from the normal pattern prototype in the feature space. The statistical characteristics of anomaly scores are modeled through Gaussian distribution, and a dynamic adaptive threshold is established using the Sigma rule to achieve robust anomaly state determination. This results in a complete anomaly detection system from multi-source feature fusion, context awareness to temporal decision-making, which significantly improves the accuracy and robustness of anomaly event detection in complex scenarios.
[0021] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.
Claims
1. A method for detecting abnormal events in video, characterized in that, include: S1. Based on the big data set of original videos of known abnormal events, the pre-trained target detector locates and marks the confidence of targets in the big data frames of original videos of known abnormal events, extracts the target motion cue feature vector, and establishes spatiotemporal cubes of each target in the original videos of known abnormal events. S2. Based on the big data set of original videos of known abnormal events, a teacher-student network architecture is pre-trained to perform anomaly perception distillation on targets in the spatiotemporal cube of each target in the original videos of known abnormal events, extract the action feature vector of the target's abnormal motion behavior, and establish a sample set of abnormal events of each target in the original videos of known abnormal events. S3. Based on the multimodal temporal Transformer architecture, the teacher-student network architecture is used as the visual feature encoder and the target detector is used as the motion feature encoder. Feature fusion is performed on the spatiotemporal cubes of each target and the sample sets of each target abnormal event in the original video of known abnormal events. The abnormal scores under the unit time frame in the big data of the original video of known abnormal events are verified, and it is determined whether each target in the big data of the original video of known abnormal events is in an abnormal state.
2. The video abnormal event detection method according to claim 1, characterized in that, Step S1 specifically includes: The backbone network CSPDarknet based on YOLOv7 is pre-trained using original videos of known anomalous events and the COCO dataset. It utilizes the bounding boxes and class labels of targets in large data frames of original videos of known anomalous events to adjust the weights of the backbone network relative to each specified scene for the single-stage target detector at the anchor point, thus obtaining the single-stage target detector with anchor point. Based on the single-stage target detector of the anchor point, the original video frames of the original video big data set of known abnormal events are input one by one to generate the bounding box coordinates and category labels of the original video frames. Substitute them into the sigmoid function to obtain the matching degree between the bounding box of the original video frame and the bounding box of the real object. Based on the bounding box coordinates and class labels of the original video frame, calculate the area of the intersection region of the predicted box and the ground truth box and the area of their union region to determine the probability of object presence at the bounding box coordinates of the original video frame. Based on the matching degree between the bounding box of the original video frame and the bounding box of the real object, and the probability of object existence at the bounding box coordinates of the original video frame, the confidence of the bounding box coordinates and class label of the original video frame is calculated, and a sequence of bounding box coordinates and class label of the original video frame is constructed.
3. The video anomaly event detection method according to claim 2, characterized in that, Step S1 also includes: Based on a large dataset of original videos containing known anomalous events, and following the dense optical flow algorithm, the mean and standard deviation of the optical flow motion vector between each adjacent frame in the sequence of bounding box coordinates and category labels of the original video frames are marked. The optical flow amplitude of each pixel in the bounding box of the original video frame is calculated. The bounding box of the original video frame is divided into several intervals according to the magnitude of the amplitude, and the histogram of the optical flow amplitude of the bounding box of the original video frame is obtained. The mean and standard deviation of the optical flow motion vector between each adjacent frame in the bounding box of the original video frame are concatenated with the histogram of the optical flow amplitude in the bounding box of the original video frame to obtain the motion cue feature vector of each target in the original video frame. Based on the multi-target tracking algorithm, the motion cue feature vectors of the same target in the original video frame are associated to generate the trajectory vector of the target in the original video frame. The bounding box and category label of each target in the original video big data set of known abnormal events are extracted to obtain the three-dimensional data block of the bounding box and category label of each target in the original video big data set of known abnormal events. Based on the bounding box and category label 3D data block of each target in the original video big data set of known abnormal events, the trajectory vector image regions of the targets in all the captured original video frames are unified to the same spatial size, resulting in a spatiotemporal cube set of each target in the original video of known abnormal events.
4. The video abnormal event detection method according to claim 3, characterized in that, Step S2 specifically includes: Based on Bayesian prior distribution, the original video big data set of known abnormal events is filtered and divided to establish a reference knowledge base of abnormal behavior action feature vectors of known abnormal events in each specified scenario. Based on the Teacher-Student deep learning network architecture, and following the ResNet-50 deep residual network, a teacher network is established. The abnormal behavior action feature vectors of known abnormal events in various specified scenarios are used as a reference to the knowledge base to pre-train the teacher network. Based on the spatiotemporal cube sets of each target in the original video of known anomalous events, the spatiotemporal cube sets of each target are sorted and filtered according to the confidence of the bounding box coordinates and category labels of the original video frames to construct a high-confidence sample dataset of each target anomalous event in the original video of known anomalous events. Based on the Teacher-Student deep learning network architecture, and using the MobileNetV2 lightweight neural network, a student network is established. The high-confidence sample dataset of each target abnormal event in the original video of known abnormal events is used as the input to the teacher network and the student network. The multi-level abnormal features output by the teacher network and the corresponding level of abnormal features output by the student network are obtained.
5. The video anomaly event detection method according to claim 4, characterized in that, Step S2 also includes: For the abnormal features corresponding to the student network output level, an adaptive pooling layer is used for convolution processing; Based on the multi-level anomaly features output by the teacher network and the corresponding level anomaly features output by the student network from the set of spatiotemporal cubes of each target in the original video of the known anomaly event, the anomaly feature vectors of the teacher network level and the anomaly feature vectors of the student network level of each target spatiotemporal cube in the original video of the known anomaly event are constructed by the difference splicing method. Based on the multi-level perceptual distillation loss function, the sum of feature differences at each level between the abnormal feature vectors of each target spatiotemporal cube teacher network level and the abnormal feature vectors of each target spatiotemporal cube student network level in the original video of known abnormal events is calculated. Then, a weighted regression is performed on the loss of each level to update the intermediate layer weights of the abnormal feature vectors of each target spatiotemporal cube student network level in the student network, thereby generating a sample set of each target abnormal event in the original video of known abnormal events.
6. The video anomaly event detection method according to claim 5, characterized in that, Step S3 specifically includes: Based on the Teacher-Student deep learning network architecture, the set of spatiotemporal cubes of each target in the original video of the known anomalous event is used as input, and the sequence of visual anomalous feature vectors of each spatiotemporal cube of the target in the original video of the known anomalous event is used as output. Based on the set of spatiotemporal cubes of each target in the original video of the known abnormal event and the motion cue feature vectors of each target in the original video frame, the motion feature vector sequence of each spatiotemporal cube of the target in the YOLOv7 backbone network CSPDarknet is generated. Based on a sliding window, the visual anomaly feature vector sequence of each target spatiotemporal cube in the original video of the known anomaly event is concatenated with the motion feature vector sequence of each target spatiotemporal cube in the original video of the known anomaly event and substituted into the linear projection layer according to a unit time, so as to obtain the fused multimodal anomaly feature sequence of each target spatiotemporal cube in the original video of the known anomaly event.
7. The video abnormal event detection method according to claim 6, characterized in that, Step S3 also includes: Based on the fused multimodal anomaly feature sequence of each target spatiotemporal cube in the original video of the known anomaly event, the multimodal temporal Transformer architecture is substituted into it. According to the self-attention mechanism, the correlation strength between the fused multimodal anomaly features at any two unit time points in the fused multimodal anomaly feature sequence is calculated to obtain the fused multimodal anomaly context-aware feature sequence of each target spatiotemporal cube in the original video of the known anomaly event. Based on the fused multimodal anomaly context-aware feature sequence of each target spatiotemporal cube in the original video of known anomaly events and the big data set of original videos of known anomaly events, the spatial distance between the fused multimodal anomaly context-aware feature of each target spatiotemporal cube and the corresponding anomaly event feature in the big data set of original videos of known anomaly events is calculated using the Euclidean distance formula, and then normalized to assign an anomaly score in the big data set of original videos of known anomaly events per unit time frame. Based on the Gaussian distribution, the mean and standard deviation of the abnormal score distribution in the original video big data of known abnormal events per unit time frame are verified. The Sigma rule threshold function is established to obtain the adaptive abnormal discrimination threshold in the original video big data of known abnormal events per unit time frame. Based on the anomaly adaptive discrimination threshold in the original video big data of known abnormal events per unit time frame, determine whether the anomaly score in the original video big data of known abnormal events per unit time frame exceeds the threshold. If yes, determine that the current target in the original video big data of known abnormal events is in an abnormal state. If no, determine that the current target in the original video big data of known abnormal events is not in an abnormal state.