Video abnormal behavior detection method based on target tracking fusion model
By improving the YOLOv8 object detection model and a custom deep neural network, and combining the object tracking algorithm and the ternary loss function, the problems of accuracy and computational efficiency of existing video anomaly detection methods in complex scenarios are solved, and efficient and detailed anomaly behavior detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-24
- Publication Date
- 2026-04-03
AI Technical Summary
Existing video anomaly detection methods struggle to accurately detect abnormal behavior in complex scenarios, fail to effectively mitigate static background information, incur high computational costs, and cannot provide detailed information about anomaly occurrences.
A YOLOv8 target detection model based on a bidirectional feature pyramid network is used for frame-by-frame detection. An abnormal behavior classification is performed by combining a target tracking algorithm and a custom deep neural network. Kalman filtering and IOU algorithm are used to optimize target tracking, and a ternary loss function is used to replace the cross-entropy loss function for classification.
It improves detection accuracy in complex scenarios, reduces the processing of static background information, lowers computational overhead, and provides detailed information on anomalies, making it suitable for real-time applications with limited resources.
Smart Images
Figure CN121789103A_ABST
Abstract
Description
Technical fields:
[0001] This invention relates to the field of video anomaly detection, and more particularly to a method for detecting abnormal video behavior based on a target tracking fusion model. Background technology:
[0002] In recent years, the urbanization process has continued to accelerate, and the population agglomeration effect has become increasingly significant, which has undoubtedly brought numerous difficulties to public safety assurance work. To properly address these challenges, it is of great research significance to utilize artificial intelligence technology to accurately analyze, identify, spatially locate, and rapidly search for target objects, and to use abnormal behavior detection systems to effectively identify and judge abnormal behavior. This is crucial for the timely development of scientific and reasonable response strategies and for rapid emergency response.
[0003] Currently, the definition of anomaly may vary in different environments and scenarios. There are three main types of common video anomaly detection methods: trajectory-based anomaly detection, global pattern-based anomaly detection, and representation learning-based anomaly detection.
[0004] The above three methods are commonly used in the field of video abnormal behavior detection, but they still have problems in real-world applications, mainly in the following aspects:
[0005] The models are relatively simple and unable to detect anomalous behavior in complex scenarios. Currently used models generally have simple structures, and their operation is mostly based on basic rules or features for anomaly detection. However, in real-world applications, anomalous behavior is often hidden within complex contexts, which may encompass numerous variables, various interactions, and dynamic changes. Due to the limitations of the model's structure and algorithm, it is difficult to effectively handle these complex elements, leading to an inability to accurately identify anomalous behavior in complex situations and resulting in the omission of important anomalous behavior information.
[0006] The current model lacks effective mechanisms to weaken or filter out redundant information from a large proportion of static background. In practical applications, video or image data often contains a significant amount of static background information, which is largely redundant for anomaly detection. This forces the model to expend considerable computational resources during training to process this unimportant background information, thus reducing its efficiency and accuracy. If this background information could be effectively weakened or filtered, the model could focus more on detecting genuine anomalies, thereby improving its performance and accuracy.
[0007] Complex models incur high computational costs. While complex models may exhibit higher accuracy in certain situations, this is often accompanied by significantly higher computational costs. Specifically, during model training, the demand for data and computing resources is greater; and during the inference phase, processing time is also longer. For applications with stringent real-time requirements, computationally expensive models are difficult to implement in practice. Furthermore, because high computational costs lead to higher expenses, such models are also unsuitable for resource-constrained environments.
[0008] The current model lacks sufficient information to pinpoint the specific aspects of the anomaly: Detecting abnormal behavior in videos requires more than simply identifying the anomaly; providing detailed information about the occurrence of the anomaly is crucial. This information can assist relevant personnel in assessing the abnormal behavior and implementing more timely and efficient response strategies. However, current models may only provide limited information such as the time and location of the anomaly, making it difficult to conduct a more in-depth and thorough analysis and description of the specific aspects of the anomaly. Summary of the Invention:
[0009] The purpose of this invention is to overcome the shortcomings of existing methods and propose a video abnormal behavior detection method based on deep learning, particularly a video abnormal behavior detection method based on a target tracking fusion model, in order to solve the problem of the performance of traditional methods in video abnormal behavior detection.
[0010] A video anomaly behavior detection method based on a target tracking fusion model, characterized by the following steps:
[0011] Step 1: The YOLOv8 target detection model, which is improved based on the bidirectional feature pyramid network, is used to detect targets in the video frame by frame.
[0012] Step 2: The target tracking algorithm tracks the detected target frame by frame to prevent target frame loss;
[0013] Step 3: Use a custom deep neural network abnormal behavior classification model to classify the behavior of the target in each frame and detect abnormal behavior.
[0014] Step 1 includes the following steps:
[0015] Step 1.1: Use open-source video processing tools to process the source video data {X} s Y s Perform frame segmentation to generate image format data {X}. g Y s}, and retain the data containing information for each frame:
[0016] {X g Y s}=G({X s Y s})
[0017] Where G represents the video processing process.
[0018] Step 1.2: Use a feature extraction backbone network with 10 convolutional layers to extract features from each frame of the video, generate a gradient stream with data information, and output a feature map:
[0019]
[0020] Where [] represents the association operation, W is the weight of the input channel, and f(x) is a 3×3 convolution operation. This represents the dot product operation.
[0021] Step 1.3: A weighted fusion bidirectional feature pyramid network is used to perform hierarchical extraction of the feature maps output by the feature extraction backbone network. First, feature information is fused from top to bottom, and then feature information interacts from bottom to top. Skip connections are used to aggregate feature information between input and output, generating feature maps with different levels of information. Finally, the features are quickly normalized and weighted fused.
[0022]
[0023] Where I is the input tensor, O is the output tensor, ω is the updatable weight, and ε is a constant, defaulting to 0.0001.
[0024] Step 1.4: Use the decoupled output layer to classify and locate the features output by the weighted fusion bidirectional feature pyramid network, and output the category and location information of the target.
[0025] Step 2 includes the following steps:
[0026] Step 2.1: Input the results with target detection boxes output by the YOLOv8 target detection model improved by the bidirectional feature pyramid network into the target tracking algorithm, generate high and low confidence targets according to the confidence level, and use the Kalman filter algorithm to generate predicted target detection boxes;
[0027] Step 2.2: Considering the potential discrepancy between the predicted target detection boxes generated by the Kalman filter algorithm and the high-confidence detection boxes, the Intersection over Union (IOU) algorithm is used to calculate the intersection-union ratio (IoU) between trajectories, and then a binary matching algorithm is used to obtain the optimal result.
[0028]
[0029] Where IOU stands for Intersection over Union, A is the target bounding box, and B is the predicted bounding box;
[0030] Step 2.3: To prevent occlusion in the target video from affecting the predicted trajectory matching, low-confidence detection boxes are associated with the predicted trajectory boxes that fail to match, and this process is continued. If the predicted trajectory continues to fail to match, the predicted trajectory boxes are discarded. If a low-confidence detection box does not match the target, it is deleted.
[0031] Step 2.4: Associate and save the matched target bounding boxes and predicted trajectory bounding boxes, and then output them.
[0032] Step 3 includes the following steps:
[0033] Step 3.1: Since the monitoring equipment is far from the ground, a wider field of view is needed to detect the scene. Therefore, a custom network with a wide model width and multiple convolutional kernels is built to classify abnormal behavior.
[0034] Step 3.2: Since abnormal behavior occurs very infrequently in reality, making it impossible to collect a large amount of data, ternary loss is used to classify the limited amount of abnormal behavior data. For each batch of data, N sample boxes are selected, and one of the detection boxes is chosen as the anchor point. Corresponding positive and negative samples are selected to form a triplet. The L2 distance between the anchor point and the positive and negative examples is calculated, and the model loss is also calculated.
[0035]
[0036] Where x1, y1, x2, and y2 are the coordinates of the calculation points.
[0037] L=max(d(a,p)-d(a,n)+margin,0)
[0038] Where a is the anchor point, p is the positive sample, n is the negative sample, d is the L2 distance, and margin is the allowance.
[0039] The beneficial effects of this invention are as follows: Considering the limitations of current video abnormal behavior detection methods in practical applications, and the problems existing in the practical application of these methods, this invention designs a video abnormal behavior detection method based on a target tracking fusion model. This method first trains a YOLOv8 target detection model based on a bidirectional feature pyramid network, using preprocessed video data to detect abnormal behaviors in the video. The detected results are then input into a target detection and tracking method to preserve the bounding boxes and prevent target loss. Simultaneously, a custom classification network model capable of accurately classifying abnormal behaviors is designed, and a ternary loss function replaces the model's built-in cross-entropy loss function. This end-to-end process, using contrastive learning to classify target behaviors, improves the model's detection accuracy in scenarios with few samples and class imbalance. Attached image description:
[0040] Figure 1 This is a flowchart of a video abnormal behavior detection method based on a target tracking fusion model.
[0041] Figure 2 This is an overall structural diagram of a video abnormal behavior detection method based on a target tracking fusion model.
[0042] Figure 3 This is a structural diagram of the YOLOv8 object detection model, which is an improvement on the bidirectional feature pyramid network.
[0043] Figure 4 This is a diagram of the target tracking algorithm model structure.
[0044] Figure 5 This is a diagram of the custom abnormal behavior classification model structure.
[0045] Figure 6 This is a graph showing the results of abnormal behavior detection. Detailed implementation method:
[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0047] Figure 1 This is a schematic diagram illustrating the specific process of implementing the present invention. Figure 2 The overall model structure of the present invention is shown. Figure 3 This is the target detection module structure of the present invention. Figure 4 This is the target tracking algorithm model structure in this invention. Figure 5 This is the custom abnormal behavior classification model structure in this invention, such as... Figure 1 , 2 As shown in Figures 3, 4, and 5, the method includes:
[0048] Step 1.1: Use open-source video processing tools to process the source video data {X} s Y s Perform frame segmentation to generate image format data {X}. g Y s}, and retain the data containing information for each frame:
[0049] {X g Y s}=G({X s Y s})
[0050] Where G represents the video processing process.
[0051] Step 1.2: Use a feature extraction backbone network with 10 convolutional layers to extract features from each frame of the video, generate a gradient stream with data information, and output a feature map:
[0052]
[0053] Where [] represents the association operation, W is the weight of the input channel, and f(x) is a 3×3 convolution operation. This represents the dot product operation.
[0054] Step 1.3: A weighted fusion bidirectional feature pyramid network is used to perform hierarchical extraction of the feature maps output by the feature extraction backbone network. First, feature information is fused from top to bottom, and then feature information interacts from bottom to top. Skip connections are used to aggregate feature information between input and output, generating feature maps with different levels of information. Finally, the features are quickly normalized and weighted fused.
[0055]
[0056] Where I is the input tensor, O is the output tensor, ω is the updatable weight, and ε is a constant, defaulting to 0.0001.
[0057] Step 1.4: Use the decoupled output layer to classify and locate the features output by the weighted fusion bidirectional feature pyramid network, and output the category and location information of the target.
[0058] Step 2.1: Input the results with target detection boxes output by the YOLOv8 target detection model improved by the bidirectional feature pyramid network into the target tracking algorithm, generate high and low confidence targets according to the confidence level, and use the Kalman filter algorithm to generate predicted target detection boxes;
[0059] Step 2.2: Considering the potential discrepancy between the predicted target detection boxes generated by the Kalman filter algorithm and the high-confidence detection boxes, the Intersection over Union (IOU) algorithm is used to calculate the intersection-union ratio (IoU) between trajectories, and then a binary matching algorithm is used to obtain the optimal result.
[0060]
[0061] Where IOU stands for Intersection over Union, A is the target bounding box, and B is the predicted bounding box;
[0062] Step 2.3: To prevent occlusion in the target video from affecting the predicted trajectory matching, low-confidence detection boxes are associated with the predicted trajectory boxes that fail to match, and this process is continued. If the predicted trajectory continues to fail to match, the predicted trajectory boxes are discarded. If a low-confidence detection box does not match the target, it is deleted.
[0063] Step 2.4: Associate and save the matched target bounding boxes and predicted trajectory bounding boxes, and then output them.
[0064] Step 3.1: Since the monitoring equipment is far from the ground, a wider field of view is needed to detect the scene. Therefore, a custom network with a wide model width and multiple convolutional kernels is built to classify abnormal behavior.
[0065] Step 3.2: Since abnormal behavior occurs very infrequently in reality, making it impossible to collect a large amount of data, ternary loss is used to classify the limited amount of abnormal behavior data. For each batch of data, N sample boxes are selected, and one of the detection boxes is chosen as the anchor point. Corresponding positive and negative samples are selected to form a triplet. The L2 distance between the anchor point and the positive and negative examples is calculated, and the model loss is also calculated.
[0066]
[0067] Where x1, y1, x2, and y2 are the coordinates of the calculation points.
[0068] L=max(d(a,p)-d(a,n)+margin,0)
[0069] Where a is the anchor point, p is the positive sample, n is the negative sample, d is the L2 distance, and margin is the allowance.
[0070] The effects of the present invention will be further explained and compared below with simulation experiments.
[0071] Simulation experimental conditions:
[0072] The hardware of the simulation experimental platform of this invention is: CPU: Intel(R) Core(TM) i9-1390K, GPU: NVIDIA GEFORCE RTX 4090 with 24G of video memory, and software platform: Ubuntu 22.04 system.
[0073] Comparative analysis of simulation experiment results:
[0074] To evaluate the performance of the YOLOv8 object detection model improved based on the bidirectional feature pyramid network, the model was evaluated using the general evaluation metric mAP, which measures object detection accuracy, and the metric FPS, which measures model inference speed. The results are shown in Table 1.
[0075] Table 1 shows the comparative experimental results of each model on the video abnormal behavior dataset.
[0076]
[0077] Meanwhile, in order to verify the overall detection accuracy of a video anomaly detection method based on a target tracking fusion model, as well as the extent and effectiveness of the improvement in detection accuracy by each improved model part in the method, an ablation experiment was conducted. The experiment shows that the bidirectional feature pyramid network, target tracking network model, ternary loss function and custom anomaly classification network proposed in this invention all help to improve the robustness of the model.
[0078] Table 2 Comparison Results of Ablation Experiments
[0079]
[0080] It should be understood that any parts not described in detail in this specification belong to the prior art.
[0081] The above description, in conjunction with the accompanying drawings, is merely a specific embodiment and process of the present invention. However, the scope of protection of the present invention is not limited thereto. Any person skilled in the art should understand that this is only an illustrative example, and various changes and substitutions can be made to this embodiment without departing from the essence of the present invention. The scope of the present invention is defined only by the appended claims.
[0082] The embodiments described with reference to the accompanying drawings are exemplary and are only used to explain the invention. They should not be construed as limiting the invention. The specific scope of the embodiments of the invention is not limited thereto. On the contrary, all embodiments of the invention include all variations and modifications that fall within the spirit and scope of the appended claims.
Claims
1. A video anomaly behavior detection method based on a target tracking fusion model, characterized in that, Includes the following steps: Step 1: The YOLOv8 target detection model, which is improved based on the bidirectional feature pyramid network, is used to detect targets in the video frame by frame. Step 2: The target tracking method tracks the detected target frame by frame to prevent target frame loss; Step 3: Use a custom deep neural network abnormal behavior classification model to classify the behavior of the target in each frame and identify abnormal behaviors.
2. The video abnormal behavior recognition method based on the target detection and tracking fusion model according to claim 1, characterized in that, Step 1 includes the following steps: Step 1.1: Use open-source video processing tools to process the source video data {X} s Y s Perform frame segmentation to generate image format data {X}. g Y s }, and retain the data containing information for each frame: Step 1.2: Use a feature extraction backbone network with 10 convolutional layers to extract features from each frame of the video, generate a gradient stream with data information, and output a feature map: Step 1.3: Use a weighted fusion bidirectional feature pyramid network to extract feature maps from the feature extraction backbone network in a hierarchical manner. First, the feature information is fused from top to bottom, and then the feature information is interacted from bottom to top. At the same time, skip connections are used to aggregate the feature information between input and output, generating feature maps with different levels of information, and then the features are quickly normalized and weighted fused. Step 1.4: Use the decoupled output layer to classify and locate the features output by the weighted fusion bidirectional feature pyramid network, and output the category and location information of the target.
3. The video abnormal behavior detection method based on the target tracking fusion model according to claim 1, characterized in that, Step 2 includes the following steps: Step 2.1: Input the results with target detection boxes output by the YOLOv8 target detection model improved by the bidirectional feature pyramid network into the target tracking algorithm, generate high and low confidence targets according to the confidence level, and use the Kalman filter algorithm to generate predicted target detection boxes; Step 2.2: Considering that there may be a gap between the predicted target detection box generated by the Kalman filter algorithm and the high confidence detection box, the IOU algorithm is used to calculate the intersection-union ratio between trajectories, and then the binary matching algorithm is used to obtain the optimal result; Step 2.3: To prevent occlusion in the target video from affecting the predicted trajectory matching, low-confidence detection boxes are associated with the predicted trajectory boxes that fail to match, and this process is continued. If the predicted trajectory continues to fail to match, the predicted trajectory boxes are discarded. If a low-confidence detection box does not match the target, it is deleted. Step 2.4: Associate and save the matched target bounding boxes and predicted trajectory bounding boxes, and then output them.
4. The video abnormal behavior recognition method based on the target detection and tracking fusion model according to claim 1, characterized in that, Step 3 includes the following steps: Step 3.1: Since the monitoring equipment is far from the ground, a wider field of view is needed to detect the scene. Therefore, a custom network with a wide model width and multiple convolutional kernels is built to classify abnormal behavior. Step 3.2: Since abnormal behavior occurs very infrequently in reality, it is impossible to collect a large amount of data. Therefore, the ternary loss is used to classify the small amount of abnormal behavior data. For each batch of data, N sample boxes are selected, and one of the detection boxes is selected as the anchor point. The corresponding positive and negative samples are selected to form a triplet. The L2 distance between the anchor point and the positive and negative examples is calculated respectively, and the model loss is calculated.