An abnormal behavior detection method based on tph-yolov5

Abstract

The application discloses an abnormal behavior detection method based on a TPH-yolov5, which comprises the following steps: inputting a test image into a trained TPH-yolov5 network model to identify abnormal behaviors; wherein the abnormal behaviors include smoking, illegal intrusion, illegal photographing, illegal carrying of a computer and illegal telephone call; and the application improves the accuracy of detecting abnormal behaviors.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to an abnormal behavior detection method based on TPH-yolov5. Background Technology

[0002] With the development of computer and artificial intelligence technologies, abnormal behavior detection is becoming increasingly important in the fields of economy, management, and public safety. Dynamic analysis of crowd behavior is a highly anticipated area in video surveillance. "Abnormal events / behaviors" refer to sudden, irregular events that attract attention compared to more frequent routine events. For security reasons, how to detect abnormal crowd behavior in public places such as crowded streets, large shopping malls, theaters, airports, and train stations has attracted increasing attention. However, because abnormal behavior targets are mixed with normal targets, missed detections and false detections are prone to occur, reducing detection accuracy. Therefore, high-precision identification and further anomaly detection of abnormal behavior targets in public scenes is one of the current challenges. Summary of the Invention

[0003] In view of this, the present invention provides an abnormal behavior detection method based on TPH-yolov5 to solve the above-mentioned technical problems.

[0004] This invention discloses an abnormal behavior detection method based on TPH-YOLOv5, which includes:

[0005] The test images are input into the trained TPH-yolov5 network model to identify abnormal behaviors, including smoking, illegal intrusion, illegal photography, illegal computer use, and illegal phone calls.

[0006] Furthermore, in the TPH-YOLOv5 network model:

[0007] Based on the YOLOv5 network, CSPDarknet53 was selected as the backbone, and three transformer prediction heads were added at the end;

[0008] Using a PANet-like structure, the four transformer prediction heads at the neck output use the feature maps of the neck transformer encoder block, and the output feature maps serve as TPH prediction heads, each corresponding to the feature maps of targets of different sizes.

[0009] The head is responsible for detecting the location and category of targets using feature maps extracted from the backbone network;

[0010] CBAM injects the attention map along two independent dimensions of the feature map: channel and space. Then, it multiplies the attention by the input feature map to perform adaptive feature refinement on the input feature map.

[0011] Furthermore, it also includes:

[0012] The ms test strategy is executed on a single trained TPH-yolov5 network model, that is: the original test image is scaled by different ratios and flipped horizontally to obtain L images;

[0013] L images and the original test image are input into different trained TPH-yolov5 network models, and non-maximum suppression fusion is used to test the prediction. The accuracy of detecting abnormal behavior in the original test image is obtained by weighted bounding box fusion.

[0014] Furthermore, non-maximum suppression, soft NMS, or weighted box fusion are used to integrate multiple different trained TPH-yolov5 network models into a single box.

[0015] When using non-maximum suppression, if the intersection-union ratio of cuboids is higher than a preset threshold, they are considered to belong to the same object; for each object, the non-maximum suppression method retains only the bounding box with the highest confidence and deletes the other bounding boxes.

[0016] When using soft nonmaximum suppression, the decay function is set based on the confidence of adjacent bounding boxes based on the cross-union ratio, instead of setting their confidence scores to zero and deleting them;

[0017] When weighted box fusion is used, all boxes are merged to form the final result; the TPH-yolov5 network model will output feature maps of four different scales to detect objects of different scales.

[0018] Furthermore, the merging step of the weighted box fusion is as follows:

[0019] Step 1: Each predicted bounding box of each trained TPH-YOLOv5 network model is added to a single list B. List B is sorted in descending order of the confidence score C of each predicted bounding box, and then divided into box clusters and fused box clusters according to the range of the confidence score C. List L is used to store box clusters, and list F is used to store fused box clusters. Each position in list L contains a group of boxes or a single box, forming a cluster. Each position in list F contains only one box, which is a fused box of a box cluster in list L.

[0020] Step 2: Loop through the predicted boxes in list B and find the matching boxes in list F;

[0021] Step 3: If a matching box is found, add the matching box to the position pos corresponding to the matching box in list L and list F; recalculate the coordinates and confidence scores of the boxes in F[pos], using all the accumulated boxes in box cluster L[pos], a total of T boxes;

[0022] Step 4: Repeat steps 2 and 3 until all predicted boxes in list B have been processed. Then, rescale the confidence scores in list F by multiplying them by the number of boxes in a cluster and dividing by the number of trained TPH-yolov5 network models, N.

[0023] Furthermore, step 3 also includes:

[0024] If no matching box is found, the predicted box in list B is added to the end of lists L and F as a new entry; then the process continues to iterate through the next predicted box in list B.

[0025] Furthermore, step 4 also includes:

[0026] If the number of frames in a cluster is small, the confidence score needs to be reduced using the following formula:

[0027]

[0028] or

[0029]

[0030] Where C is the confidence score, N is the number of trained TPH-yolov5 network models, and T is the number of boxes in F[pos].

[0031] Furthermore, in step 3:

[0032] The following fusion formula is used to fuse all accumulated boxes in the box cluster L[pos]:

[0033]

[0034] Where C is the confidence score, N is the number of trained TPH-yolov5 network models, T is the number of boxes in L[pos], and (X1,2) and (Y1,2) are the coordinates of the two diagonal vertices of the corresponding boxes in F[pos].

[0035] After merging all the boxes accumulated in the box cluster L[pos], a merged box is obtained. The coordinates of the merged box are the weighted sum of the coordinates of T boxes, where the weight is the confidence score of the corresponding box.

[0036] Furthermore, it also includes:

[0037] The bounding boxes output by the TPH header are input into the model after weighted bounding box fusion for processing, and the results are visualized and analyzed for anomalies.

[0038] After visualizing the results and analyzing anomalies, the trained TPH-yolov5 network model can be optimized using an additional self-trained classifier.

[0039] The output of the trained classifier is the category of the abnormal behavior and the predicted bounding box.

[0040] Furthermore, using image patches cropped from the training set as the classification training set, and selecting ResNet18 as the classifier network, a self-trained classifier is obtained.

[0041] Due to the adoption of the above technical solutions, the present invention has the following advantages: In order to better test the model performance, this method adds multi-scale testing (ms test) and multi-model ensemble strategy during the inference process to obtain more convincing detection results; In addition, by visualizing abnormal cases, it is found that the proposed architecture has good localization ability but poor classification ability. A self-training classifier is provided, which uses image patches cropped from the training data as the classification training set, ultimately improving the accuracy of detecting abnormal behavior. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0043] Figure 1 This is a structural diagram of the CBAM according to an embodiment of the present invention;

[0044] Figure 2 This is a structural diagram of a self-trained classifier according to an embodiment of the present invention;

[0045] Figure 3 This is a structural diagram of the THP-yolov5 network model according to an embodiment of the present invention;

[0046] Figure 4 This is a flowchart illustrating an abnormal behavior detection method based on TPH-yolov5 according to an embodiment of the present invention. Detailed Implementation

[0047] The present invention will be further described in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0048] This method involves data acquisition and augmentation, multi-model integration, object detection module, MS inference module, and self-training classifier module.

[0049] Includes the following steps:

[0050] 1) Create a dataset: Create a network anomaly dataset, and divide the dataset into a training set and a test set in a 7:3 ratio;

[0051] 2) Building upon the YOLOv5 network, a Transformer predictor head is added to detect objects of different sizes. This Transformer predictor head then replaces the original predictor head to explore the predictive potential of self-attention mechanisms. A Convolutional Block Attention (CBAM) model is also integrated to find attention regions in scenes with dense objects. Several useful strategies are applied, such as data augmentation, multi-scale testing, multi-model ensemble, and the use of additional classifiers to enhance model accuracy. The TPH-YOLOv5 network model has four probe heads, used to detect small, medium, and large objects respectively. To better test the model's performance, multi-scale testing (ms testing) and multi-model ensemble strategies are added during inference to obtain more convincing detection results. Furthermore, by visualizing anomalous cases, it was found that the proposed architecture has good localization capabilities but poor classification capabilities, especially in some similar categories. To address this issue, a self-trained classifier (ResNet18) is provided, using image patches cropped from the training data as the classification training set.

[0052] 3) Input the augmented training set into the network. Select CSPDarknet53 for the backbone, add three transformer prediction heads at the ends, and select a PANet-like structure for the neck to generate feature maps. The feature maps are selected by NMS and WPF, input into the self-trained classifier, and finally generate the category and prediction box.

[0053] Specifically, see Figure 4 This method mainly includes:

[0054] 1. Data acquisition and expansion module:

[0055] Data Collection: 3000 images were obtained through a self-made dataset and website searches, of which 2700 contained abnormal behaviors such as smoking, illegal intrusion, illegal photography, unauthorized computer use, and unauthorized phone calls. 80% of the images in the dataset were selected as training samples, and 20% as test samples. The images were labeled using the LabelImg annotation software, and the annotations were saved as XML files according to the PascalVoc format used by ImageNet.

[0056] Data augmentation: The effectiveness of data augmentation lies in expanding the dataset, making the model more robust to images acquired in different environments. Photometric distortion and geometric distortion are widely used. For photometric distortion, the image's hue, saturation, and values ​​need to be adjusted. When dealing with geometric distortion, random scaling, cropping, translation, shearing, and rotation are added. In addition to the global pixel enhancement methods mentioned above, there are some more unique data augmentation methods. Some researchers have proposed methods that combine multiple images for data augmentation, namely MixUp, CutMix, and Mosaic.

[0057] MixUp randomly selects two samples from the training images and performs a random weighted sum, with the sample labels corresponding to the weighted sum. Assume the two samples (x...)... i ,y i ),(x j ,y j ),but:

[0058] Where x is the input vector.

[0059] Where y is the one-hot encoding of the label.

[0060] λ∈[0,1] is the probability value, λ: Beta(α, α), that is, λ follows a Beta distribution with parameter α.

[0061] Unlike typical image occlusion methods that use a zero-pixel "black cloth" to cover the image, CutMix uses a region from another image to cover the occluded area. Essentially, it crops a random rectangular region from one image onto another to generate a new image. Label processing is the same as mixUp, determining the new mixed label ratio based on the proportion of the two original samples in the new sample.

[0062] Mosaic is an improved version of CutMix. It stitches together four images using a mosaic technique, greatly enriching the background of the detected object. Furthermore, batch normalization calculates activation statistics for the four different images at each layer.

[0063] The TPH-YOLOv5 network model uses mixUp, Mosaic, and traditional methods to augment the data. The TPH-YOLOv5 network model can be found here. Figure 3 As shown.

[0064] 2. Multi-model ensemble method:

[0065] Deep learning neural networks are a non-linear approach. They offer greater flexibility and can scale proportionally with the amount of training data. One drawback of this flexibility is that they learn through stochastic training algorithms, meaning they are sensitive to the details of the training data and may find different weight sets each time they are trained, resulting in different predictions. This makes neural networks highly varianced. A successful approach to reducing the variance of neural network models is to train multiple models instead of a single model and combine the predictions from these models. There are three different methods for integrating boxes from different object detection models: Non-maximum suppression (NMS), soft NMS, and Weighted Box Fusion (WBF).

[0066] In the NMS method, if the overlap, intersection, and union (IoU) of cuboids exceed a certain threshold, they are considered to belong to the same object. For each object, NMS retains only the bounding box with the highest confidence and deletes the others. Therefore, the box filtering process depends on the choice of this single IoU threshold, which has a significant impact on model performance.

[0067] SoftNMS makes a subtle change to NMS, setting the decay function based on the confidence score of adjacent bounding boxes based on the IoU value, instead of setting their confidence scores to zero and deleting them.

[0068]

[0069] Where s i It is the Iou value, N t It is the set threshold.

[0070] Both NMS and SoftNMS exclude some bounding boxes, while WPF merges all boxes to form the final result. The TPH-YOLOv5 network model outputs feature maps at four different scales for detecting objects at different scales. The merging steps in WPF are as follows:

[0071] (1) Each predicted box of each trained TPH-yolov5 network model is added to a single list B. List B is sorted in descending order of the confidence C of each predicted box, and then divided into box clusters and fused box clusters according to the range of the confidence C value; wherein, list L is used to store box clusters, and list F is used to store fused box clusters; each position in list L contains a group of boxes or a single box, forming a cluster; each position in list F contains only one box, which is a fused box of a box cluster in list L;

[0072] (3) Iterate through the predicted boxes in B and try to find matching boxes in list F. A match is defined as a box that overlaps significantly with the question box (OU>THR). The optimal threshold is approached when THR=0.55.

[0073] (4) If no match is found, add the box in list B to the end of lists L and F as a new entry; continue to the next box in list B.

[0074] (5) If a match is found, add this box to the position pos corresponding to the matching box in list L and list F.

[0075] (6) Recalculate the coordinates and confidence scores of the boxes in F[pos], using all T accumulated boxes in the box family L[pos], with the following fusion formula:

[0076]

[0077] Where C is the confidence score, N is the number of trained TPH-yolov5 network models, T is the number of boxes in L[pos], and (X1,2) and (Y1,2) are the coordinates of the two diagonal vertices of the corresponding boxes in F[pos].

[0078] After merging all the boxes accumulated in the box cluster L[pos], a merged box is obtained. The coordinates of the merged box are the weighted sum of the coordinates of T boxes, where the weight is the confidence score of the corresponding box.

[0079] (7) After processing all boxes in B, rescale the confidence scores in list F: multiply by the number of boxes in a cluster, then divide by the number of trained TPH-yolov5 network models N. If the number of boxes in a cluster is small, it may mean that only a few models can predict it. To reduce the confidence scores in this case, follow these steps:

[0080]

[0081] or

[0082]

[0083] 3. Target Detection Module:

[0084] (1) backbone:

[0085] Commonly used backbones include VGG, ResNet, and CSPDarknet53, which have proven to have powerful feature extraction capabilities for classification and other problems. However, the backbone structure still needs to be fine-tuned to better suit specific tasks. This algorithm selects CSPDarknet53 as the backbone and adds three transformer prediction heads at the end.

[0086] (2) Neck:

[0087] The neck design aims to better utilize the features extracted from the backbone. It reprocesses and rationally uses the feature maps extracted from the backbone at different stages. The neck consists of several bottom-up paths and several top-down paths. This algorithm's neck uses a PANet-like structure, and the four transformer prediction heads output by the neck utilize the feature maps of the neck transformer encoder block.

[0088] (3) Head:

[0089] The head is responsible for detecting the location and category of targets using feature maps extracted from the backbone network. The feature maps output by the four transformer prediction heads in the neck serve as TPH prediction heads, each corresponding to the feature maps of targets of different sizes.

[0090] (4) CBAM:

[0091] CBAM is a simple yet effective attention module for convolutional neural networks, and its network structure is as follows: Figure 1 As shown, given an intermediate feature map in the convolutional neural network, CBAM injects the attention map along two independent dimensions: the channel and the spatial dimension. Then, the attention is multiplied by the input feature map to perform adaptive feature refinement on the input feature map.

[0092] Given an intermediate feature map F∈R C×H×W As input, the CBAM operation process is generally divided into two parts. First, the input is subjected to global max pooling and mean pooling by channel. The two pooled one-dimensional vectors are then fed into a fully connected layer for operation and added together to generate a one-dimensional channel attention M. c ∈R C×1×1 Next, the channel attention is multiplied by the input element to obtain the channel attention-adjusted feature map F′. Then, F′ is subjected to global max pooling and mean pooling in space. The two two-dimensional vectors generated by pooling are concatenated and then convolved to finally generate the two-dimensional spatial attention M. s ∈R 1×H×W Then, the spatial attention is element-wise multiplied with F′, and the CBAM attention generation process can be described as follows:

[0093]

[0094] in This indicates that corresponding elements are multiplied. Before the multiplication operation, channel attention and spatial attention need to be broadcast according to the spatial dimension and channel dimension, respectively.

[0095] 4.ms test module:

[0096] Five different TPH-YOLOv5 network models were trained based on different perspectives of model ensemble. During the inference phase, a millisecond (ms) test strategy was first executed on a single model. The implementation details of the ms test include the following three steps.

[0097] (1) Scale the test image to 1.3 times.

[0098] (2) Reduce the image size to 1x, 0.83x and 0.67x respectively.

[0099] (3) Flip the image horizontally.

[0100] Finally, six images of different scales were input into the trained TPH-YOLOv5 network model, and NMS fusion was used to test the predictions. The same MS test operation was performed on different trained TPH-YOLOv5 network models, and the last five predictions were fused using WBF to obtain the final result.

[0101] 5. Self-trained classifier module:

[0102] After training the dataset using the TPH-YOLOv5 network model, the results of anomaly analysis were visualized, leading to the conclusion that the TPH-YOLOv5 network model has good localization capabilities but poor classification capabilities. Further investigation revealed very low accuracy for certain categories. To address this issue, an additional self-trained classifier can be used. First, a training set was constructed by cropping the ground truth bounding boxes and resizing each image patch to 64×64. Then, ResNet18 was selected as the classifier network.

[0103] See Figure 2 With the help of this self-trained classifier, this method improves the AP value by approximately 0.8% to 1.0%.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An abnormal behavior detection method based on TPH-YOLOv5, characterized in that, include: The test images are input into the trained TPH-yolov5 network model to identify abnormal behaviors, including smoking, illegal intrusion, illegal photography, illegal computer use, and illegal phone calls. In the TPH-yolov5 network model: Based on the YOLOv5 network, CSPDarknet53 was selected as the backbone, and three transformer prediction heads were added at the end; Using the PANet structure, the four transformer prediction heads at the neck output use the feature maps of the neck transformer encoder block, and the output feature maps are used as TPH prediction heads, which correspond to the feature maps of targets of different sizes. The head is responsible for detecting the location and category of targets using feature maps extracted from the backbone network; CBAM injects the attention map along two independent dimensions of the feature map: channel and space. Then, it multiplies the attention by the input feature map to perform adaptive feature refinement on the input feature map. Also includes: The ms test strategy is executed on a single trained TPH-yolov5 network model, that is: the original test image is scaled by different ratios and flipped horizontally to obtain L images; L images and the original test image are input into different trained TPH-yolov5 network models, and non-maximum suppression fusion is used to test the prediction; the accuracy of detecting abnormal behavior in the original test image is obtained by weighted bounding box fusion. Multiple different trained TPH-yolov5 network models are integrated into a single bounding box using non-maximum suppression, soft NMS, or weighted box fusion. When using non-maximum suppression, if the intersection-union ratio of cuboids is higher than a preset threshold, they are considered to belong to the same object; for each object, the non-maximum suppression method retains only the bounding box with the highest confidence and deletes the other bounding boxes. When using soft nonmaximum suppression, the decay function is set based on the confidence of adjacent bounding boxes based on the cross-union ratio, instead of setting their confidence scores to zero and deleting them; When weighted bounding box fusion is used, all boxes are merged to form the final result; the TPH-yolov5 network model will output feature maps of 4 different scales to detect objects of different scales. Also includes: The bounding boxes output by the TPH header are input into the model after weighted bounding box fusion for processing, and the results are visualized and analyzed for anomalies. After visualizing the results and analyzing anomalies, the trained TPH-yolov5 network model can be optimized using an additional self-trained classifier. The output of the trained classifier is the category of the abnormal behavior and the predicted bounding box.

2. The method according to claim 1, characterized in that, The steps for merging the weighted boxes are as follows: Step 1: Each predicted bounding box of each trained TPH-YOLOv5 network model is added to a single list B. List B is sorted in descending order of the confidence score C of each predicted bounding box, and then divided into box clusters and fused box clusters according to the range of the confidence score C. List L is used to store box clusters, and list F is used to store fused box clusters. Each position in list L contains a group of boxes or a single box, forming a cluster. Each position in list F contains only one box, which is a fused box of a box cluster in list L. Step 2: Loop through the predicted boxes in list B and find the matching boxes in list F; Step 3: If a matching box is found, add the matching box to the position pos corresponding to the matching box in list L and list F; recalculate the coordinates and confidence scores of the boxes in F[pos], using all the accumulated boxes in box cluster L[pos], a total of T boxes; Step 4: Repeat steps 2 and 3 until all predicted boxes in list B have been processed. Then, rescale the confidence scores in list F by multiplying them by the number of boxes in a cluster and dividing by the number of trained TPH-yolov5 network models, N.

3. The method according to claim 2, characterized in that, Step 3 also includes: If no matching box is found, the predicted box in list B is added to the end of lists L and F as a new entry; then the process continues to iterate through the next predicted box in list B.

4. The method according to claim 2, characterized in that, Step 4 also includes: If the number of frames in a cluster is small, the confidence score needs to be reduced using the following formula: or Where C is the confidence score, N is the number of trained TPH-yolov5 network models, and T is the number of boxes in F[pos].

5. The method according to claim 2, characterized in that, In step 3: The following fusion formula is used to fuse all accumulated boxes in the box cluster L[pos]: Where C is the confidence score, N is the number of trained TPH-yolov5 network models, T is the number of boxes in L[pos], and (X1,2) and (Y1,2) are the coordinates of the two diagonal vertices of the corresponding boxes in F[pos]. After merging all the boxes accumulated in the box cluster L[pos], a merged box is obtained. The coordinates of the merged box are the weighted sum of the coordinates of T boxes, where the weight is the confidence score of the corresponding box.

6. The method according to claim 1, characterized in that, Using image patches cropped from the training set as the classification training set, and selecting ResNet18 as the classifier network, a self-trained classifier is obtained.

Images