A method for detecting abnormal crowd behavior based on improved YOLOV4 algorithm

Abstract

The present invention provides a method for detecting abnormal behavior of a crowd based on an improved YOLOV4 algorithm, comprising the following steps: S1: on the basis of Darknet53, two layers of convolutional modules are added to the backbone network area to deepen the network length and transmit more position information to the deep network; S2: using the exponential moving average, i.e., EMA, to update the network parameters; S3: performing exponential average weighted calculation on the parameters obtained by each epoch training of the model; S4: using the Mixup function to replace the Moscia data enhancement function used in the YOLOV4 algorithm; S5: upsampling the high-level features in the feature extraction network and then combining these features, and again using the FPN+PAN structure to perform feature fusion on these features, so as to transmit more position information and semantic information during feature fusion. The present invention applies the improved version of the YOLOV4 algorithm to the abnormal behavior detection system of the crowd, and realizes the recognition and early warning of video image features such as knife holding, arson, and smoke inhalation.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal crowd behavior detection, and in particular to a method for detecting abnormal crowd behavior based on an improved YOLOV4 algorithm. Background Art

[0002] Intelligent video anomaly detection technology can be used to detect and warn of abnormal crowd behavior, thus protecting people’s lives and property.

[0003] In the literature, anomaly detection is divided into traditional methods such as Gaussian mixture model, hidden Markov model, optical flow method and spatiotemporal technology. Traditional abnormal behavior detection methods require more manual participation in area selection and feature extraction, lack objectivity and most abnormal scenes are relatively simple, which cannot meet the accuracy and speed requirements of today's crowd abnormal event detection. Crowd abnormal event detection based on deep learning is more conducive to feature extraction and scene migration.

[0004] However, there are some shortcomings in the current research on abnormal event detection in the crowd. The main ones are that abnormal behaviors are defined differently in different scenarios and are difficult to generalize; the low probability of abnormal events leads to an imbalance of positive and negative samples, making it difficult to learn enough abnormal behavior features; and the real-time performance of abnormal detection is poor, which cannot meet the needs of current video target tracking. With the rapid application of deep learning in the field of target detection, the detection accuracy and real-time performance of abnormal targets in the crowd can meet the requirements of monitoring video data processing by using target detection. Among them, the YOLOV4 algorithm is relatively mature in the field of target detection, and its detection accuracy and detection rate are better than SSD, YOLOV3, and Faster-RCNN. At present, in the research on the existing YOLOV4 improved algorithm, the main improvement idea is to improve the feature extraction ability of the network and thus improve the MAP value of the model.

[0005] The main improvement method of this application for the YOLOV4 algorithm is to add two layers of convolution modules in the backbone network area to transmit more position information to the deep network, and use the moving exponential average (EMA) to update the network parameters to optimize the training model; then use Mixup to replace the Mosica data enhancement method in YOLOV4 to increase the training data set; finally, improve the feature fusion structure (PAN) of YOLOV4 to transmit more semantic information during feature fusion.

[0006] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a method for detecting abnormal behavior of a crowd based on an improved YOLOV4 algorithm to solve the problems existing in the prior art.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] The present invention provides a method for detecting abnormal behavior of a crowd based on an improved YOLOV4 algorithm, comprising the following steps:

[0010] S1: Based on Darknet53, two layers of convolution modules are added to the backbone network area to increase the network length and transmit more position information to the deep network;

[0011] S2: Use the exponential moving average, or EMA, to update network parameters;

[0012] S3: Perform exponential average weighted calculation on the parameters obtained from each epoch training of the model;

[0013] S4: The Mixup function is used to replace the Moscia data enhancement function used in the YOLOV4 algorithm;

[0014] S5: Upsample the high-level features in the feature extraction network and combine them, and use the FPN+PAN structure to fuse these features again to transmit more position information and semantic information during feature fusion.

[0015] Preferably, the EMA in step S2 is used to estimate the local mean of the variable, so that the value of the variable at time t does not only depend on the data at the current time, but is a weighted average of the data near time t, so that the updated value becomes smoother and will not be affected by abnormal data at a certain time, thereby improving the robustness of the model; the EMA algorithm is shown in Formula 1:

[0016] V t =β·V t-1 + (1-β) · θ t (1);

[0017] Where V t represents the moving average of all parameters updated for the tth time, β represents the weight parameter, and θ t Represents all parameter weights obtained in the tth update;

[0018] The algorithm for exponential average weighted calculation in step S3 is shown in Formula 2:

[0019] N a =rN a +(1-r)N a-1 (2);

[0020] In the formula, N on the right side of the equal sign a This is the result of the nth model training. a ' is the result of weighted calculation, N a-1 It is the result of the weighted calculation of the n-1th model, and r is a proportion parameter.

[0021] Preferably, the Mixup function in step S4 uses a uniform distribution of 0.2-0.8 to select the coefficient of image fusion.

[0022] By adopting the above technical solution, the present invention has the following beneficial effects:

[0023] This paper proposes a method for detecting abnormal crowd behavior based on the improved YOLOV4 algorithm. Experimental results show that the MAP obtained by training the model with the improved version of the YOLOV4 algorithm on this data set is nearly 5% higher than that of the original algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A diagram showing the framework of adding two convolutional modules to the backbone feature extraction network.

[0026] Figure 2 This is the effect diagram of the Moscia data enhancement method used by YOLOV4;

[0027] Figure 3 This is the effect picture after the improved version of Mixup image enhancement;

[0028] Figure 4 This is a block diagram of the PAN structure used by YOLOV4;

[0029] Figure 5 It is a block diagram of the improved PAN structure;

[0030] Figure 6 This is a diagram of the test effect comparison experiment;

[0031] Figure 7 Improved MAP value of YOLOV4 algorithm;

[0032] Figure 8 There are three types of Precision and Recall values;

[0033] Fig. 9 This is the crowd density detection effect diagram;

[0034] Fig.10 This is the flow chart of the abnormal behavior detection system for crowds. DETAILED DESCRIPTION

[0035] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] This paper draws on the idea of ​​Resnet residual network and deepens the network length to pass more position information to the deep network. However, if more convolution structures are added to the shallow network structure, the overall network will be difficult to train and there will be no obvious improvement in the effect; if convolution structures are added to the deep network, the overall network parameters will increase significantly, which will cause a significant decrease in the model detection speed. Therefore, based on Darknet53, Figure 1 By adding two layers of convolutional modules to the backbone network area shown in the figure, the number of parameters of the backbone feature extraction network increases slightly and the detection speed hardly decreases.

[0037] In order to improve the test index of the model and increase its robustness, the present invention proposes to use the exponential moving average (EMA) to update the network parameters. EMA is used to estimate the local mean of the variable, so that the value of the variable at time t does not only depend on the data at the current time, but also performs a weighted average around time t, so that the updated value becomes smoother and will not be affected by a certain abnormal data, thereby improving the robustness of the model. The original EMA algorithm is shown in Formula 1:

[0038] V t =β·V t-1 + (1-β) · θ t (1);

[0039] Where V t represents the moving average of all parameters updated for the tth time, β represents the weight parameter, and θ t Represents all parameter weights obtained in the tth update;

[0040] YOLOV4 does not use dynamic coefficients to update the network weights. On this basis, the EMA algorithm introduced is improved accordingly. The improvement of the present invention is mainly to perform exponential average weighted calculation on the parameters obtained by each epoch training of the model. The algorithm is shown in Formula 2:

[0041] Na =rN a +(1-r)N a-1 (2);

[0042] In the formula, Na on the right side of the equal sign is the result obtained by the nth model training, and N on the left side is a ' is the result of weighted calculation, N a-1 is the result of the n-1th model weighted calculation, and r is a ratio parameter. As the model accuracy gradually improves in the later stages of training, the obtained model parameters are more accurate. -1 The proportion parameter value needs to be lowered, and the proportion parameter value needs to be increased in the initial stage of training. This experiment mainly divides the model training into four stages. The first stage sets r to 0.9950, the second stage sets r to 0.9970, the third stage sets r to 0.9990, and the fourth stage sets r to 0.9998. The experimental results show that the test indicators and robustness are higher than the original model.

[0043] The YOLOV4 algorithm uses the Moscia data enhancement method, which is similar to CutMix in theory. CutMix splices two pictures, while Moscia reads four pictures each time, scales, flips, and merges them into one picture to detect them. Figure 2 shown.

[0044] In order to increase the training data set and improve the robustness of the algorithm, Mixup is used to replace the Moscia data enhancement function. However, the Mixup function uses the Beta distribution function. The principle of the Beta distribution function is to directly superimpose the image and fuse the image semantic features. However, such an image cannot show the rich semantic information in the original image. Therefore, on this basis, the image fusion method of the Mixup function is modified to a certain extent. The image fusion coefficient is selected by a uniform distribution of 0.2-0.8, which can make the original image features more richly expressed. The effect is as follows: Figure 3 shown.

[0045] The YOLOV4 algorithm uses the PAN structure, which is an improvement on the FPN structure. The FPN structure refers to the feature map obtained by fusing high-level features with low-level features in the feature extraction network structure from top to bottom, and the PAN structure refers to the feature map obtained by fusing low-level features with high-level features from bottom to top. The feature fusion method of FPN+PAN has higher semantic information than the FPN in YOLOV3. The structure is as follows Figure 4 shown.

[0046] In order to further improve the effect of model fusion features, a new structure is adopted in FSSD

[14] . The method adopted is to upsample the high-level features in the feature extraction network and then combine these features. The FPN+PAN structure is used to fuse these features again. This can transmit more position information and semantic information during feature fusion. The improved PAN structure is shown in Figure 2. Figure 5 shown.

[0047] The present invention will be further described below in conjunction with specific experimental data:

[0048] This experiment is a programming experiment conducted on a server running the Ubuntu 18.0 operating system.

[0049] The dataset is annotated with images through labellmg. During the model training process, the parameter settings are shown in Table 1:

[0050] Table 1 YOLOV4 improved algorithm model training parameters

[0051]

[0052] The comparison of target detection between the improved YOLOV4 algorithm and the model trained by the YOLO V4 algorithm is shown below. Figure 6 As shown in the figure. The detection accuracy of the three categories of fire, smoke, and knife has been improved. In Figure (a), the original YOLOV4 algorithm produced a false detection when detecting the knife category, while in Figure (c), the improved YOLOV4 algorithm can correctly detect the knife category. The experimental results show that the confidence of the detection results of the improved algorithm is higher than that of the original YOLOV4 algorithm, and the improved algorithm is more robust.

[0053] Combination Figure 7-8 As shown in the figure, this experiment mainly evaluates the training model from three evaluation indicators, namely: Mean Average Precision (mAP), Recall, and Frames Per Second (FPS), among which Precision needs to be introduced. The algorithm is shown in formulas 3 and 4:

[0054]

[0055] In the formula, Tp represents the number of positive samples that are predicted and correct, Fp represents the number of positive samples that are predicted but are actually incorrect, and Fn represents the number of negative samples that are predicted but are actually incorrect. The area surrounded by the PR curve is the AP value of a certain category, and the MAP value is the average of the AP values ​​of all analogies. The larger the MAP value, the higher the accuracy of the model in identifying the target.

[0056] In order to further verify the robustness of the improved YOLOV4 algorithm of the present invention, the original YOLOV4 algorithm, the EMA improved network parameter (YOLOV4_ema) algorithm, and the improved algorithm of the present invention were verified on this data set. The test average precision mean (MAP) and the number of frames per second (FPS) of the detected image are shown in Table 2:

[0057] Table 2 Comparison results of YOLOV4 improved algorithm

[0058] Table 2 Comparison results of yolov4 improved algorithm

[0059]

[0060] From the comparative experimental data, it can be seen that the improved YOLOV4 algorithm of the present invention has a higher detection average precision mean, with a MAP value of 81.41%, the MAP value of the original YOLOV4 is 76.43%, and the MAP value of YOLOV4_ema is 78.18%. The experiment shows that the improved YOLOV4 model has higher detection accuracy and stronger robustness. At the same time, in terms of video processing speed, the FPS of the improved YOLOV4 algorithm is slightly lower than that of the original algorithm, but it can meet the needs of real-time detection of surveillance videos.

[0061] Crowd density detection is a hot research topic today. Inspired by multi-branch deep convolutional neural networks, Zhang et al.

[15] proposed a multi-column convolutional neural network (MCNN) to estimate the number of people. The dataset used in the MCNN experiment is ShanghaiTech, which contains 1,198 images and 330,165 annotation headers. The entire dataset consists of Part_A and Part_B. Part_A contains 482 images of high-density crowds, and Part_B is an image data collected from the streets in the center of Shanghai, where the crowd density is lower than that of Part_A.

[0062] The evaluation indicators used by MCNN are mean absolute error MAE and mean square error MSE. The algorithm is shown in formulas 5 and 6:

[0063]

[0064] The MAE and MSE values ​​obtained using the pre-trained model are shown in Table 3:

[0065] Table 3 Evaluation index values ​​of MCNN algorithm training model

[0066] Table 3 Evaluation index value of MCNN algorithm training model

[0067]

[0068] The effect of detecting crowd density is as follows Fig. 9 shown.

[0069] This application divides the crowd density into low, medium, high and very high levels according to the current number of people. A crowd abnormal event detection system framework based on YOLO and MCNN algorithms is proposed. The process of the crowd abnormal behavior detection system is as follows: Fig.10 shown.

[0070] In summary, the present invention proposes a method for detecting abnormal crowd behavior based on the improved YOLOV4 algorithm. The experimental results show that the MAP obtained by training the model with the improved version of the YOLOV4 algorithm on this data set is nearly 5% higher than that of the original algorithm.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting abnormal behavior of a crowd based on an improved YOLOV4 algorithm, characterized in that: The steps include: S1: Based on Darknet53, two layers of convolution modules are added to the backbone network area to increase the network length and transmit more position information to the deep network; S2: Use the exponential moving average, or EMA, to update network parameters; S3: Perform exponential average weighted calculation on the parameters obtained from each epoch training of the model; S4: The Mixup function is used to replace the Moscia data enhancement function used in the YOLOV4 algorithm; S5: Upsample the high-level features in the feature extraction network and combine them, and use the FPN+PAN structure to fuse these features again to transmit more position information and semantic information during feature fusion.

2. The method for detecting abnormal behavior of a crowd based on the improved YOLOV4 algorithm according to claim 1, characterized in that: The EMA in step S2 is used to estimate the local mean of the variable, so that the value of the variable at time t does not only depend on the data at the current time, but also performs a weighted average around time t, so that the updated value becomes smoother and will not be affected by a certain abnormal data, thereby improving the robustness of the model; the EMA algorithm is shown in Formula 1: V t =β·V t-1 + (1-β)·θ t (1); Where V t represents the moving average of all parameters updated for the tth time, β represents the weight parameter, and θ t Represents all parameter weights obtained at the tth update.

3. The method for detecting abnormal behavior of a crowd based on the improved YOLOV4 algorithm according to claim 1, characterized in that: The Mixup function in step S4 selects the image fusion coefficient using a uniform distribution of 0.2-0.8.

Images