An improved YOLO v5-based high-altitude fireworks detection method

Abstract

The application discloses a high-altitude fireworks detection method based on an improved YOLOv5, and relates to the technical field of video detection, and in particular relates to a high-altitude fireworks detection method based on an improved YOLOv5. In the case of a plurality of high-altitude camera video streams, a multi-thread queue algorithm is applied, a frame extraction and uploading thread is set for each video stream, and the frame-extracted pictures of all threads are transmitted into an infinitely long queue. A high-altitude angle fireworks dataset is established, and data cleaning and dataset labeling work are carried out. The backbone network in the YOLOv5 network is modified, and each standard convolution layer after a CSP structure is replaced by a hybrid attention module. Training hyperparameters are set, the improved YOLOv5 network in the third step is trained based on the fireworks dataset established in the second step, a fireworks detection model is obtained after training is completed, and the pictures obtained in the first step are input into the fireworks detection model for smoke and flame detection. The improved target recognition algorithm is used to extract a single frame of a video stream to complete detection of fire abnormal phenomena.

Description

TECHNICAL FIELD

[0001] The application relates to a neural network, an image recognition technology, and belongs to the field of image processing and deep learning, and more particularly to a high-altitude fireworks detection method based on an improved YOLO v5. BACKGROUND

[0002] An application with the application number CN202110589355.9 and the invention name of a video-based fireworks detection method is provided. The invention comprises the steps of image acquisition, image combination, fireworks target detection, and deep learning. The three-channel color image input is modified into a multi-channel image composed of images at different time points of the same camera. A joint algorithm composed of multiple detection algorithms is used to track the fireworks target. The invention is mainly used for safety monitoring.

[0003] Although the patent uses multiple detection algorithms including YOLO, it does not change the network structure of these algorithms. It only optimizes the channel data information of the input picture and combines multiple images. It cannot fundamentally and significantly improve the accuracy of fireworks detection. At the same time, this algorithm only considers the fireworks tracking of a single camera video stream, but in actual application scenarios, the algorithm needs to face multiple video streams. The tracking method mentioned in the invention has certain limitations, and the effectiveness of this tracking method cannot be guaranteed at high altitudes with small fireworks targets.

[0004] Fireworks detection technology is an important part of fire prevention technology. In recent years, residents use fire more frequently, and industrial electrical equipment is more complex and diverse. Existing sensing detection equipment is increasingly difficult to cope with new fire prevention situations. Therefore, domestic and foreign scholars have conducted a large amount of research on fire detection and have made significant progress.

[0005] Existing fire detection algorithms can be roughly divided into two categories: feature-based models and learning-based models [1] . The former aims to analyze the characteristics of flames, smoke, and other features, and realizes fire warning by summarizing the characteristic rules of fire areas. The latter obtains a classifier or feature extraction model with learning ability through intelligent algorithms. Chen et al. analyzed a large number of flames [2] and proposed a flame recognition criterion based on a color model. However, this algorithm only considers color information, resulting in a high false alarm rate of the detection result. Dimit et al. used the dynamic jumping feature of flames [3] and proposed a flame boundary detection model based on wavelet time-frequency characteristics. In order to realize early warning, Tore et al. used the characteristic of a large amount of smoke in the early stage of fire [4] and proposed a fire warning based on a smoke model.

[0006] With the rapid development of target detection technology, the single feature recognition in traditional methods is difficult to achieve high accuracy. The judgment of smoke and flame through the advanced features of fireworks target is a more effective method, but it still faces many challenges.

[0007] Due to the existence of some uncontrolled conditions, the specific method cannot adapt to the complex and variable high-altitude camera scene. Different light will cause the change of object edge and its shadow, affecting the imaging quality and image processing effect [9] ; The target has the shielding condition, which will cause the difficulty of model recognition of global features, resulting in large detection error

[10] ; The color, texture and appearance information of the same target at different angles are very different, which leads to inaccurate target recognition.

[0008] Most algorithms have low accuracy for small target detection, and there are missed and misjudged cases. Although the target detection technology based on deep learning has gradually replaced the traditional manual feature extraction method and become the mainstream, there is still room for improvement in the detection of small targets

[11] , The reasons for this problem include low resolution, image blur, less information, and more noise; Using image data enhancement technology can produce some effect, however, many studies have shown that the image processing effect of the original data is always limited, changing the neural network structure will better solve the problem.

[0009] When a large amount of real-time video data is analyzed online, there will be a large number of pictures that need to be quickly detected. For example, in the process of capturing multiple RTSP video streams, algorithms need to be used to ensure that within a certain time interval, each video stream can have a frame of picture detected to ensure the efficiency and stability of the system.

[0010] These technical problems restrict the development of fire and smoke detection technology, and it is urgent to propose a technical solution to make the fire and smoke detection system based on deep learning better for fire warning.

[0011] The continuous development of image processing and deep learning in recent years has further upgraded the fire detection technology. In recent years, more advanced target detection networks include FPN network [5] , Fast RCNN network [6] , SSD network [7] and YOLO series network [8] and so on. Among them, the YOLO series network has been iterated for many versions, and the latest one is YOLO v5, which has a strong advantage in the rapid deployment of the model. The obtained model has the advantages of small file size, fast inference speed, short training time, etc. The target detection model trained can ensure high accuracy and performance at the same time.

[0012] The high-efficiency target detection algorithm can improve the accuracy of the fireworks recognition, and after further improvement, the high-altitude camera can be used for fire warning; in the case of multiple video streams, a reasonable frame extraction interval can be set, and the multi-thread technology can be used to extract frames of the multiple video streams and store the multiple images in a specific queue for smoke and fire detection in sequence. SUMMARY

[0013] The purpose of the present application is to overcome the deficiencies in the prior art, and a high-altitude fireworks detection method based on improved YOLO v5 is proposed, which extracts single frames of video streams through the improved target recognition algorithm to detect fire abnormal phenomena.

[0014] The purpose of the present application is realized through the following technical solutions.

[0015] The high-altitude fireworks detection method based on improved YOLO v5 includes the following processes:

[0016] Step 1: In the case of multiple high-altitude camera video streams, a multi-thread queue algorithm is applied, a frame extraction upload thread is set for each video stream, and the frame extraction pictures of all threads are transmitted into an infinite queue;

[0017] Step 2: Establish a high-altitude angle of fireworks dataset, and perform data cleaning and dataset labeling work;

[0018] Step 3: Modify the backbone network in YOLO v5 network, replace each standard convolution layer after CSP structure with a hybrid attention module;

[0019] Among them, after the feature extraction of the backbone network, the feature fusion is completed by applying the Neck part, and finally the target prediction is completed by using the CIoU loss function in the Head part, the optimal target frame is selected from multiple frames by the nms non-maximum suppression algorithm, and the final detection result is formed;

[0020] Step 4: Set the training hyperparameters, train the improved YOLOv5 network based on the fireworks dataset established in step 2, and obtain the smoke and fire detection model after training, and input the pictures obtained in step 1 into the smoke and fire detection model for smoke and fire detection.

[0021] In the first step, the number of video streams is determined, and different threads are created for different video streams to ensure that the image upload of each video stream is not disturbed by each other; in addition, according to the detection time of YOLO v5 network and the number of video streams, the frame extraction interval of each video stream is adjusted accordingly.

[0022] In the second step, the preliminarily acquired fireworks dataset images are cleaned, and then the dataset images meeting the high-angle are converted into jpg format, and then the images are labeled using a labeling tool to form corresponding xml files.

[0023] In the third step, the YOLO v5 network adopts a pre-trained model YOLO v5l provided by an official, and the YOLO v5 network includes three parts of a backbone network, a Neck and a Head connected in sequence, the backbone network is mainly composed of a Focus structure and a CSP structure, and the backbone network is used for acquiring features of a training image or an image to be recognized, a standard convolution layer is arranged after each CSP structure, the standard convolution layer is replaced by a hybrid attention module, and the hybrid attention module is composed of a channel attention module and a spatial attention module connected in sequence;

[0024] The structure of the Neck is FPN+PAN, feature fusion is performed, and the calculated features are transmitted to the Head part; the main body of the Head part is a detector, grid prediction is performed on a feature map, and the step is recycled until the coordinates of a detection frame are generated, finally, a CIoU-nms non-maximum suppression method is used to remove redundant target frames, and then a detection result is output;

[0025] In the fourth step, the hyperparameters include the depth and width of the network, a learning rate, a batch size, an epoch and a training dataset used.

[0026] Compared with the prior art, the technical scheme of the present application has the following beneficial effects:

[0027] The improved YOLO v5-based high-altitude fireworks detection method provided by the present application has the following advantages: on the one hand, in order to improve the accuracy of the model after training, a lightweight attention module is introduced into the original backbone network; the attention module enables the features to cover more parts of the object to be recognized, which can effectively improve the feature extraction capability of the neural network for the input image, thereby improving the performance of the model in the small target detection scene and reducing the false positives and false negatives in high-altitude fireworks detection; on the other hand, since the actual application scenario usually requires the model to capture multiple video streams for separate detection, the original algorithm can only recognize a single video stream, by introducing a multi-thread queue algorithm, the problems of picture accumulation and data buffer overflow when processing pictures transmitted by multiple cameras are avoided, thereby ensuring the stability of the system when multiple video streams are simultaneously transmitted.

[0028] Based on the high-quality high-altitude fireworks dataset, the attention mechanism is introduced into the backbone network of YOLO v5, which improves the detection accuracy. On the basis of optimizing each link of the fireworks detection, the fireworks detection method can be designed to be deployed on the server to build a fireworks detection system. The improved fireworks detection method can effectively detect the smoke and flame at the initial stage of the fire in the high-altitude camera angle, so as to achieve the effect of early warning and promote the development of fire prevention technology. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The flowchart of the high-altitude fireworks detection method based on the improved YOLO v5 of the application;

[0030] Figure 2 The flowchart of the multi-threaded queue algorithm;

[0031] Figure 3 The schematic diagram of introducing the mixed attention mechanism into the standard convolution layer;

[0032] Figure 4 The schematic diagram of the channel attention module;

[0033] Figure 5 The schematic diagram of the spatial attention module; DETAILED DESCRIPTION

[0034] In order to make the purpose, technical scheme and advantages of the application more clear, the embodiments of the application will be further described in detail below with reference to the drawings.

[0035] The existing fireworks detection method and technology have many deficiencies, and the application aims to start from multiple links in the detection process, improve the accuracy of the fireworks detection, guarantee the robustness of the detection system, so that it can detect the small target smoke and flame at high altitude with high accuracy, and still maintain the high efficiency of the detection system in the case of multiple video streams. Therefore, the technical scheme of the application introduces the attention mechanism into the YOLO v5 backbone network by adding multiple attention modules, and changes the algorithm for processing the video stream in the process of detecting fireworks, aiming at improving the accuracy and stability when facing multiple video stream tasks.

[0036] There are multiple standard convolution layers in the backbone network of the original YOLO v5, which can be described as filters at each level, which realizes feature dimension reduction by compressing the number of data and parameters, enhances the nonlinearity of the network while deepening the network level. The attention mechanism allows the neural network to dynamically focus on some parts of the input that contribute to the recognition of the current target

[12] The order and length of the input are changed to improve the efficiency of the neural network according to the relevance. The attention mechanism is usually introduced into the neural network by adding an attention module. According to the principle of the attention module, the attention module can be divided into a spatial attention module, a channel attention module and a hybrid attention module. In the process of improving the backbone network of YOLO v5, the original standard convolution layer is replaced by a hybrid attention module to improve the performance of the network layer. At the same time, due to the light weight of the module, the influence on the inference time is relatively small, which can be generally ignored, and this is also proved in the subsequent experiment.

[0037] The original YOLO v5 can only recognize single video stream when detecting fireworks. In actual scenarios, 60-80 video streams need to be considered to avoid data buffer problems and even detection system crashes. Multi-threading refers to a technology for implementing concurrent execution of multiple threads from software or hardware

[13] A computer with multi-threading capability can execute more than one thread at the same time due to hardware support, thereby improving overall processing performance. Based on the principle of multi-threading, an infinitely long queue is set to store the pictures obtained by frame extraction of multiple video streams. Considering the detection time of the YOLO v5 network and the number of video streams, a reasonable frame extraction interval is set to ensure that the picture queue does not gradually become longer in actual application.

[0038] The present application modifies the structure of the original target detection network for the case of multiple high-altitude camera video streams, and proposes a high-altitude firework detection method based on improved YOLO v5, which solves the detection problem of small target smoke and flame in the early stage of fire. As Figure 1 shown, the process includes the following steps:

[0039] First step: multi-threaded queue algorithm

[0040] In the case of multiple high-altitude camera video streams, a multi-threaded queue algorithm is applied, a frame extraction upload thread is set for each video stream, and the frame extraction pictures of all threads are transmitted into an infinitely long queue. Among them, the number of video streams needs to be determined, and different threads are created for different video streams to ensure that the image upload of each video stream is not disturbed by each other. In addition, the frame extraction interval of each video stream should be set to ensure that the YOLO v5 network can complete the detection task of the frame extraction picture in time. According to the number of video streams and the detection time of the YOLO v5 network, the frame extraction interval of each video stream needs to be adjusted accordingly.

[0041] As Figure 2Flow of the multi-threaded queue algorithm shown. In the present application, the pictures detected by the YOLO v5 network are obtained by frame extraction from multi-channel overhead camera video streams. To ensure the concurrency of multiple video streams, a queue is set up to store the pictures, and then a thread is created for each camera. In the thread, the RTSP video stream corresponding to the overhead camera needs to be determined first, and then the frame rate of the video stream is obtained. By multiplying the frame rate by the frame extraction interval time, the number of frames that can be extracted per second can be set through a loop. Finally, the frame-extracted pictures are pushed into the queue. Since the frame-extracted pictures of all video stream threads are not directly transmitted to the picture detection thread, but the results are transmitted to the picture queue, the connection between picture extraction of multi-channel video streams and picture detection of the YOLO v5 network is indirectly completed through the enqueue and dequeue operations of the queue, so the concurrency between the overhead cameras can be well guaranteed.

[0042] In the python language used in the multi-threaded queue algorithm, the data between multiple threads is shared, and when multiple threads exchange data with a queue, the safety and consistency of the data cannot be guaranteed, and confusion and loss may occur. Therefore, in addition to considering the control problem in the multi-threaded method, the queue library function is used to set up the following concurrent control mechanism: if there are only two threads, when one thread is pushing pictures into the queue, a signal is sent to the other thread, and after the specified number of pictures is pushed into the queue by the thread, the other thread pushes pictures into the queue; for more than or equal to 3 threads, the method is extended to send signals to threads that are not currently pushing pictures.

[0043] In addition, in order to ensure that the frame-extracted pictures of each video stream can be detected in a timely manner and avoid the situation where the latest enqueued pictures are accumulated for a long time due to the gradual lengthening of the picture queue, the frame extraction interval time t i The following formula needs to be met:

[0044]

[0045] wherein, the average time required for the YOLO v5 network to detect a picture, the average time required for the dequeue operation of the picture queue, n is the number of video streams, the average time required for the enqueue operation of the picture queue, a time reserved to take into account the instability of data transmission and image detection speed, which can be adjusted manually.

[0046] Second step: establishment and data enhancement of overhead angle fireworks data set

[0047] A high-altitude angle fireworks dataset is established, and data cleaning and labeling dataset work is performed. Among them, the preliminary obtained fireworks dataset images need to be cleaned, then the dataset images that meet the high-altitude angle (i.e. at least 500 meters from the fireworks target) are converted to jpg format, and then the image is labeled using a labeling tool to form the corresponding xml file.

[0048] Considering that the model trained by the network will be used in actual scenarios in the future, a total of 1800 images were collected from the network and public cameras, and after further optimization and screening, there were still 1264 images left, which constituted the dataset for training the model. Use the Sprite Annotation Assistant to add labels to each image, including smoke and flame, and finally form the xml label file corresponding to the image, which contains the four coordinates of the target box and the category of the object in the box, in the format of PASCAL VOC. Randomly divide it into training set and validation set, of which the training set has 948 images and the validation set has 316 images.

[0049] Data augmentation before the image enters the network training will enrich the distribution of data, significantly improve the generalization and robustness of the model, and prevent overfitting. Use the Mosaic data augmentation method to randomly scale, randomly crop, and randomly arrange the images to splice them; take adaptive image scaling measures, which can uniformly scale the original image to a standard size, and adjusting the black edge filling can improve the inference speed by about 40%.

[0050] Step 3: Introduction of attention mechanism in YOLO v5

[0051] Modify the backbone network in YOLO v5 network, replace each standard convolution layer after CSP structure with a hybrid attention module. After the backbone network extracts features, the Neck part is used to complete feature fusion, and finally the CIoU loss function is used in the Head part to complete target prediction. Through the nms non-maximum suppression algorithm, the optimal target box is selected from multiple boxes to form the final detection result.

[0052] The YOLO v5 network can adopt the pre-trained model YOLO v5l provided by the official, and the YOLO v5 network includes three parts of backbone network, neck and head connected in turn. The backbone network (Backbone) is mainly composed of Focus structure and CSP structure, and its function is to obtain the features of the training image or the image to be recognized. When the image data just enters the Backbone, the Focus module performs slicing operation on the image, and after convolution operation, a two-fold down-sampling feature map without information loss is obtained. The CSP structure in the Backbone can help the neural network to realize more rich gradient combination, while reducing the calculation amount, improving the inference speed and accuracy. There is a standard convolution layer after each CSP structure, which is replaced by a hybrid attention module

[14] As shown in Figure 3 , the hybrid attention module is composed of one channel attention module and one spatial attention module connected in turn, which has the characteristics of spatial attention module and channel attention module, and has more excellent performance, and still maintains the characteristics of lightweight, which can effectively improve the learning weight distribution ability of the neural network structure.

[0053] The structure of the Neck is FPN+PAN [15,16] , which can perform feature fusion and transmit the calculated features to the Head part. The main part of the Head part is the detector, which performs grid prediction on the feature map, and the subsequent steps can be repeated until the coordinates of the detection frame are generated. Finally, the CIoU-nms non-maximum suppression method is used to remove redundant target frames, and the detection result can be output.

[0054] Many attention modules support plug and play, which can change the performance of the network by replacing the standard convolution layer with the module in a specific way, and the activation function is changed from SiLU to Leaky ReLU.

[0055] SiLU: , where σ represents the sigmoid function.

[0056] Leaky ReLU: .

[0057] (1) For the channel attention module, as shown in Figure 4 , assuming that the input is a feature map, first perform maximum pooling and average pooling (pooling size ), get a features; then through two fully connected layers, the number of neurons of the first fully connected layer is c / r (r is a set parameter), which is a dimension reduction method, and the second fully connected layer is dimensioned to C neurons. The advantage of this is to increase more nonlinear processing and to fit the complex correlation between channels; then a sigmoid layer is obtained features, and finally a raw and feature map full multiplication operation, matrix full multiplication can obtain feature maps with different channel importance. Therefore, it mainly learns the parameters of the two fully connected layers, and then updates the parameters of the two fully connected layers using the final classification loss.

[0058]

[0059] wherein, AvgPool(F) and MaxPool(F) represent average pooling and maximum pooling of the input feature map F respectively. MLP represents a set multi-layer perceptron, and σ represents a sigmoid activation function. W0 and W1 represent the process of allocating weights in MLP, W0 represents the process of feature map through the first fully connected layer, and W1 represents the process of feature map through the second fully connected layer, and the activation function used is Leaky ReLU. and respectively represent the feature maps of F after average pooling and maximum pooling in the channel attention module, and M c (F) represents a one-dimensional channel attention map obtained by calculation.

[0060] (2) As shown in Figure 5 , the input of the spatial attention module is the feature map output by the channel attention module. First, maximum pooling and average pooling are performed along the channel dimension, and the size of the feature map obtained by each pooling is ; then the two pooled feature maps are connected based on the channel, becoming a feature map with a size of ; then a convolution operation is performed on the obtained feature map with a kernel size of , a convolution kernel number of 1, and a sigmoid activation function is used, and finally a matrix full multiplication operation is performed. From the overall module, this channel cascading method can better allocate weights.

[0061]

[0062] wherein, AvgPool(F) and MaxPool(F) represent average pooling and maximum pooling of the input feature map F along the channel dimension. f 7x7 represents a convolution operation on the two connected feature maps with a kernel size of convolution operation, and σ represents a sigmoid activation function. and respectively represent the feature maps after F is subjected to average pooling and maximum pooling in the spatial attention module, and M s (F) represents a two-dimensional spatial attention map calculated.

[0063] (3) The channel attention module and the spatial attention module are sequentially connected in a series module manner to form a hybrid attention module.

[0064]

[0065]

[0066] wherein, M c (F) represents a one-dimensional channel attention map with a size of ; M s (F) represents a two-dimensional spatial attention map with a size of . represents a feature obtained after the input feature F is subjected to the channel attention module, represents a feature obtained after the feature map is subjected to the spatial attention module, represents element-level multiplication.

[0067] The neck after the backbone is a key link in the target detection framework, and the network design thereof adopts an FPN+PAN structure. The FPN is a classic structure of a feature pyramid, and can construct a high-level semantic feature map at all scales. After passing through multiple layers of networks in the FPN, the feature information at the bottom layer is very fuzzy, and the PAN can help to make up and strengthen the positioning information. The neck can mix and combine important features extracted by the backbone, which is conducive to the specific task learning of the head, such as classification, regression and other common tasks.

[0068] The features after fusion are input into the head part, which includes multiple detectors, and the prediction boxes are obtained after grid prediction and combination. The loss function used here is CIoU, which not only considers the overlapping area and center distance of the bounding box, but also introduces a penalty term of the aspect ratio, and is more inclined to optimize the direction of increasing the overlapping area.

[0069] The weight function α:

[0070] The consistency v of the width-height ratio:

[0071] The final CIoU Loss is defined as:

[0072] wherein, IoU represents the intersection of the joint bounding box, represents the distance between the centers of two bounding boxes A and B, p represents the calculation of the Euclidean distance, and c represents the diagonal distance of the smallest rectangle formed by the two bounding boxes. and respectively represent the width and height of the detection frame detected by the model, and w and h respectively represent the width and height of the detection frame labeled in the original data set.

[0073] The method used to remove redundant prediction frames is the nms non-maximum suppression algorithm, which selects the optimal target detection frame through the confidence.

[0074] Step 4: Set the training hyperparameters, train the improved YOLOv5 network based on the smoke and fire data set established in step 2, and obtain the smoke and fire detection model after training. The images obtained in step 1 are input into the smoke and fire detection model for smoke and flame detection. The hyperparameters that need to be set include the depth and width of the network, learning_rate, batch_size, epoch, and the training data set used, etc. If necessary, the loss function can also be modified, and the non-maximum suppression algorithm can be replaced. After completing the settings, the new improved network training model is used to design comparative experiments and ablation experiments to verify the effect of the smoke and fire detection method.

[0075] In order to balance the operation amount and detection accuracy of the network as much as possible within the allowable range, the depth and width of the network are kept unchanged, and the pre-trained model YOLO v5l provided by YOLO v5 is used. The network depth parameter and network width parameter are both 1.00. In order to achieve the best performance of the model, a pre-experiment is conducted to explore better hyperparameters. The best learning rate is searched in a fixed numerical interval, and the learning rate learning_rate is finally selected as 0.001. In order to prevent overfitting and consider hardware factors, the batch_size is set to 4. The pre-experiment shows that the loss value of the model Loss value decreases sharply in 0-100 epochs, and slowly decreases in 100-120 epochs until it tends to be stable, so the epoch is set to 120.

[0076] The experiment of training the model needs to run under the Pytorch framework and CUDA parallel computing architecture, and needs to integrate cuDNN to accelerate the computer to run the experiment. The operating system during actual operation of the experiment is Ubuntu 20.04, the graphics card is RTX3070, the Python version is 3.8, the CUDA version is 11.1, the cuDNN version is 8.0, and the version of the Pytorch framework is 1.8.0.

[0077] The evaluation standard of model training is mean average precision (mAP)

[17] , which is derived from average precision (AP). Average accuracy refers to the calculation of the maximum precision corresponding to each possible value of recall rate, and then the average of these precision values can be obtained, which can measure the goodness of the model on a single class. The mAP value additionally calculates the average of AP values across all classes, which can measure the goodness of the learned model across all classes.

[0078]

[0079] where q represents a single target category in multi-target detection, Q represents a set of all target categories, N represents the number of all target categories, and AP(q) can calculate the AP value of a single target category in detection.

[0080] To verify the better effect of this method of adding attention modules, the number and type of data sets are kept unchanged, and other several attention modules SE, ECA [18,19] are added to the same position of the backbone network, together with the original YOLO v5 network, to form a comparative experiment and an ablation experiment, and the experimental results are shown in Table 1. In order to reduce the error in the training process of the experiment, the hyperparameter epoch is changed for multiple experiments, the number of epoch settings is 100 and 120, and the speed of the trained model for detecting pictures is calculated, so as to obtain the inference speed.

[0081] Table 1

[0082]

[0083] Note: The "inferring time" in the table is the time for the trained model to detect 418 pictures, fire mAP@0.5 and smoke mAP@0.5 are the data of the trained flame and smoke, all classes mAP@0.5 is the data obtained by adding and averaging all target categories, and the number after @ represents the threshold for determining positive and negative samples.

[0084] From the analysis of the experimental data, it can be concluded that the addition of attention modules has a certain effect on the performance improvement of the YOLO v5 network, and all networks with added attention modules have higher mAP than the original network. In terms of model inference time, the addition of various attention modules slightly increases the inference time, which is due to the increase in the complexity of the model caused by the addition of new parts in the original network structure. In terms of convergence speed, there is no significant difference between all networks on this dataset, which may be due to the small number of target categories (only two). From the performance of each module, the selected hybrid attention module added to YOLO v5 has the best performance.

[0085] Although the functions and working processes of the present application are described above in combination with the drawings, the present application is not limited to the specific functions and working processes described above. The above specific embodiments are merely illustrative and not limiting. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope of protection of the claims, which are all within the scope of protection of the present application.

[0086] References

[0087]

[0088] [2] Chen, T.H., Wu, P.H. and Chiou, Y.C. (2004) An Early Fire-Detection Method Based on Image Processing. International Conference on Image Processing (ICIP), 24-27 October 2004, 1707-1710.

[0089] [3] K. Dimitropoulos, P. Barmpoutis and N. Grammalidis. Spatio-Temporal Flame Modeling and Dynamic Texture Analysis for Automatic Video-Based Fire Detection. in IEEE Transactions on Circuits and Systems for Video Technology, vol. 25, no. 2, pp. 339-351, Feb. 2015, doi: 10.1109 / TCSVT.2014.2339592.

[0090] [4] Toreyin, B.U., Dedeoglu, Y. and Cetin, A.E. (2005) Flame Detection in Video Using Hidden Markov Models. Proceedings of IEEE International Conference on Image Processing, 2, 1230-1233.

[0091] [5] Golnaz Ghiasi, Tsung-Yi Lin, Quoc V. Le. NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 7036-7045.

[0092] [6] Ross Girshick. Fast R-CNN. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1440-1448.

[0093] [7] Liu, W. et al. (2016). SSD: Single Shot MultiBox Detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds) Computer Vision - ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, vol 9905. Springer, Cham.

[0094] [8] Bochkovskiy A, Wang C Y, Liao H Y M. Yolov4: Optimal speed and accuracy of object detection[J]. arXiv preprint arXiv:2004.10934, 2020.

[0095] [9] Arad B, Kurtser P, Barnea E, et al. Controlled lighting and illumination-independent target detection for real-time cost-efficient applications. the case study of sweet pepper robotic harvesting[J]. Sensors, 2019, 19(6): 1390.

[0096]

[0097]

[11] Zhang H, Zhang L, Yuan D, et al. Infrared small target detection based on local intensity and gradient properties[J]. Infrared Physics & Technology, 2018, 89: 88-96.

[0098]

[12] Rush A M, Chopra S, Weston J. A neural attention model for abstractive sentence summarization[J]. arXiv preprint arXiv:1509.00685, 2015.

[0099]

[13] Zhang Z, Huang K, Tan T. Multi-thread parsing for recognizing complex events in videos[C]. European conference on computer vision. Springer, Berlin, Heidelberg, 2008: 738-751.

[0100]

[14] Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module[C]. Proceedings of the European conference on computer vision (ECCV). 2018: 3-19.

[0101]

[15] Xu H, Yao L, Zhang W, et al. Auto-fpn: Automatic network architecture adaptation for object detection beyond classification[C]. Proceedings of the IEEE / CVF international conference on computer vision. 2019: 6649-6658.

[0102]

[16] Yang J, Fu X, Hu Y, et al. PanNet: A deep network architecture for pan-sharpening[C]. Proceedings of the IEEE international conference on computer vision. 2017: 5449-5457.

[0103]

[17] Shafiee M J, Chywl B, Li F, et al. Fast YOLO: A fast you only look once system for real-time embedded object detection in video[J]. arXiv preprint arXiv:1709.05943, 2017.

[0104]

[18] Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7132-7141.

[0105]

[19] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks. CVPR, 2020.

Claims

1. An improved YOLO v5-based high-altitude fireworks detection method, characterized in that, The method comprises the following steps: First step: in the case of multi-path high-altitude camera video stream, a multi-threaded queue algorithm is applied to set a frame extraction and uploading thread for each video stream, and the frame extraction pictures of all threads are transmitted into an infinite-length queue; Wherein, the frame extraction and uploading thread is realized by the following way: obtaining the frame rate of the video stream, multiplying the frame rate by the frame extraction interval time, and cyclically intercepting the video frame picture, and pressing the frame extraction picture into an infinite-length queue; The frame extraction interval time is dynamically adjusted by the following formula: ， wherein, is the first is the frame interval time of the video stream, is the average time required for the YOLO v5 network to detect a picture, is the average time required for the dequeue operation of the picture queue, and n is the number of video streams, is the average time required for the enqueue operation of the picture queue, is the time appropriately reserved in consideration of the instability of data transmission and image detection speed; Second step: a high-altitude angle fireworks data set is established, and data cleaning and labeled data set work is carried out; Third step: the backbone network in YOLO v5 network is modified, and each standard convolution layer after CSP structure is replaced by a hybrid attention module; Wherein, after the backbone network extracts features, the Neck part is applied to complete feature fusion, and finally the CIoU loss function is used in the Head part to complete target prediction, and the optimal target box is selected from multiple boxes by nms non-maximum suppression algorithm to form the final detection result; Fourth step: set the training hyperparameters, train the improved YOLO v5 network based on the fireworks data set established in the second step, and obtain the fireworks detection model after training, and input the picture obtained in the first step into the fireworks detection model for smoke and flame detection.

2. The high-altitude fireworks detection method based on improved YOLO v5 according to claim 1, characterized in that, In the first step, the number of video streams is determined, and different threads are created for different video streams to ensure that the image uploading of each video stream is not disturbed by each other; in addition, according to the detection time of YOLO v5 network and the number of video streams, the frame extraction interval of each video stream is adjusted accordingly.

3. The high-altitude fireworks detection method based on improved YOLO v5 according to claim 1, characterized in that, In the second step, the initially obtained fireworks data set images are cleaned, and then the data set images meeting the high-altitude angle are converted into jpg format, and then the images are labeled using a labeling tool to form corresponding xml files.

4. The high-altitude fireworks detection method based on improved YOLO v5 according to claim 1, characterized in that, In the third step, the official pre-trained model YOLO v5l is used in the YOLO v5 network, and the YOLO v5 network comprises a backbone network, a Neck and a Head connected in sequence, the backbone network mainly comprises a Focus structure and a CSP structure, which is used to obtain the features of training images or to-be-recognized images, and a standard convolution layer is arranged after each CSP structure, which is replaced by a hybrid attention module composed of a channel attention module and a spatial attention module connected in sequence; The structure of the Neck is FPN+PAN, which performs feature fusion and transmits the calculated features to the Head part; the main body of the Head part is a detector, which performs grid prediction on the feature map, and the step is repeated until the coordinates of the detection box are generated, and finally the CIoU-nms non-maximum suppression method is used to remove redundant target boxes to output the detection result.

5. The high-altitude fireworks detection method based on improved YOLO v5 according to claim 1, characterized in that, In the fourth step, the hyperparameters include the depth and width of the network, learning_rate, batch_size, epoch, and the training data set used.

Images