Early forest fire smoke detection system and method

Abstract

The invention discloses an early forest fire smoke automatic detection system and method, and belongs to the field of forest fire monitoring and early warning. The system comprises a YOLOv7-Smokey small-range smoke detection network, a PyTorch framework, a TensorRT technology and an NVIDIA Jetson Nano embedded platform, and is characterized in that the YOLOv7-Smokey small-range smoke detection network is connected with the TensorRT technology through the NVIDIA Jetson Nano embedded platform. The method comprises the following steps: S1, carrying out image data collection and data enhancement on early forest fire smoke through a network and public data; s2, making an early forest fire smoke detection data set; s3, a YOLOv7-Smokey backbone network is designed, and training is carried out; s4, deploying the YOLOv7-Smokey network model to an NVIDIA (Noise Visual Identity Architecture) Jetson Nano embedded platform; s5, performing reasoning acceleration processing on the YOLOv7-Smokey network by using a TensorRT technology; s6, performing real-time early forest fire smoke detection on an NVIDIA Jetson Nano platform by using a network camera; and S7, transmitting a reasoning result to a server in a video stream pushing mode. The method is used for deploying an early forest fire smoke detection system on an NVIDIA Jetson Nano embedded platform, the real-time performance of detection is ensured while the detection precision is ensured, the operation consumption is reduced, and marginalized intelligent detection is realized.

Description

Technical Field

[0001] The present invention belongs to the field of forest fire smoke monitoring and early warning, and specifically relates to a real-time early forest fire smoke detection system and method based on a YOLOv7-Smokey network and an edge embedded AI platform. Background Art

[0002] Forests are precious resources for human survival. They have many ecological functions such as conserving water resources, maintaining water and soil, improving the environment, purifying the air, and regulating the climate. Forests play a vital role in the balance of the ecological environment. However, forest fires cause serious damage to forests. Once a forest fire occurs, not only will the forest resources in the forest be destroyed, but it will also seriously affect the living environment of wild animals, destroy biological diversity and the ecological balance of the forest, and pose a great threat to the safety of people's lives and property. In order to minimize the damage and losses caused by forest fires, it is particularly important to detect forest fires in a timely and accurate manner. Smoke is usually a feature produced before a forest fire occurs. Therefore, by detecting smoke, early forest fire alarms can be effectively issued. However, existing detection methods generally have the disadvantages of low detection performance for small smoke target images and videos of early fire scenes taken at a long distance, and the use of back-end servers for centralized detection requires a large amount of electricity and network resources to transmit monitoring videos. It is difficult to meet the demand for real-time monitoring of forest fire smoke in remote mountainous areas.

[0003] The concept of edge computing provides a possible solution to the above problems, which is conducive to the realization of edge-based early forest fire smoke monitoring at the video acquisition end. However, due to the huge number of parameters of the convolutional neural network itself, it is difficult to deploy it on edge embedded platforms and mobile devices. Therefore, running the early forest fire smoke detection model based on convolutional neural network on the edge embedded platform has become an urgent problem to be solved. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The present invention overcomes the shortcomings of the existing technologies and proposes an early forest fire smoke detection network YOLOv7-Smokey which is easy to implement, highly practical and can be deployed on an edge embedded platform. It can realize accurate and real-time detection of early forest fire smoke on the edge while ensuring the detection accuracy.

[0006] (II) Technical solution

[0007] The technical solution adopted by the present invention is:

[0008] An early forest fire smoke detection system, the system comprising:

[0009] YOLOv7-Smokey network for detecting early forest fire smoke;

[0010] PyTorch framework, used to build and train the YOLOv7-Smokey network;

[0011] TensorRT technology, used to accelerate the reasoning of the YOLOv7-Smokey network;

[0012] NVIDIA Jetson Nano embedded platform, used to deploy and run the YOLOv7-Smokey network;

[0013] A webcam for capturing real-time video stream and inputting it into the NVIDIA Jetson Nano embedded platform;

[0014] The video stream push module is used to transmit the inference results to the server.

[0015] Preferably, the YOLOv7-Smokey network includes:

[0016] A network structure optimized for small target detection, the network structure comprising a pruned detection layer and a feature extraction layer;

[0017] Anchors recalculated by K-means method to adapt to the characteristics of early forest fire smoke detection.

[0018] Preferably, the system further comprises:

[0019] The data enhancement module is used to adjust the brightness, contrast, and saturation of the collected image data, as well as perform cropping, scaling, position transformation, and noise addition operations;

[0020] The DCGAN network is used for unsupervised learning to enhance image data and generates new samples through the game between the generator and the discriminator.

[0021] An early forest fire smoke detection method comprises the following steps:

[0022] Step 1: Collect image data of early forest fire smoke through the Internet and public data and perform data enhancement;

[0023] Step 2: Create an early forest fire smoke detection dataset;

[0024] Step 3: Design and train the YOLOv7-Smokey backbone network model;

[0025] Step 4: Deploy the YOLOv7-Smokey backbone network model obtained in step 3 to the NVIDIA Jetson Nano embedded platform;

[0026] Step 5: Use TensorRT technology to accelerate the reasoning of the YOLOv7-Smokey network;

[0027] Step 6: Real-time early forest fire smoke detection on the NVIDIA Jetson Nano platform using a webcam;

[0028] Step 7: The inference results are transmitted to the server through video streaming.

[0029] Preferably, the data enhancement in step one includes enhancing the image data using a DCGAN network.

[0030] Preferably, the step three specifically includes the following contents:

[0031] Prune the detection layer of the YOLOv7 network, delete the original 101-layer detection head, and only keep the L17 and L20 detection heads;

[0032] The feature extraction of 24 layers and 37 layers was modified to 11 layers and 24 layers respectively, and the feature information of the lower layers was used to expand the features of small target smoke;

[0033] The obtained features are concat-fused with the up-sampling results, and finally two detection heads, L17 and L20, are used for early forest fire smoke detection.

[0034] Recalculate Anchors.

[0035] Preferably, the recalculating Anchors specifically includes:

[0036] Use coordinate transformation to convert the height and width of the anchor box into a ratio relative to the length of the grid;

[0037] Use the K-means method to re-cluster the anchors of forest fire smoke data.

[0038] A computer program product includes computer program code for implementing the above-mentioned early forest fire smoke detection method, and the computer program product is stored on a computer-usable storage medium.

[0039] A computer program product includes computer program code for implementing the above-mentioned early forest fire smoke detection system, and the computer program product is stored on a computer-usable storage medium.

[0040] (III) Beneficial effects

[0041] The present invention designs the YOLOv7-Smokey early forest fire smoke detection model for the real-time requirements of edge embedded platform detection, and accelerates the inference of this model through TensorRT, and then deploys it on NVIDIA JetsonNano. At the same time, based on the proposed system, an early forest fire smoke detection method is further proposed. Through the present invention, the marginalization of forest fire smoke detection can be achieved, the computing speed can be improved, and there is no loss of accuracy, which reduces the computing consumption while ensuring the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of the method of the present invention;

[0043] Figure 2 This is the network structure diagram of YOLOv7-Smokey of the present invention;

[0044] Figure 3 Flow chart of the DCGAN network of the present invention;

[0045] Figure 4 This is the DCGAN network loss curve of the present invention;

[0046] Figure 5 A sample schematic diagram for generating a DCGAN network of the present invention;

[0047] Figure 6 This is a flow chart of the K-means algorithm of the present invention;

[0048] Figure 7 This is a diagram of the YOLOv7-Smokey visualization training process of the present invention;

[0049] Figure 8 This is a video streaming effect diagram of the present invention. DETAILED DESCRIPTION

[0050] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0051] First, the nouns mentioned in the present invention are explained:

[0052] NVIDIA Jetson Nano: An embedded AI platform equipped with a quad-core Cortex-A57 processing chip, including 4GB of LPDDR memory and a 128-core Maxwell GPU, it can provide 472 GFLOPS of computing performance and run a variety of algorithms and AI frameworks.

[0053] TensorRT: A high-performance deep learning inference optimizer that provides low-latency, high-throughput deployment inference for deep learning applications.

[0054] PyTorch: It is the Python version of Torch, an open source neural network framework developed by Meta, which is specifically designed for GPU-accelerated deep neural network programming. Torch is a classic tensor library that operates on multi-dimensional matrix data and is widely used in machine learning and other math-intensive applications.

[0055] The present invention proposes an early forest fire smoke detection system, which includes a YOLOv7-Smokey network, a PyTorch framework, a TensorRT technology, and an NVIDIA Jetson Nano embedded platform.

[0056] The present invention also provides an early forest fire smoke detection method based on the above system, please refer to Figure 1 The detection method uses YOLOv7 as the target detection model. In order to meet the lightweight requirements of the marginal detection model, the present invention prunes and modifies the YOLOv7 model structure and proposes the YOLOv7-Smokey model, whose structure diagram is shown in the figure. Figure 2 As shown. Specifically, since early forest fire smoke is mostly small targets, the detection layer in YOLOv7 is pruned, the detection head using the original 101 layer (L23) detection is deleted, and the 24 layer (L6) and 37 layer (L8) feature extraction are modified to 11 layers (L4) and 24 layers (L6) respectively, so that the features of small target smoke are expanded, and the obtained features are concat fused with the results after upsampling (UP) (L13, L16), and finally the two detection heads of L17 and L20 are used for early forest fire smoke detection, making the model lighter and more specific; at the same time, the K-means method is used to recalculate the anchors of the L17 and L20 detection heads to make the anchors more suitable for the characteristics of early forest fire smoke detection. The above process is built and trained based on the PyTorch framework in the server. The entire training process uses TensorRT technology to compress the model to achieve lightweight parameters in the reasoning process, realize lightweight compression of the network and deploy it on the NVIDIA Jetson Nano edge embedded platform. The reasoning results are transmitted back to the server through video streaming to achieve real-time detection of early forest fire smoke.

[0057] The following is an explanation of an early forest fire smoke detection system and method proposed by the present invention in conjunction with relevant drawings and specific examples. The specific contents are as follows.

[0058] Embodiment 1:

[0059] The present invention proposes an early forest fire smoke detection system, which adopts the detection model YOLOv7-Smokey, which is built and trained based on the PyTorch framework. In order to highlight the beneficial effects of the present invention, the TensorRT technology is used to accelerate the reasoning of the above model, and it is deployed on the NVIDIA JetsonNano edge embedded platform. The real-time reasoning results can be transmitted back to the server side by video streaming. The specific implementation steps of the present invention are:

[0060] Step 1: Collect image data of early forest fire smoke through the Internet and public data and perform data enhancement.

[0061] Step 1-1: Use general enhancement techniques to enhance the collected image data, including adjusting brightness, contrast, saturation, and performing data enhancement processing by cropping, scaling, position transformation, and adding noise.

[0062] Step 1-2: Use the DCGAN network to enhance the image data obtained in step 1-1. Different from traditional data enhancement methods, DCGAN is an unsupervised learning method that can generate new samples through the mutual game between two networks. The network flow chart is as follows: Figure 3 Its objective function is defined as:

[0063]

[0064] Among them, G is the generator, D is the discriminator, x is the real data, P data(x) is the distribution of real data, z is the generated data, P z(z) is the distribution for generating data.

[0065] The essence of DCGAN is a game problem of maximum and minimum. Its ultimate goal is to maximize the data output by the generator, and it is difficult for the discriminator to judge its authenticity, that is, to maximize the ability of G to generate data and minimize the probability of D judging the output data as forged.

[0066] The image data obtained in step 1-1 is used as the input of the DCGAN network, and the initial learning rate is set to 2×10 -4 , the batch size is 64, and the training is performed for 2300 epochs. Then the trained DCGAN network is used to expand and enhance the image data. The loss curves of the discriminator D and the generator G during the training process are as follows: Figure 4 As shown, the generated samples are Figure 5 shown.

[0067] Step 2: Process the early forest fire smoke detection dataset.

[0068] Step 2-1: Classify and organize the early forest fire smoke detection dataset obtained in step 1 and divide it into training set, test set and validation set.

[0069] Step 2-2: Use the LabelImg tool to manually annotate the image data obtained by preprocessing in step 2-1, and generate an xml annotation file corresponding to each image. The file contains the manually annotated smoke true value frame coordinate information and the corresponding label information.

[0070] Step 3: Design the YOLOv7-Smokey backbone network and train it.

[0071] Step 3-1: Network structure design. In view of the characteristics of small targets in early forest fire smoke, the detection layer of the YOLOv7 network is pruned, the original 101-layer (L23) detection head is deleted, and only the detection heads of the L17 and L20 layers are retained; and the 24-layer (L6) and 37-layer (L8) feature extraction are modified to 11 layers (L4) and 24 layers (L6) respectively, using the feature information of the lower layer to expand the features of the small target smoke. The network structure is as follows: Figure 2 shown.

[0072] Step 3-2: Recalculate Anchors.

[0073] Step 3-2.1: Coordinate conversion. Since CNN is invariant to translation transformations, and the position of the anchor box is fixed by each grid, it is only necessary to use the K-means method to calculate the width and height of the anchor box. Since the height and width of the Anchors calculated using the width and height of the annotation file are scaled relative to the entire image, when YOLOv7-Smokey uses the anchor box to predict the Bounding Box, the position coordinates are proportional to the length of the grid. Therefore, the height and width of the anchor box need to be converted to a ratio relative to the length of the grid. The conversion formula is as follows:

[0074]

[0075] Using Euclidean distance will make small boxes produce less error values ​​than large boxes. If you want to use anchor boxes to get better IoU indicators, and the IoU indicator is not related to the size of the box, you need to define a new distance formula as follows:

[0076] D(box,centroid)=1-IoU(box,centroid)

[0077] When calculating Anchors, the center coordinates of all boxes are set to 0, which ensures that all boxes are located in the same position. The similarity between boxes can be calculated using the new distance calculation method.

[0078] Step 3-2.2: Use K-means to calculate. K-means is a partitioning clustering method that uses a specially specified distance as an indicator of clustering effect. The closer the distance between two objects, the greater the similarity between them for the K-means clustering method. The flowchart of the algorithm is as follows: Figure 6 As shown, the execution process is as follows:

[0079] 1) Given some training samples {x (1) ,x (2) ,…,x (n)}, for each training sample there is x (i) ∈R n , that is, an n-dimensional space vector;

[0080] 2) Randomly initialize and select k cluster centroids μ1, μ2, …, μ k ∈R n ;

[0081] 3) Repeat the following process until convergence:

[0082]

[0083] For each sample i, use the following formula to calculate the class it should belong to:

[0084]

[0085] For each class μ j , recalculate the centroid of the class.

[0086] Among them, k is the number of clusters required, c (i) Indicates the class that is closest to class i and class k, c (i) The value of is any one from 1 to k. Through the above method, the centroid can be found to no longer change, that is, the algorithm converges and the clustering effect is obtained.

[0087] The present invention uses the K-means method to re-cluster the Anchors of the forest fire smoke data obtained in step 2, iterates 21 times in total, and obtains the final convergence results, L17[18,26,27,56,39,36], L20[49,72,63,138,96,88].

[0088] Step 3-3: Use the image data annotated in step 2 to train the network designed in steps 3-1 and 3-2. Set the initial learning rate to 0.01, use the Adam optimizer, set the batch size to 8, and train for 500 epochs. The visual training process is as follows: Figure 7 As shown; the training program is designed based on Python, the operating system is Windows 11 64bit, and the workstation used to train the model uses NVIDIA RTX 2080Ti GPU, which has 4352 CUDA cores to accelerate the deep learning training process.

[0089] Step 4: Deploy the YOLOv7-Smokey network model to the NVIDIA Jetson Nano embedded platform.

[0090] Step 5: Use TensorRT technology to accelerate the inference processing of the YOLOv7-Smokey network. TensorRT is a high-performance deep learning inference optimizer, which can be regarded as a deep learning framework with only forward propagation. It can parse and optimize the existing network model, including eliminating invalid layers and operations, operation fusion, inter-layer fusion, and weight quantization.

[0091] Step 6: Real-time Early Forest Fire Smoke Detection on NVIDIA Jetson Nano Platform Using Webcam.

[0092] Step 7: The inference result is transmitted to the server by means of video stream push. The present invention realizes the real-time detection of video stream HTTP service by network camera, and the effect is as follows: Figure 8 shown.

[0093] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0095] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. An early forest fire smoke detection system, characterized in that: The system comprises: YOLOv7-Smokey network for detecting early forest fire smoke; PyTorch framework, used to build and train the YOLOv7-Smokey network; TensorRT technology, used to accelerate the reasoning of the YOLOv7-Smokey network; NVIDIA Jetson Nano embedded platform, used to deploy and run the YOLOv7-Smokey network; A webcam for capturing real-time video stream and inputting it into the NVIDIA Jetson Nano embedded platform; The video stream push module is used to transmit the inference results to the server.

2. The early forest fire smoke detection system according to claim 1, characterized in that: The YOLOv7-Smokey network includes: A network structure optimized for small target detection, the network structure comprising a pruned detection layer and a feature extraction layer; Anchors recalculated by K-means method to adapt to the characteristics of early forest fire smoke detection.

3. The early forest fire smoke detection system according to claim 1 or 2, characterized in that: The system further comprises: The data enhancement module is used to adjust the brightness, contrast, and saturation of the collected image data, as well as perform cropping, scaling, position transformation, and noise addition operations; The DCGAN network is used for unsupervised learning to enhance image data and generates new samples through the game between the generator and the discriminator.

4. An early forest fire smoke detection method based on the system according to any one of claims 1 to 3, characterized in that: The following steps are involved: Step 1: Collect image data of early forest fire smoke through the Internet and public data and perform data enhancement; Step 2: Create an early forest fire smoke detection dataset; Step 3: Design and train the YOLOv7-Smokey backbone network model; Step 4: Deploy the YOLOv7-Smokey backbone network model obtained in step 3 to the NVIDIA Jetson Nano embedded platform; Step 5: Use TensorRT technology to accelerate the reasoning of the YOLOv7-Smokey network; Step 6: Real-time early forest fire smoke detection on the NVIDIA Jetson Nano platform using a webcam; Step 7: The inference results are transmitted to the server through video streaming.

5. The early forest fire smoke detection method according to claim 4, characterized in that: The data enhancement described in step 1 includes enhancing the image data using a DCGAN network.

6. The early forest fire smoke detection method according to claim 4, characterized in that: The step three specifically includes the following contents: Prune the detection layer of the YOLOv7 network, delete the original 101-layer detection head, and only keep the L17 and L20 detection heads; The feature extraction of 24 layers and 37 layers was modified to 11 layers and 24 layers respectively, and the feature information of the lower layers was used to expand the features of small target smoke; The obtained features are concat-fused with the up-sampling results, and finally two detection heads, L17 and L20, are used for early forest fire smoke detection. Recalculate Anchors.

7. The early forest fire smoke detection method according to claim 6, characterized in that: The recalculation of Anchors specifically includes: Use coordinate transformation to convert the height and width of the anchor box into a ratio relative to the length of the grid; Use the K-means method to re-cluster the anchors of forest fire smoke data.

8. A computer program product, characterized in that The computer program product comprises a computer program code for implementing the method according to any one of claims 4 to 7, and the computer program product is stored on a computer-usable storage medium.

9. A computer program product, characterized in that The computer program product comprises a computer program code for implementing the system according to any one of claims 1 to 3, and the computer program product is stored on a computer-usable storage medium.

Images