Smoke and fire detection method of edge computing embedded platform

Abstract

The invention discloses a firework detection method of an edge computing embedded platform, and relates to the technical field of deep learning firework detection, and the method comprises the following steps: S1, constructing an edge computing embedded hardware platform; s2, screening and labeling the smoke and fire pictures to obtain a to-be-trained smoke and fire data set; s3, building and optimizing a network model; s4, converting the model; s5, deploying the model to a hardware platform of edge calculation; s6, inputting the image into a model reasoning module to obtain a reasoning result, inputting the reasoning result into a post-processing module, analyzing prediction information, and drawing on an original image; and S7, uploading a reasoning result to a server, and sending a signal by the server according to the result. According to the smoke and fire detection method of the edge computing embedded platform, a smoke and fire detection system is deployed on hardware of the edge computing embedded platform, data processing and analysis are carried out more efficiently, and real-time and accurate fire detection and alarm services are provided.

Description

Technical Field

[0001] The present invention relates to the field of deep learning fireworks detection technology, and in particular to a fireworks detection method for an edge computing embedded platform. Background Art

[0002] In recent decades, computer hardware has developed rapidly, and GPU performance has increased from generation to generation, which has promoted the rapid development of deep learning algorithms. Using deep learning for fire detection has obvious advantages over using sensors or traditional image processing methods. It can automatically extract smoke and fire features from a large number of data sets to obtain better smoke and fire detection results. Smoke and fire detection based on deep learning has also become an important means of fire detection.

[0003] This method of firework detection based on deep learning improves the performance of detection, but its computational complexity and demand for hardware computing power will also greatly increase. In actual projects, projects that apply complex neural networks are often deployed on the host side to achieve high-speed computing with high-performance GPUs, and have better performance when processing larger networks. However, the cost is high power consumption and large size, which is not easy to deploy and install, and does not have an advantage in the deployment and implementation of deep learning algorithms.

[0004] With the development of embedded hardware, the computing power of mobile devices has been greatly improved. People are no longer satisfied with deploying fireworks detection on the GPU of large host devices. Instead, they deploy deep neural network models on a variety of embedded hardware to implement fireworks target detection algorithms in actual production and life. This is the significance of algorithm research. Summary of the invention

[0005] The purpose of the present invention is to provide a method for detecting fireworks on an edge computing embedded platform, which has the advantages of high precision, good real-time performance, strong stability, and bandwidth saving, and is suitable for scenarios where multi-node fireworks detection is performed in a large area.

[0006] To achieve the above object, the present invention provides a method for detecting fireworks on an edge computing embedded platform, comprising the following steps:

[0007] Step S1: construct an edge computing embedded hardware platform, including an image acquisition module, a graphics acceleration module, a neural network processing module, and a video encoding module, and integrate them on the same circuit board;

[0008] Step S2: Collect a large number of fireworks images in various scenes, screen the collected fireworks images, and then use the data set to label the smoke and flames to obtain the fireworks data set to be trained;

[0009] Step S3: Build and optimize the network model, and use the model on the GPU server to train model parameters for the fireworks data set to obtain a model weight file that can predict image features and generate bounding boxes and prediction categories;

[0010] Step S4: Model conversion, first perform model quantization, then perform quantization accuracy analysis, optimize the model, and obtain a model weight file that can be loaded on the neural network processor mounted on the edge computing embedded platform;

[0011] Step S5: deploy the model to the edge computing hardware platform and build program modules such as image acquisition, preprocessing, reasoning, and post-processing;

[0012] Step S6: The image of the scene in the monitoring area is obtained through the image acquisition module, and is input into the model inference module to obtain the inference result, and the inference result is input into the post-processing module to parse the predicted bounding box position and category and other information, and the graphics acceleration module is used to draw on the original image;

[0013] Step S7: Upload the inference result to the server, and the server sends a signal based on the result.

[0014] Preferably, in step S4, a hybrid quantization function is used to optimize the model according to the accuracy loss values ​​of different network layers in the model.

[0015] Preferably, hybrid quantization performs quantization and quantization accuracy analysis on the trained fireworks detection model, and modifies the configuration file of hybrid quantization according to the quantization accuracy analysis result, and finally generates a fireworks detection model that can be run on the target embedded platform.

[0016] Preferably, in step S5, a graphics acceleration module is used in the image pre-processing and post-processing parts to perform hardware accelerated drawing.

[0017] Preferably, the model inference in step S6 is to load the fireworks detection model obtained after hybrid quantization into the edge computing embedded hardware platform, obtain the target scene image, and calculate the coordinates and category of the fireworks target in the image.

[0018] Preferably, in step S6, the image is subjected to an optimization process of image defogging and image scaling by a preprocessing module.

[0019] Preferably, in step S7, an abnormal situation is detected, and the embedded end of edge computing sends an alarm signal to the server end, sends the image of the detected fireworks to the server end in real time, and issues an alarm;

[0020] If no abnormality is detected, the embedded end sends a safety signal to the server end.

[0021] Preferably, in step S7, when the edge computing embedded terminal detects abnormal conditions such as fireworks, it will operate the pan-tilt head to aim the fire extinguishing device at the fire source to extinguish the fire.

[0022] Preferably, in step S7, the edge computing embedded end receives instructions from the server end, and the edge computing device encodes the video frames after real-time target detection and pushes the stream to the server end.

[0023] Therefore, the present invention adopts the above-mentioned method for detecting fireworks on an edge computing embedded platform, which has the following beneficial effects:

[0024] (1) The present invention discloses a method for detecting fire and smoke on an edge computing embedded platform. The method adopts the concept of edge computing, moves part of the computing and storage resources from a centralized data center or cloud to the edge of the network, and performs computing and data processing on the source device where the data is generated, so as to reduce data transmission delay, reduce bandwidth consumption, increase data processing speed and enhance privacy protection. The method deploys a fire and smoke detection system on the hardware of the edge computing embedded platform, which can process and analyze data more efficiently and provide real-time and accurate fire detection and alarm services.

[0025] (2) The present invention performs hybrid quantization after model accuracy analysis, that is, different quantization types are assigned according to the different accuracy loss values ​​of different network layers of the smoke and fire detection model, so that when performing model inference on an embedded hardware platform, the inference speed and detection accuracy are balanced, and image processing such as image scaling and drawing detection frames on images are all accelerated by hardware, further improving the real-time performance of smoke and fire detection, thereby ensuring that the smoke and fire detection method of the present invention can meet the dual requirements of real-time and high precision for fire detection in multiple scenarios.

[0026] (3) The present invention not only detects fire in time, but also installs a fire extinguishing device to extinguish the fire in time after the fire is detected.

[0027] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a structural framework diagram of the smoke and fire detection system of the edge computing embedded platform of the present invention;

[0029] Figure 2 A design roadmap for the smoke and fire detection system of the edge computing embedded platform of the present invention;

[0030] Figure 3 This is a step diagram of the method for detecting fireworks on the edge computing embedded platform of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0032] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0033] Example

[0034] See also Figure 1-3 , the present invention provides a method for detecting fireworks on an edge computing embedded platform, comprising the following steps:

[0035] Step S1: Build an edge computing embedded hardware platform, including an image acquisition module, a graphics acceleration module, a neural network processing module, and a video encoding module, and integrate them on the same circuit board.

[0036] Step S2: Collect a large number of fireworks images in various scenes, screen the collected fireworks images, and then use the data set to label the smoke and flames to obtain the fireworks data set to be trained.

[0037] Step S3: Build and optimize the network model, and use the model on the GPU server to train model parameters for the fireworks dataset to obtain a model weight file that can predict image features and generate bounding boxes and prediction categories.

[0038] Step S4: Model conversion. First, perform model quantization to quantize the 32-bit floating-point model trained by the GPU into an 8-bit fixed-point model. Then perform quantization accuracy analysis. Use the hybrid quantization function according to the accuracy loss values ​​of different network layers in the model to optimize the model and obtain a model weight file that can be loaded on the neural network processor on the edge computing embedded platform.

[0039] Hybrid quantization is to quantize and analyze the quantization accuracy of the trained fireworks detection model, and modify the hybrid quantization configuration file according to the quantization accuracy analysis results, and finally generate a fireworks detection model that can run on the target embedded platform.

[0040] Step S5: Deploy the model to the edge computing hardware platform, build program modules such as image acquisition, preprocessing, reasoning, and post-processing, and use the graphics acceleration module for hardware-accelerated drawing in the image preprocessing and post-processing parts.

[0041] Step S6: The image of the scene in the monitoring area is obtained through the image acquisition module. After the image is optimized for defogging and image scaling in the preprocessing module, the acquired image is input into the model inference module to obtain the inference result. The inference result is then input into the post-processing module to parse the predicted bounding box position and category information, and the graphics acceleration module is used to draw it on the original image.

[0042] Model inference is to load the fireworks detection model obtained after hybrid quantization into the edge computing embedded hardware platform, obtain the target scene image, and calculate the coordinates and category of the fireworks target in the image.

[0043] Step S7: Upload the inference result to the server, and the server sends a signal based on the result. If an abnormality (fire) is detected, the embedded end of edge computing sends an alarm signal to the server, pushes the screen of the detected abnormality, and notifies the administrator in time. The embedded end of edge computing also operates the pan-tilt system to aim the fire extinguishing device at the fire source to extinguish the fire. If no abnormality is detected, the embedded end sends a safety signal to the server.

[0044] When the edge computing embedded end detects abnormal conditions such as fireworks, it will operate the pan / tilt to aim the fire extinguishing device at the fire source to extinguish the fire. The edge computing embedded end receives instructions from the server end, and the edge computing device encodes the video frames after real-time target detection and pushes the stream to the server end.

[0045] The video image acquisition module and the smoke and fire detection function of the camera are integrated into an embedded hardware device of edge computing with Rockchip RK3588 artificial intelligence chip as the core. The deep learning model is first loaded on the neural network processor NPU of RK3588, and then the collected image data is directly input into the neural network processor NPU for smoke and fire detection, without transmitting the collected original video to the server through the network, which reduces the computing pressure of the server-side smoke and fire detection algorithm when processing multiple videos. Finally, the result of the smoke and fire detection is reported to the server. The video is only transmitted when the server-side views a certain camera position, which reduces the bandwidth pressure caused by transmitting the original video and reduces the waste of resources. With such embedded hardware on the edge computing end, stable and real-time fire monitoring in a large area can be achieved around the clock. By using the rknn-toolkit2 tool provided by Rockchip to perform hybrid quantization on the model, the data volume of the smoke and fire detection model based on deep learning is compressed while maintaining the smoke and fire detection accuracy, so that the smoke and fire detection speed and detection accuracy are balanced, so that the smoke and fire detection system with RK3588 as the hardware master can meet the dual requirements of real-time and high-precision fire warning.

[0046] A method for detecting fireworks on an edge computing embedded platform, the method steps are as follows:

[0047] Step S1: This embodiment uses Rockchip RK3588's artificial intelligence chip as the core to build an edge computing embedded hardware platform, which is equipped with a high-performance neural network processing module, a high-performance GPU for graphics acceleration, and has powerful video encoding capabilities, and uses a camera as an image acquisition module.

[0048] Step S2: Collect a large number of fireworks images in various scenes, screen the collected fireworks images, and then use LabelImg dataset annotation software to annotate the two types of labels, smoke and flame, to obtain the fireworks dataset to be trained.

[0049] Step S3: Build and optimize the YOLOv5 network model for fireworks detection, use YOLOv5 on the GPU server to train the model parameters of the fireworks dataset prepared above, obtain the pt model weight file that can predict image features and generate bounding boxes and prediction categories, and convert it into an onnx model.

[0050] Step S4: Use the rknn-toolkit2 tool provided by Rockchip to quantize the model, quantize the 32-bit floating-point model trained by GPU into an 8-bit fixed-point model, and then perform quantization accuracy analysis. Use the hybrid quantization function according to the accuracy loss values ​​of different network layers in the model to optimize the model and obtain a rknn model file that can be loaded on the neural network processor on the edge computing embedded platform.

[0051] Step S5: Deploy the above-obtained rknn model to the RK3588 edge computing hardware platform, build program modules such as image acquisition, preprocessing, reasoning, and post-processing, and use the graphics acceleration module for hardware accelerated drawing in the image preprocessing and post-processing parts.

[0052] Step S6: The image of the scene in the monitoring area is obtained through the image acquisition module. After the image is optimized for defogging and image scaling in the preprocessing module, the acquired image is input into the model inference module to obtain the inference result. The inference result is then input into the post-processing module to parse the predicted bounding box position and category information, and the graphics acceleration module is used to draw it on the original image.

[0053] Step S7: Upload the above reasoning results to the server. If an abnormality (fire) is detected, the RK3588 embedded terminal sends an alarm signal to the server, pushes the abnormality detection screen, and notifies the administrator in time. The RK3588 embedded terminal also operates the pan / tilt to aim the fire extinguishing device at the fire source to extinguish the fire. If no abnormality is detected, the RK3588 embedded terminal sends a safety signal to the server.

[0054] Therefore, the present invention adopts the above-mentioned method for detecting fireworks on an edge computing embedded platform. The present invention discloses a method for detecting fireworks on an edge computing embedded platform. The method adopts the edge computing thinking, moves part of the computing and storage resources from a centralized data center or cloud to the edge of the network, performs computing and data processing on the source device where the data is generated, so as to reduce data transmission delay, reduce bandwidth consumption, improve data processing speed and enhance privacy protection. The fireworks detection system is deployed on the hardware of the edge computing embedded platform, which can process and analyze data more efficiently and provide real-time and accurate fire detection and alarm services. The present invention performs hybrid quantization after performing model accuracy analysis, that is, assigns different quantization types according to the different accuracy loss values ​​of different network layers of the fireworks detection model, so that when the model is inferred on the embedded hardware platform, the inference speed and the detection accuracy are balanced, and image processing such as image scaling and drawing a detection frame on the image are all accelerated by hardware, so as to further improve the real-time performance of fireworks detection, thereby ensuring that the fireworks detection method of the present invention can meet the dual requirements of real-time performance and high precision of fire detection in multiple scenarios. The present invention not only detects fire in a timely manner, but also installs a fire extinguishing device to achieve timely fire extinguishing after the fire is detected.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for detecting fireworks on an edge computing embedded platform, characterized in that: The following steps are involved: Step S1: construct an edge computing embedded hardware platform, including an image acquisition module, a graphics acceleration module, a neural network processing module, and a video encoding module, and integrate them on the same circuit board; Step S2: Collect a large number of fireworks images in various scenes, screen the collected fireworks images, and then use the data set to label the smoke and flames to obtain the fireworks data set to be trained; Step S3: Build and optimize the network model, and use the model on the GPU server to train model parameters for the fireworks data set to obtain a model weight file that can predict image features and generate bounding boxes and prediction categories; Step S4: Model conversion, first perform model quantization, then perform quantization accuracy analysis, optimize the model, and obtain a model weight file that can be loaded on the neural network processor mounted on the edge computing embedded platform; Step S5: deploy the model to the edge computing hardware platform and build program modules such as image acquisition, preprocessing, reasoning, and post-processing; Step S6: The image of the scene in the monitoring area is obtained through the image acquisition module, and is input into the model inference module to obtain the inference result, and the inference result is input into the post-processing module to parse the predicted bounding box position and category and other information, and the graphics acceleration module is used to draw on the original image; Step S7: Upload the inference result to the server, and the server sends a signal based on the result.

2. The method for detecting fireworks on an edge computing embedded platform according to claim 1, characterized in that: In step S4, the model is optimized using a hybrid quantization function according to the accuracy loss values ​​of different network layers in the model.

3. The method for detecting smoke and fire on an edge computing embedded platform according to claim 2, characterized in that: Hybrid quantization quantizes the trained fire and smoke detection model and performs quantization accuracy analysis. The configuration file of the hybrid quantization is modified according to the results of the quantization accuracy analysis. Finally, a fire and smoke detection model that can be run on the target embedded platform is generated.

4. The method for detecting smoke and fire of an edge computing embedded platform according to claim 3, characterized in that: In step S5, a graphics acceleration module is used to perform hardware accelerated drawing in the image pre-processing and post-processing parts.

5. The method for detecting smoke and fire of an edge computing embedded platform according to claim 4, characterized in that: In step S6, the model inference is to load the fireworks detection model obtained after hybrid quantization into the edge computing embedded hardware platform, obtain the target scene image, and calculate the coordinates and category of the fireworks target in the image.

6. The method for detecting smoke and fire on an edge computing embedded platform according to claim 5, characterized in that: In step S6, the image is subjected to an optimization process of image defogging and image scaling by a preprocessing module.

7. The method for detecting smoke and fire of an edge computing embedded platform according to claim 6, characterized in that: In step S7, an abnormal situation is detected, and the embedded end of edge computing sends an alarm signal to the server end, sends the image of the detected fireworks to the server end in real time, and issues an alarm; If no abnormality is detected, the embedded end sends a safety signal to the server end.

8. The method for detecting smoke and fire on an edge computing embedded platform according to claim 7, characterized in that: In step S7, when the edge computing embedded terminal detects abnormal conditions such as fireworks, it will operate the pan-tilt head to aim the fire extinguishing device at the fire source to extinguish the fire.

9. The method for detecting smoke and fire on an edge computing embedded platform according to claim 8, characterized in that: In step S7, the edge computing embedded end receives instructions from the server end, and the edge computing device encodes the video frames after real-time target detection and pushes the stream to the server end.

Images