Smoke and fire target detection method and device for edge calculation

Abstract

The invention provides a smoke and fire target detection method and device for edge calculation. A YOLOv5 model is adopted as a detection model, collected and labeled data are used for training the detection model, then model quantification and model application frames are carried out, and accurate detection of a smoke and fire target is achieved. The detection device comprises a data acquisition device and an edge computing device, the edge computing device comprises a stream pulling module, a decoding module, a reasoning module and a result module, and real-time processing of video streams and accurate detection of smoke and fire targets can be achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and target recognition, and specifically relates to a method and device for detecting fireworks targets for edge computing. Background Art

[0002] With the rapid development of artificial intelligence of things (AIoT) and edge computing technology, artificial intelligence (AI) has been widely used in various scenarios from consumers to enterprises. Due to the sharp increase in the number of connected devices, as well as considerations such as privacy protection, data confidentiality, low latency requirements, and bandwidth limitations, more and more cloud servers need to train deep learning models on edge devices. At the same time, due to the limited resources of edge computing devices, it is difficult to support the deployment of deep neural network models with huge computational workloads. Therefore, under the premise of ensuring the accuracy of model detection, the network model can be lightweight by reasonably reducing the number of parameters of the neural network model to reduce the computational pressure on the edge.

[0003] Fire is a devastating natural disaster that can cause huge damage in a short period of time and seriously threaten the safety of life and property. Real-time, fast and accurate monitoring of fires and reducing losses are of great practical value. In recent years, advances in computer vision have enabled vision-based forest fire detection technology to overcome the shortcomings of traditional sensors. More and more fire detection technologies use artificial intelligence (AI) methods to monitor natural landscapes through cameras, use AI technology to analyze video images, and identify flames and smoke. AI models collaborate with edge computing. By deploying trained algorithm models to lightweight edge devices, deep learning target detection methods can be applied to smart security, smart transportation, and forest fire prevention. In forest fire prevention and fire identification, it is of great significance to use edge computing to optimize algorithm models and improve detection accuracy and efficiency.

[0004] Therefore, how to establish the full-process target detection model deployment from cloud servers to edge computing devices and achieve model lightweighting to improve the processing efficiency of reasoning tasks is a key issue that needs to be urgently solved in this field. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a method and device for detecting fireworks targets for edge computing. The YOLOv5 model is used as the detection model, and the collected and annotated data is used to train it, and then the model is quantized and the model is applied to achieve accurate detection of fireworks targets. The detection device of the present invention includes a data acquisition device and an edge computing device, wherein the edge computing device includes a streaming, decoding, reasoning and result module, which can realize real-time processing of video streams and accurate detection of fireworks targets.

[0006] A method for detecting fireworks targets based on edge computing, characterized by the following steps:

[0007] Step 1, obtaining video data: using VLC software to obtain RTSP video stream, where the video stream is a real monitoring video of the woods and forest scenes;

[0008] Step 2, video decoding: decode the RTSP video stream into video frames through FFmpeg, and extract the picture frames containing smoke and flames from the real woods and forest scene monitoring video;

[0009] Step 3, data labeling: label the smoke and flames contained in each frame of the image to obtain the data label of the image. All images and their labels constitute the training data set;

[0010] Step 4, detection model training: With the training data set as input and the target bounding box and category probability as output, the YOLOv5 detection model is trained under the PyTorch framework to obtain the trained pt detection model file fire_smoke_det.pt;

[0011] Step 5, detection model quantization: First, convert the pt model file fire_smoke_det.pt to ONNX format to obtain the file fire_smoke_det.onnx; then, use the deep learning compiler MLIR to quantize the model, and use the onxx2bmodel tool to convert fire_smoke_det.onnx to the Bmodel file fire_smoke_det.bmodel;

[0012] Step 6, model application: Use the quantized model to detect fireworks images in real-time video to obtain the target bounding box and category probability of smoke and flame detection;

[0013] Step 7, result storage: store the test results.

[0014] Specifically, the labelimg tool is used to label the smoke and flames contained in the image, and the labeling information includes the position of the target box containing the smoke or flame and the target category.

[0015] Specifically, the YOLOv5 model mainly includes a backbone network Backbone, a neck network Neck and a prediction module Prediction, wherein the backbone network Backbone includes a standard convolutional layer, a CSP module and an SSPF module, and uses a deep convolutional neural network to extract features from the input image; the neck network Neck integrates multi-scale features from the backbone network Backbone and performs feature fusion; the prediction module Prediction performs target detection on the feature map processed by the neck network Neck, and outputs the target bounding box and category probability.

[0016] The present invention also provides a firework target detection device for edge computing, which is characterized by comprising: a data acquisition device and an edge computing device, wherein the data acquisition device comprises a monitoring camera with video and photo functions for acquiring monitoring videos; the edge computing device comprises a stream pulling module, a decoding module, an inference module and a result module;

[0017] The stream pulling module uses VLC software to obtain RTSP video stream;

[0018] The decoding module decodes the RTSP video stream into video frames through FFmpeg;

[0019] The inference module is used for data annotation, model training, model quantization and model application;

[0020] The result module is used to store the detection results.

[0021] Specifically, the stream pulling module establishes a connection with the RTSP server through the VLC client to obtain the RTSP video stream, and uses the network communication module to transmit the video stream data.

[0022] Specifically, the decoding module uses hardware acceleration technology to process video frames after decoding the RTSP video stream through FFmpeg.

[0023] The beneficial effects of the present invention are as follows: by training the YOLOv5 detection model with collected and annotated data, a detection model for effectively detecting fireworks targets can be obtained, thereby achieving more accurate fireworks target detection; due to the use of hardware acceleration and lightweight deployment model methods, the device operation is more efficient, and real-time processing of video frames and rapid detection of fireworks targets are achieved, meeting the needs of real-time monitoring; since the result module can locally store inference data, the practicality and data management capabilities of the device are improved; the present invention has high detection accuracy and efficiency, and also reduces delay and bandwidth consumption, has important practical application value, and is particularly suitable for scenarios such as forest fire prevention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of a method for detecting fireworks targets for edge computing of the present invention;

[0025] Figure 2 This is an example diagram of the smoke and fire detection effect of the present invention;

[0026] Figure 3 It is a schematic diagram of a fireworks target detection device for edge computing of the present invention; DETAILED DESCRIPTION

[0027] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0028] The present invention provides a method for detecting fireworks targets based on edge computing, such as Figure 1 As shown, the specific implementation process is as follows:

[0029] Step 1, obtain video data: by accessing the surveillance camera (gun camera), use VLC software to obtain the RTSP video stream. Assume that the streaming address is rtsp: / / admin:password@192.168.220.1 / Streaming / Channels / 101, and extract the data of the woods and forest scenes in real time from the RTSP video stream data.

[0030] Step 2, video decoding: decode the RTSP video stream into video frames through FFmpeg, and extract picture frames containing smoke and flames from real woods and forest scene monitoring videos.

[0031] Step 3, data labeling: label the smoke and flames contained in each frame of the image to obtain the data label of the image. All images and their labels constitute the training data set. Preferably, the labelimg tool is used to label the smoke and flames contained in the image. The labeling information includes the position of the target frame containing the smoke or flame and the target category:

[0032] Images = {P 1 , P 2 , ..., P i , ..., P n} (1)

[0033] Labels = {l 1 , l 2 , ..., l j , ..., l n} (2)

[0034] Among them, Images represents all the images in the dataset, P n Represents a frame of image, Label represents the label corresponding to the image in the data set, and each lj Stored in txt format, that is, the location of the target box and the target category, expressed as:

[0035] l i ={x i ,y i , w i ,h i , class i} (3)

[0036] class i ∈{smoke, flame} (4)

[0037] Among them, the detection box obtained by detecting the i-th frame image, x i is the horizontal coordinate of the center of the box, y i is the center ordinate of the box, w i is the width of the box, h i is the height of the box, class i It is the target category, belonging to one of the categories of flame and smoke.

[0038] Step 4, detection model training: With the training data set as input and the target bounding box and category probability as output, the YOLOv5 detection model is trained under the PyTorch framework to obtain the trained pt detection model file fire_smoke_det.pt.

[0039] The YOLOv5 model mainly includes the backbone network Backbone, the neck network Neck and the prediction module Prediction. The backbone network Backbone includes standard convolutional layers, CSP modules and SSPF modules, and uses a deep convolutional neural network to extract features from the input image; the neck network Neck integrates multi-scale features from the backbone network Backbone and performs feature fusion; the prediction module Prediction performs target detection on the feature map processed by the neck network Neck, and outputs the target bounding box and category probability.

[0040] Step 5, test model quantization: In order to improve the reasoning speed of the model and reduce the amount of model reasoning calculation, first, convert the pt file to ONNX (Open Neural Network Exchange) format. ONNX is an extensible computational graph model that defines the specifications of operators and data types, so that deep learning models can be converted between different frameworks to obtain an onnx file (fire_smoke_det.onnx); then use the deep learning compiler MLIR (Multi-Level Intermediate Representation) to quantize the model, and use the onxx2bmodel tool to convert it into a Bmodel file (fire_smoke_det.bmodel) for model training and application detection at the edge.

[0041] Step 6, model application: Use the quantized model to detect fireworks images in real-time video, obtain the target bounding box and category probability of smoke and flame detection, and realize real-time inference operation of video frames.

[0042] Step 7, result storage: store the test results.

[0043] Figure 2 An example diagram of the detection effect using the method of the present invention is given, including a fireworks target detection frame and its category probability.

[0044] The present invention also provides a firework target detection device for edge computing, such as Figure 3 As shown, it mainly includes: data acquisition equipment and edge computing equipment. Among them, the data acquisition equipment includes a surveillance camera with video and photo functions, which is used to obtain surveillance videos; the edge computing equipment includes a streaming module, a decoding module, an inference module and a result module:

[0045] The stream pulling module is used to obtain the video stream. Specifically, the RTSP video stream is obtained by using the VLC software, including establishing a connection with the RTSP server through the VLC client to obtain the RTSP video stream, using the network communication module to transmit the video stream data, and realizing stable reception of the video stream on the edge device.

[0046] The decoding module is used for video decoding and acceleration. After the edge computing device is powered on, the decoding module is initialized, and the FFmpeg module is used for hardware decoding to decode the RTSP video stream into video frames. The video frames can also be processed using hardware acceleration technology.

[0047] The inference module is mainly used for fire and smoke target detection. The video frames processed by the decoding module are input into the inference module. First, the images are annotated, and then the model is deployed, including model training, model quantization, and model application. The trained fire and smoke detection model is used to implement inference operations on real-time video frames.

[0048] The result module is used to store the detection results of fireworks targets, and can also determine the key frames containing fireworks targets. Monitoring personnel can remotely log in to the IP address of the edge device to view the videos and pictures of the fireworks, and determine the specific location of the fire based on the anchor coordinates, category and camera ID.

[0049] The present invention uses the YOLOv5 model as the basis of the detection method. YOLOv5 is a classic model in the field of target detection. It has a deep network structure and advanced optimization methods. It performs well in the task of detecting fireworks targets and is very suitable for detection and subsequent classification tasks. By deploying a lightweight fireworks target detection model to an edge computing device and utilizing hardware acceleration and optimized reasoning modules, real-time processing of video frames and rapid detection of fireworks targets are achieved to meet the needs of real-time monitoring. At the same time, the result module can locally store reasoning data, including real-time video and key frames, to enhance the practicality and data management capabilities of the system. The present invention not only improves the accuracy and efficiency of detection, but also reduces latency and bandwidth consumption. It has important practical application value and is particularly suitable for scenarios such as forest fire prevention.

Claims

1. A method for detecting fireworks targets based on edge computing, characterized in that Here are the steps: Step 1, obtaining video data: using VLC software to obtain RTSP video stream, where the video stream is a real monitoring video of the woods and forest scenes; Step 2, video decoding: decode the RTSP video stream into video frames through FFmpeg, and extract the picture frames containing smoke and flames from the real woods and forest scene monitoring video; Step 3, data labeling: label the smoke and flames contained in each frame of the image to obtain the data label of the image. All images and their labels constitute the training data set; Step 4, detection model training: With the training data set as input and the target bounding box and category probability as output, the YOLOv5 detection model is trained under the PyTorch framework to obtain the trained pt detection model file fire_smoke_det.pt; Step 5, detection model quantization: First, convert the pt model file fire_smoke_det.pt to ONNX format to obtain the file fire_smoke_det.onnx; then, use the deep learning compiler MLIR to quantize the model, and use the onxx2bmodel tool to convert fire_smoke_det.onnx to the Bmodel file fire_smoke_det.bmodel; Step 6, model application: Use the quantized model to detect fireworks images in real-time video to obtain the target bounding box and category probability of smoke and flame detection; Step 7, result storage: store the test results.

2. The method for detecting fireworks targets based on edge computing according to claim 1, characterized in that: Use the labelimg tool to label the smoke and flames contained in the image. The labeling information includes the location of the target box containing the smoke or flame and the target category.

3. The method for detecting fireworks targets based on edge computing according to claim 1, characterized in that: The YOLOv5 model mainly includes a backbone network Backbone, a neck network Neck and a prediction module Prediction, wherein the backbone network Backbone includes a standard convolutional layer, a CSP module and an SSPF module, and uses a deep convolutional neural network to extract features from the input image; the neck network Neck integrates multi-scale features from the backbone network Backbone and performs feature fusion; the prediction module Prediction performs target detection on the feature map processed by the neck network Neck, and outputs the target bounding box and category probability.

4. A firework target detection device for edge computing, characterized in that include: Data collection equipment and edge computing equipment, wherein the data collection equipment includes a surveillance camera with video and photo functions, which is used to obtain surveillance videos; The edge computing device includes a streaming module, a decoding module, an inference module, and a result module; The stream pulling module uses VLC software to obtain RTSP video stream; The decoding module decodes the RTSP video stream into video frames through FFmpeg; The inference module is used for data annotation, model training, model quantization and model application; The result module is used to store the detection results.

5. The edge computing-oriented fireworks target detection device according to claim 4, characterized in that: The stream pulling module establishes a connection with the RTSP server through the VLC client to obtain the RTSP video stream, and uses the network communication module to transmit the video stream data.

6. The edge computing-oriented fireworks target detection device according to claim 5, characterized in that: The decoding module uses hardware acceleration technology to process video frames after decoding the RTSP video stream through FFmpeg.

Images