Multi-mode forest fire identification system and method based on unmanned aerial vehicle

Abstract

The invention discloses a multi-mode forest fire identification system and method based on an unmanned aerial vehicle, and belongs to the technical field of fireproof monitoring, and the system comprises the unmanned aerial vehicle, a monitoring center, a user terminal, a path planning unit and a wireless communication unit. Wherein the unmanned aerial vehicle is loaded with a high-definition camera, a position sensor, an infrared sensor and an edge computing server; a multi-modal fire recognition algorithm model runs on the edge server, and the multi-modal fire recognition algorithm model comprises an image recognition algorithm, an infrared data analysis algorithm, a data fusion module and a decision support module; and the processor is used for processing the visible light video data acquired by the high-definition camera and the infrared data acquired by the infrared sensor in real time and outputting an analysis result. According to the invention, the early warning speed and accuracy of a fire disaster can be obviously improved, remote real-time monitoring is realized, false alarms can be reduced, and rapid and accurate fire source positioning is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of fire prevention monitoring, and in particular to a multi-modal forest fire identification system and method based on an unmanned aerial vehicle. Background Art

[0002] Currently, most monitoring systems for forest fires are based on fixed-point monitoring or satellite image analysis. Common problems include monitoring blind spots, difficulty in locating fire sources, and delayed alarms.

[0003] For example, patent number CN102180212A discloses a forest fire monitoring station, which uses fixed-point monitoring and combines meteorological data to judge the fire situation. Although it provides a certain degree of monitoring, its ability to judge fires in dense forest areas is weak, the response time is long, and it cannot meet the needs of rapid dynamic fire prevention.

[0004] Based on this, the present invention designs a multimodal forest fire identification system and method based on drone to solve the above problems. Summary of the invention

[0005] In view of the above-mentioned shortcomings of the prior art, the present invention provides a real-time monitoring system for forest fire prevention with wider monitoring coverage, more accurate fire source positioning and faster response speed, which integrates drone aerial photography and infrared sensing technology to address the limitations of the prior art.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] A multimodal forest fire identification system based on a drone, comprising a drone, a monitoring center, a user terminal, a path planning unit and a wireless communication unit;

[0008] The drone is equipped with a high-definition camera, a position sensor, an infrared sensor and an edge computing server;

[0009] High-definition cameras to collect visible-light video data of the forest;

[0010] Infrared sensors, used to collect infrared data of forests;

[0011] Position sensor, used to obtain the location of the drone;

[0012] The edge server runs a multimodal fire identification algorithm model, which includes an image recognition algorithm, an infrared data analysis algorithm, a data fusion module, and a decision support module; the edge server is used to process the visible light video data collected by the high-definition camera and the infrared data collected by the infrared sensor in real time, and output the analysis results; the visible light video data, the infrared data, and the analysis results are all transmitted to the monitoring center;

[0013] A path planning unit, used to design the flight path of the drone;

[0014] The drone is used to fly according to the flight path designed by the path planning unit and carries a high-definition camera to collect visible light video data in the area, and carries an infrared sensor to collect infrared data;

[0015] After the drone discovers a fire, it stays at the location and monitors the type of fire, visible light video data, and infrared data online and in real time. The current visible light video data and infrared data at the fire location are bound with the fire alarm information and transmitted to the monitoring center.

[0016] A backup UAV is used to start when the UAV detects a fire, fly to the location of the UAV, and then continue to fly along the flight path designed by the path planning unit;

[0017] The monitoring center is used to receive visible light video data, infrared data and analysis results from one or more edge servers in real time, and when the analysis results contain fire alarm information and guidance suggestions, send the fire alarm information and guidance suggestions to the user terminal;

[0018] User terminal, used to view fire alarm information and guidance suggestions.

[0019] Furthermore, the fire alarm information includes the occurrence of the fire and the location of the fire.

[0020] Furthermore, the analysis results are output as follows:

[0021] When there is no fire, the "area is normal" message is output;

[0022] When there is a fire, fire alarm information and guidance suggestions are output; the fire alarm information includes the occurrence of the fire and the location of the fire.

[0023] Furthermore, the edge server communicates with the monitoring center via a wireless communication unit.

[0024] Furthermore, the monitoring center communicates with the user terminal through a wireless communication unit. When the user issues an instruction on the user terminal to view the visible light video data, infrared data and analysis results of a certain time, the monitoring center transmits the visible light video data, infrared data and analysis results of that time to the user terminal.

[0025] Furthermore, image recognition algorithms are used to identify flame and smoke features in visible light video data.

[0026] Furthermore, the infrared data analysis algorithm is used to identify abnormal hot spots in the infrared data.

[0027] Furthermore, the data fusion module is used to fuse the recognition results of the visible light video data and the infrared data to output the fire judgment result;

[0028] The decision support module is used to generate fire alarm information and guidance suggestions when a fire occurs, and send the fire alarm information and guidance suggestions to the monitoring center, and the monitoring center sends the fire alarm information and guidance suggestions to the user terminal.

[0029] Furthermore, the fusion method is:

[0030] (1) Grayscale processing, regional interconnection, binarization and expansion operations are performed on infrared data;

[0031] (2) Perform a difference operation on the grayscale image and the binary image of the infrared data to obtain a suspicious image with scene changes;

[0032] (3) Calculate the similarity of the suspicious images of the previous and next frames to confirm whether flames and smoke exist;

[0033] (4) After flames and smoke appear in the infrared image, determine whether the RGB values ​​on the corresponding visible light video data meet the colors of flames and smoke. If not, determine that there is no fire and output "area normal" information; if so, determine that a fire has occurred;

[0034] In order to better achieve the purpose of the present invention, the present invention also provides a multi-modal forest fire identification method based on a drone, using a multi-modal forest fire identification system based on a drone, comprising the following steps:

[0035] Step 1: The drone flies according to the flight path designed by the path planning unit, and collects visible light video data in the area with a high-definition camera, and collects infrared data with an infrared sensor;

[0036] Step 2: The visible light video data collected by the high-definition camera and the infrared data collected by the infrared sensor are transmitted to the multimodal fire recognition algorithm model, and the analysis results are output:

[0037] When there is no fire, the image recognition algorithm does not identify the flame and smoke features in the visible light video data, and the infrared data analysis algorithm does not identify abnormal hot spots in the infrared data, and outputs the information "area normal";

[0038] When there is a fire, the image recognition algorithm identifies the flame and smoke features in the visible light video data, and the infrared data analysis algorithm identifies abnormal hot spots in the infrared data; the data fusion module fuses the recognition results of the visible light video data and the infrared data, and outputs the fire judgment result; the decision support module generates fire alarm information and guidance suggestions when a fire occurs, and sends the fire alarm information and guidance suggestions to the monitoring center, which sends the fire alarm information and guidance suggestions to the user terminal;

[0039] Step 3: After the UAV discovers a fire, it stays at that location and monitors the type of fire, visible light video data, and infrared data online in real time. The current visible light video data and current infrared data at the fire location are bound with the fire alarm information and transmitted to the monitoring center. At the same time, the backup UAV is started, flies to the location of the above-mentioned UAV, and continues to fly along the flight path designed by the path planning unit.

[0040] Compared with the prior art, the present invention has the following beneficial effects: the present invention can significantly improve the warning speed and accuracy of fires and realize remote real-time monitoring; and it adopts a multimodal fire recognition algorithm model, which can comprehensively analyze video images and infrared data, reduce false alarms, and provide strong support for emergency response by quickly and accurately locating the fire source. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 The block diagram of a multimodal forest fire identification system based on drone of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Embodiment 1: In some embodiments, please refer to the drawings of the specification Figure 1,A multimodal forest fire identification system based on UAV, includes a UAV, a monitoring center, a user terminal, a path planning unit, and a wireless communication unit;

[0045] The drone is equipped with a high-definition camera, a position sensor, an infrared sensor and an edge computing server; the high-definition camera is used to collect visible light video data of the forest; the infrared sensor is used to collect infrared data of the forest; the position sensor is used to obtain the location of the drone; the edge server runs a multimodal fire recognition algorithm model, which is used to process the visible light video data collected by the high-definition camera and the infrared data collected by the infrared sensor in real time, run the fire source recognition algorithm in real time, analyze the visible light video data and the infrared data, and output the analysis results; the visible light video data, the infrared data and the analysis results are all transmitted to the monitoring center;

[0046] The analysis results are as follows:

[0047] When there is no fire, the "area is normal" message is output;

[0048] When there is a fire, output fire alarm information and guidance suggestions; fire alarm information includes fire occurrence and fire location;

[0049] The edge server communicates with the monitoring center via a wireless communication unit;

[0050] The multimodal fire identification algorithm model can comprehensively analyze video images and infrared data, accurately identify the fire source when a forest fire occurs, and quickly and accurately locate the fire source to improve the accuracy and real-time performance of fire identification.

[0051] A path planning unit, used to design the flight path of the drone;

[0052] The drone is used to fly according to the flight path designed by the path planning unit and carries a high-definition camera to collect visible light video data in the area, and carries an infrared sensor to collect infrared data;

[0053] After the drone discovers a fire, it stays at the location and monitors the type of fire, visible light video data, and infrared data online and in real time. The current visible light video data and infrared data at the fire location are bound with the fire alarm information and transmitted to the monitoring center.

[0054] A backup UAV is used to start when the UAV detects a fire, fly to the location of the UAV, and then continue to fly along the flight path designed by the path planning unit;

[0055] Similarly, if the backup drone also finds a fire at another location beyond the set fire range threshold, the drone will stay at that location, monitor the type of fire, visible light video data, and infrared data in real time online, transmit them to the monitoring center, and start the next backup drone. The number of backup drones can be designed according to actual conditions.

[0056] The forest can be divided into multiple small grids, each of which corresponds to the maximum monitoring range of the drone, and this maximum monitoring range can be set as the fire range threshold;

[0057] The monitoring center is used to receive visible light video data, infrared data and analysis results from one or more edge servers in real time. When the analysis results contain fire alarm information and guidance suggestions, the fire alarm information and guidance suggestions are sent to the user terminal for timely response;

[0058] On the one hand, the monitoring center can see the current visible light video data, current infrared data and fire alarm information at the fire site; on the other hand, it can see the monitoring status of the backup drone.

[0059] User terminals are used to view fire alarm information and guidance suggestions so that users can take timely actions.

[0060] Preferably, the monitoring center communicates with the user terminal through a wireless communication unit. When the user issues an instruction to view the visible light video data, infrared data and analysis results of a certain time at the user terminal, the monitoring center transmits the visible light video data, infrared data and analysis results of the time to the user terminal;

[0061] Therefore, in addition to receiving timely fire alarm information and guidance suggestions, users can also view visible light video data, infrared data and analysis results at a specific time.

[0062] The multimodal fire recognition algorithm model includes an image recognition algorithm, an infrared data analysis algorithm, a data fusion module and a decision support module, and realizes rapid and accurate recognition of fire by simultaneously processing and analyzing visible light video data collected by a high-definition camera and infrared data collected by an infrared sensor.

[0063] Image recognition algorithms, using deep learning techniques, trained to identify fire and smoke features in visible light video data;

[0064] Infrared data analysis algorithms to identify abnormal hot spots in infrared data to assist in confirming the location and intensity of fires;

[0065] The data fusion module is used to fuse the recognition results of visible light video data and infrared data and output the fire judgment results to improve the accuracy and reliability of fire recognition;

[0066] Specifically, the fusion method is:

[0067] (1) Grayscale processing, regional interconnection, binarization and expansion operations are performed on infrared data;

[0068] (2) Perform a difference operation on the grayscale image and the binary image of the infrared data to obtain a suspicious image with scene changes;

[0069] (3) Calculate the similarity of the suspicious images of the previous and next frames to confirm whether flames and smoke exist;

[0070] (4) After flames and smoke appear in the infrared image, determine whether the RGB values ​​on the corresponding visible light image meet the colors of flames and smoke. If not, it is determined that there is no fire and the "area is normal" information is output; if so, it is determined that a fire has occurred;

[0071] The decision support module is used to generate fire alarm information and guidance suggestions when a fire occurs, and send the fire alarm information and guidance suggestions to the monitoring center, and the monitoring center sends the fire alarm information and guidance suggestions to the user terminal.

[0072] Fire alarm information includes fire occurrence and fire location.

[0073] The present invention can significantly improve the speed and accuracy of fire warning and realize remote real-time monitoring. It also adopts a multimodal fire recognition algorithm model, which can comprehensively analyze video images and infrared data, reduce false alarms, and provide strong support for emergency response by quickly and accurately locating the fire source.

[0074] Embodiment 2: In some embodiments, a multimodal forest fire identification method based on a drone comprises the following steps:

[0075] 1. The drone flies according to the flight path designed by the path planning unit, and collects visible light video data in the area with a high-definition camera, and collects infrared data with an infrared sensor;

[0076] 2. The visible light video data collected by the high-definition camera and the infrared data collected by the infrared sensor are transmitted to the multimodal fire recognition algorithm model, and the analysis results are output:

[0077] When there is no fire, the image recognition algorithm does not identify the flame and smoke features in the visible light video data, and the infrared data analysis algorithm does not identify abnormal hot spots in the infrared data, and outputs the information "area normal";

[0078] When there is a fire, the image recognition algorithm identifies the flame and smoke features in the visible light video data, and the infrared data analysis algorithm identifies abnormal hot spots in the infrared data; the data fusion module fuses the recognition results of the visible light video data and the infrared data, and outputs the fire judgment result; the decision support module generates fire alarm information and guidance suggestions when a fire occurs, and sends the fire alarm information and guidance suggestions to the monitoring center, which sends the fire alarm information and guidance suggestions to the user terminal;

[0079] 3. After the UAV discovers a fire, it stays at that location and monitors the type of fire, visible light video data, and infrared data online in real time. The current visible light video data and current infrared data at the fire location are bound with the fire alarm information and transmitted to the monitoring center. At the same time, the backup UAV is started, flies to the location of the above-mentioned UAV, and continues to fly along the flight path designed by the path planning unit.

[0080] The present invention can significantly improve the speed and accuracy of fire warning and realize remote real-time monitoring. It also adopts a multimodal fire recognition algorithm model, which can comprehensively analyze video images and infrared data, reduce false alarms, and provide strong support for emergency response by quickly and accurately locating the fire source.

[0081] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal forest fire identification system based on drones, characterized in that: It includes drones, monitoring centers, user terminals, path planning units, and wireless communication units; The drone is equipped with a high-definition camera, a position sensor, an infrared sensor and an edge computing server; High-definition cameras to collect visible-light video data of the forest; Infrared sensors, used to collect infrared data of forests; Position sensor, used to obtain the location of the drone; The edge server runs a multimodal fire identification algorithm model, which includes an image recognition algorithm, an infrared data analysis algorithm, a data fusion module, and a decision support module; the edge server is used to process the visible light video data collected by the high-definition camera and the infrared data collected by the infrared sensor in real time, and output the analysis results; the visible light video data, the infrared data, and the analysis results are all transmitted to the monitoring center; A path planning unit, used to design the flight path of the drone; The drone is used to fly according to the flight path designed by the path planning unit and carries a high-definition camera to collect visible light video data in the area, and carries an infrared sensor to collect infrared data; After the drone discovers a fire, it stays at the location and monitors the type of fire, visible light video data, and infrared data online and in real time. The current visible light video data and infrared data at the fire location are bound with the fire alarm information and transmitted to the monitoring center. A backup UAV is used to start when the UAV detects a fire, fly to the location of the UAV, and then continue to fly along the flight path designed by the path planning unit; The monitoring center is used to receive visible light video data, infrared data and analysis results from one or more edge servers in real time, and when the analysis results contain fire alarm information and guidance suggestions, send the fire alarm information and guidance suggestions to the user terminal; User terminal, used to view fire alarm information and guidance suggestions.

2. The multimodal forest fire identification system based on drone according to claim 1 is characterized in that: Fire alarm information includes fire occurrence and fire location.

3. The multimodal forest fire identification system based on drone according to claim 2 is characterized in that: Output analysis results, specifically: When there is no fire, the "area is normal" message is output; When there is a fire, fire alarm information and guidance suggestions are output; the fire alarm information includes the occurrence of the fire and the location of the fire.

4. The multimodal forest fire identification system based on drone according to claim 3 is characterized in that: The edge server communicates with the monitoring center through a wireless communication unit.

5. The multimodal forest fire identification system based on drone according to claim 4 is characterized in that: The monitoring center communicates with the user terminal through a wireless communication unit. When the user issues an instruction on the user terminal to view the visible light video data, infrared data and analysis results of a specific time, the monitoring center transmits the visible light video data, infrared data and analysis results of that time to the user terminal.

6. The multimodal forest fire identification system based on drone according to claim 5 is characterized in that: Image recognition algorithms for identifying fire and smoke features in visible light video data.

7. The multimodal forest fire identification system based on drone according to claim 6 is characterized in that: Infrared data analysis algorithm to identify abnormal hot spots in infrared data.

8. The multimodal forest fire identification system based on drone according to claim 7 is characterized in that: The data fusion module is used to fuse the recognition results of visible light video data and infrared data and output the fire judgment result; The decision support module is used to generate fire alarm information and guidance suggestions when a fire occurs, and send the fire alarm information and guidance suggestions to the monitoring center, and the monitoring center sends the fire alarm information and guidance suggestions to the user terminal.

9. The multimodal forest fire identification system based on drone according to claim 8, characterized in that: The fusion method is: (1) Grayscale processing, regional interconnection, binarization and expansion operations are performed on infrared data; (2) Perform a difference operation on the grayscale image and the binary image of the infrared data to obtain a suspicious image with scene changes; (3) Calculate the similarity of the suspicious images of the previous and next frames to confirm whether flames and smoke exist; (4) After flames and smoke appear in the infrared image, determine whether the RGB values ​​on the corresponding visible light video data meet the colors of flames and smoke. If not, it is determined that there is no fire and the "area is normal" information is output; if so, it is determined that a fire has occurred.

10. A multimodal forest fire identification method based on drone, using the multimodal forest fire identification system based on drone according to claim 9, characterized in that: The following steps are involved: Step 1: The drone flies according to the flight path designed by the path planning unit, and collects visible light video data in the area with a high-definition camera, and collects infrared data with an infrared sensor; Step 2: The visible light video data collected by the high-definition camera and the infrared data collected by the infrared sensor are transmitted to the multimodal fire recognition algorithm model, and the analysis results are output: When there is no fire, the image recognition algorithm does not recognize the flame and smoke features in the visible light video data, and the infrared data analysis algorithm does not recognize abnormal hot spots in the infrared data, and outputs the information "area normal"; When there is a fire, the image recognition algorithm identifies the flame and smoke features in the visible light video data, and the infrared data analysis algorithm identifies abnormal hot spots in the infrared data; The data fusion module fuses the recognition results of visible light video data and infrared data and outputs the fire judgment result; the decision support module generates fire alarm information and guidance suggestions when a fire occurs, and sends the fire alarm information and guidance suggestions to the monitoring center, which sends the fire alarm information and guidance suggestions to the user terminal; Step 3: After the UAV discovers the fire, it stays at the location and monitors the type of fire, visible light video data, and infrared data online in real time. The current visible light video data and current infrared data at the fire location are bound with the fire alarm information and transmitted to the monitoring center. At the same time, the backup UAV is started, flies to the location of the above-mentioned UAV, and continues to fly along the flight path designed by the path planning unit.

Images