GNSSINS visual combination fire detection method, system and equipment based on unmanned aerial vehicle, and medium

Abstract

The invention discloses an unmanned aerial vehicle-mounted GNSSINS visual combination fire detection method, system and device and a medium, belongs to the technical field of unmanned aerial vehicles, and aims to solve the technical problem of how to improve the positioning precision and fire recognition capability of an unmanned aerial vehicle in a complex environment. Setting a dynamic temperature threshold value, and calibrating the threshold value in real time in combination with the environment temperature; identifying a fire source diffusion trend by adopting a temperature gradient analysis method, and excluding isolated high-temperature points; an image is collected through an RGB camera and converted into an HSV color space, flame features are extracted, and the flame shape is detected in real time through a YOLOv5 model; detecting a smoke area based on a U-Net segmentation network, outputting a binary mask, carrying out space-time alignment on the binary mask and the wind speed and wind direction of the meteorological data, and predicting the fire spreading direction; thermal imaging, visible light and smoke detection results are fused through the D-S evidence theory, and when the confidence coefficients of the thermal imaging, visible light and smoke detection results are all larger than 80%, it is judged that a fire exists.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and specifically to a GNSS-INS vision combined fire detection method, system, device and medium based on an unmanned aerial vehicle. Background Art

[0002] With the improvement of environmental protection awareness and the development of technology, the monitoring and early warning of forest fires have become an important topic for environmental safety. The existing fire monitoring means mainly include the following methods:

[0003] ① Satellite remote sensing technology: Satellite remote sensing can provide large-scale and real-time forest fire monitoring data, but its positioning accuracy is greatly affected by satellite signal interference. Especially in complex weather conditions, the satellite may lose the signal, resulting in monitoring failure.

[0004] ② Fixed sensor system: Some forest fire monitoring systems rely on ground fixed sensors for monitoring, but this method cannot overcome terrain obstacles and has a limited coverage area, and cannot achieve fast and flexible fire detection.

[0005] ③ UAV application: In recent years, as a flexible aerial platform, unmanned aerial vehicles have begun to be applied to forest fire monitoring. The advantage of unmanned aerial vehicles is that they can cross complex terrains, quickly reach the fire scene, and obtain high-precision image and video data. However, most traditional UAV navigation systems rely on a single positioning technology (such as GNSS). When the GNSS signal is blocked or interfered, the positioning accuracy drops significantly, resulting in the failure of the navigation system, and further affecting the accuracy of fire detection.

[0006] Therefore, the existing technologies still face many challenges when dealing with complex terrains, extreme environments and various signal interferences. Therefore, how to improve the positioning accuracy and fire recognition ability of unmanned aerial vehicles in complex environments is a technical problem that needs to be solved urgently at present. Summary of the Invention

[0007] The technical task of the present invention is to provide a GNSS-INS vision combined fire detection method, system, device and medium based on an unmanned aerial vehicle to solve the problem of how to improve the positioning accuracy and fire recognition ability of unmanned aerial vehicles in complex environments.

[0008] The technical task of the present invention is realized in the following way. A GNSS-INS vision combined fire detection method based on an unmanned aerial vehicle is as follows:

[0009] Thermal imaging data analysis: Collect temperature data through a thermal imaging camera, set a dynamic temperature threshold, and calibrate the threshold in real time in combination with the ambient temperature; then use the temperature gradient analysis method to identify the fire spread trend and exclude isolated high-temperature points;

[0010] Visible light image processing: Collect images through an RGB camera, convert them to the HSV color space, extract flame features, and use the YOLOv5 model to detect the flame shape in real time;

[0011] Smoke recognition: Detect the smoke area based on the U-Net segmentation network and output it as a binary mask. Align the binary mask with the wind speed and direction of meteorological data in space and time to predict the direction of fire spread;

[0012] Multi-modal decision fusion: Use the D-S evidence theory to fuse the results of thermal imaging, visible light, and smoke detection. When the confidence levels of thermal imaging, visible light, and smoke detection results are all greater than 80%, it is determined that there is a fire, significantly reducing the false alarm rate.

[0013] Preferably, the specific formula for calibrating the threshold in real time in combination with the ambient temperature is as follows:

[0014] Fire source threshold = background temperature + ΔT;

[0015] Among them, the background temperature is obtained through an ambient temperature sensor; ΔT represents the training results of historical fire data.

[0016] More preferably, the smoke recognition is specifically as follows:

[0017] Meteorological data encoding: Convert the real-time wind speed and direction data into a vector form of wind speed scalar + wind direction angle, and encode it into a feature vector through a fully connected layer;

[0018] Spatio-temporal fusion network: Use the ConvLSTM network to learn the spatio-temporal dependence of smoke diffusion from the smoke mask sequence (time window ≥ 5 frames) output by the U-Net and the meteorological feature vector, and predict the smoke diffusion direction and range in the future period (such as 10 minutes);

[0019] Optimization of smoke diffusion direction and range: Combine the dynamic changes of meteorological data (such as sudden changes in wind speed), and use adaptive Kalman filtering to correct the prediction results in real time to improve the robustness in complex environments.

[0020] More preferably, the thermal imaging camera uses a FLIR Vue Pro R thermal imaging camera; the RGB camera uses a Sony α7R IV RGB camera.

[0021] A GNSS-INS vision combined fire detection system carried by a drone, which includes a drone platform, a GNSS signal receiver, inertial navigation components (IMU), a vision sensor (such as a camera), an obstacle avoidance sensor, and a data fusion module;

[0022] Among them, the UAV platform is used to collect positioning and video image data related to forest fires by installing GNSS signal receivers, inertial navigation components, and visual sensors;

[0023] The GNSS signal receiver is used by users to receive positioning signals provided by the satellite navigation system and provide high-precision positioning services in open environments;

[0024] The inertial navigation components are used to improve the real-time operating state of the UAV during dynamic flight. Even when the GNSS signal of the GNSS signal receiver is lost or interfered, the inertial navigation components can still maintain the stable positioning of the UAV;

[0025] The visual sensor is used to collect image or video data of forest areas and identify and locate forest fires through computer vision technology; it is also used to identify fire sources and smoke at a long distance and provide auxiliary data for fire location;

[0026] The obstacle avoidance sensor is used to help the UAV platform avoid obstacles through real-time point cloud processing;

[0027] The data fusion module is used to perform real-time processing on the data of the GNSS signal receiver, inertial navigation components, and visual sensors through the Kalman filtering algorithm, generate high-precision positioning information, and perform real-time identification and marking of the forest fire location.

[0028] Preferably, the UAV adopts the DJI Matrice 300RTK six-rotor UAV with a payload capacity ≥ 2.5 kg, a flight endurance ≥ 55 minutes, and supports RTK centimeter-level positioning;

[0029] The GNSS signal receiver is equipped with a U-blox ZED-F9P dual-frequency receiver, supporting multi-systems such as GPS, GLONASS, and Galileo, with a positioning accuracy of horizontal ±1 cm + 1 ppm and vertical ±1.5 cm + 1 ppm;

[0030] The inertial navigation components adopt an ADIS16470 MEMS inertial measurement unit, with a triaxial gyroscope range of ±2000° / s and a zero-bias stability of 0.8° / hr; a triaxial accelerometer range of ±16 g and a zero-bias stability of 30 μg; and a redundant IMU, specifically: dual ADIS16470 modules, synchronously collected through the SPI interface, and data cross-validation.

[0031] Preferably, the visual sensor is equipped with a FLIR Vue Pro R 640 thermal imaging camera (resolution 640×512, thermal sensitivity <50 mK) and a Sony α7R IV full-frame RGB camera (61 million pixels), supporting real-time image acquisition and fire source thermal radiation identification;

[0032] The obstacle avoidance sensor uses a Livox Mid-40 lidar (horizontal field of view angle 38.4°, vertical field of view angle 28.8°) and supports real-time point cloud processing;

[0033] The data fusion module is based on the NVIDIA Jetson AGX Xavier embedded platform, integrates a multi-threaded data fusion algorithm, supports real-time synchronous processing of GNSS signal receivers / inertial navigation components / visual data, and the processing frequency ≥ 100Hz.

[0034] Preferably, the working process of the system is as follows:

[0035] (1) Takeoff and positioning initialization: After the UAV platform takes off, it performs initial positioning through a GNSS signal receiver to obtain the current position;

[0036] (2) Data fusion and fire risk monitoring: During the flight of the UAV platform, the GNSS signal receiver, inertial navigation components, and visual sensors continuously collect data. After being processed by the data fusion module, accurate three-dimensional positioning information is generated; at the same time, the visual sensor monitors the forest area through image recognition technology to detect fire sources and smoke; during the flight, the lidar and visual sensors cooperate to construct an obstacle map. If the detected obstacle distance < 10m, an emergency obstacle avoidance is immediately triggered (lateral offset ≥ 5m);

[0037] (3) The switching mechanism when the satellite signal is lost is as follows:

[0038] Trigger condition: The GNSS signal is continuously lost ≥ 5 seconds or the positioning error > 10 meters;

[0039] Switching action: Start the visual sensor and load the pre-stored environmental feature point library; the inertial navigation component provides the initial pose, and the visual data updates the UAV position in real time through the ORB-SLAM3 algorithm; at the same time, the thermal imaging camera continuously monitors the change of the fire source thermal radiation and corrects the fire source coordinates in combination with the IMU data.

[0040] An electronic device includes: a memory and at least one processor;

[0041] Wherein, a computer program is stored on the memory;

[0042] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the GNSSINS vision combined fire detection method based on the UAV as described above.

[0043] A computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the GNSS-INS vision combined fire detection method based on an unmanned aerial vehicle as described above.

[0044] The GNSS-INS vision combined fire detection method, system, device and medium of the present invention have the following advantages:

[0045] (1) The present invention is mainly applied to the monitoring and location identification of forest fires. By combining the Global Navigation Satellite System (GNSS), Inertial Navigation System (IMU) and vision sensors, the system realizes stable positioning and efficient fire monitoring in complex terrains and harsh environments; in particular, in the case of satellite signal failure or interference, the IMU and vision sensors can continue to provide accurate positioning support.

[0046] (2) The present invention can quickly cover a wide area, identify the fire source in real time and transmit the location information to the ground control center, significantly improving the accuracy and efficiency of forest fire monitoring. Through multi-sensor fusion technology, the present invention effectively solves the problems of insufficient positioning accuracy and monitoring failure caused by environmental interference in traditional monitoring methods, and has strong market prospects and technical feasibility.

[0047] (3) By combining the Global Navigation Satellite System (GNSS), Inertial Navigation System (IMU) and vision sensors, the present invention realizes the rapid positioning and accurate identification of forest fires, and based on multi-modal data fusion technology, combines thermal imaging, visible light images and environmental parameters to achieve high-precision fire source detection and false alarm suppression, effectively solving the application bottleneck of traditional fire monitoring technology in complex terrains and harsh environments.

[0048] (4) Using the unmanned aerial vehicle as an aerial platform, the present invention can easily cross complex terrains such as mountains and rivers, realize the rapid deployment and real-time monitoring of forest fires, greatly improving the efficiency and coverage of fire monitoring, especially in remote areas where ground sensors cannot be used, and realizing fire detection across terrains.

[0049] (5) By combining the Global Navigation Satellite System (GNSS), Inertial Navigation System (IMU) and vision sensors (such as cameras), the present invention can provide accurate positioning information under different environmental conditions through the data fusion of the three. The combination of the three sensors makes up for the deficiencies of a single sensor. Especially when the satellite signal is weak or loses lock, the IMU and vision sensors can still provide stable positioning support.

[0050] (6) In an open environment, the present invention mainly uses GNSS positioning, with IMU providing attitude compensation and visual sensors assisting in fire source identification. When the GNSS signal is lost, the system automatically switches to IMU / visual combined navigation. The IMU provides short-term high-precision pose calculation, and the visual SLAM algorithm constructs an environmental map through feature point matching to achieve continuous positioning without GNSS signals.

[0051] (7) The data fusion module of the present invention adopts a loose coupling architecture, realizes multi-source data synchronization through timestamp alignment, fuses GNSS position data and IMU attitude data through Kalman filtering, and outputs high-frequency pose information. The visual fire source recognition result is matched with the GNSS / IMU positioning result through a geometric triangulation algorithm to generate the geographical coordinates of the fire source. It has the following advantages:

[0052] ① Improve positioning accuracy: By combining the use of GNSS, IMU, and visual sensors, the positioning accuracy in complex environments is improved.

[0053] ② Enhance system reliability: Even in extreme environments, the system can continue to work, ensuring that the drone is not affected by external interference.

[0054] ③ Expand the monitoring range: The drone can quickly cover a large area without being restricted by terrain, improving the efficiency of fire monitoring.

[0055] ④ Real-time fire recognition and alarm: Combining computer vision technology, it can identify the fire source in real time, accurately locate it, and quickly issue an alarm.

[0056] (8) The present invention uses the resection algorithm to dynamically calculate the fire source position by using the measured distances of three known coordinate points and combining the real-time position information of the drone. An iterative optimization method is adopted to improve the fire source positioning accuracy through initial coordinate hypothesis and error correction. The satellite and inertial navigation data are fused through a Kalman filter to correct the positioning error in real time. The filter dynamically adjusts the weight according to the GNSS signal quality and gives priority to relying on IMU data when the signal is lost to ensure positioning continuity. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention will be further described below with reference to the drawings.

[0058] Att Figure 1 is a schematic diagram of a GNSS-INS-visual combined fire detection system carried by a drone;

[0059] Att Figure 2 is a schematic diagram of the working process of a GNSS-INS-visual combined fire detection system carried by a drone. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] The following provides a detailed description of the GNSS-INS vision combined fire detection method, system, device, and medium based on an unmanned aerial vehicle (UAV) with reference to the accompanying drawings of the specification and specific embodiments.

[0061] Example 1:

[0062] This example provides a GNSS-INS vision combined fire detection method based on an unmanned aerial vehicle (UAV), and the method is as follows:

[0063] S1. Thermal imaging data analysis: Collect temperature data through a thermal imaging camera, set a dynamic temperature threshold (default ≥150°C), and calibrate the threshold in real time in combination with the ambient temperature; then use the temperature gradient analysis method to identify the fire spread trend and exclude isolated high-temperature points.

[0064] S2. Visible light image processing: Collect images through an RGB camera, convert them to the HSV color space, extract flame features (red channel saturation > 0.6, brightness > 0.8), and detect the flame shape in real time through the YOLOv5 model; among them, the training data of the YOLOv5 model contains 100,000 images of forest fires and interference sources (such as sunset glow, vehicle lights), and the accuracy rate reaches 98.2%; if it is necessary to adapt to different scene requirements, other object detection models can also be used to achieve the fire source detection function by adjusting the network structure and training strategy.

[0065] S3. Smoke recognition: Detect the smoke area based on the U-Net segmentation network and output it as a binary mask, and align the binary mask with the wind speed and wind direction of the meteorological data in space and time to predict the fire spread direction.

[0066] S4. Multi-modal decision fusion: Use the D-S evidence theory to fuse the results of thermal imaging, visible light, and smoke detection. When the confidence levels of the thermal imaging, visible light, and smoke detection results are all greater than 80%, it is determined that there is a fire, significantly reducing the false alarm rate.

[0067] The specific formula for calibrating the threshold in real time in combination with the ambient temperature in step S1 of this example is as follows:

[0068] Fire source threshold = background temperature + ΔT;

[0069] Among them, the background temperature is obtained through an ambient temperature sensor; ΔT represents the training result of historical fire data.

[0070] The smoke recognition in step S3 of this example is as follows:

[0071] S301. Meteorological data encoding: Convert the real-time wind speed and wind direction data into a vector form of wind speed scalar + wind direction angle, and encode it into a feature vector through a fully connected layer.

[0072] S302, Spatiotemporal Fusion Network: The smoke mask sequence (time window ≥ 5 frames) output by the U-Net and the meteorological feature vector are used to learn the spatiotemporal dependence of smoke diffusion through the ConvLSTM network, and predict the smoke diffusion direction and range in the future period (such as 10 minutes).

[0073] S303, Smoke Diffusion Direction and Range Optimization: Combining the dynamic changes of meteorological data (such as sudden wind speed changes), adaptive Kalman filtering is used to correct the prediction results in real time, improving the robustness in complex environments.

[0074] In this embodiment, the thermal imaging camera uses a FLIR Vue Pro R thermal imaging camera; the RGB camera uses a Sony α7RIV RGB camera.

[0075] Embodiment 2:

[0076] As shown in the appendix Figure 1 This embodiment provides a GNSS-INS vision combined fire detection system carried by a drone. The system includes a drone platform, a GNSS signal receiver, inertial navigation components (IMU), a vision sensor (such as a camera), an obstacle avoidance sensor, and a data fusion module.

[0077] Among them, the drone platform is used to collect positioning and video image data related to fire risks by installing a GNSS signal receiver, inertial navigation components, and a vision sensor; the drone uses a DJI Matrice 300RTK six-rotor drone with a load capacity ≥ 2.5 kg, a flight endurance ≥ 55 minutes, and supports RTK centimeter-level positioning.

[0078] The GNSS signal receiver is used to receive the positioning signal provided by the satellite navigation system and provide high-precision positioning services in an open environment; the GNSS signal receiver is equipped with a U-blox ZED-F9P dual-frequency receiver, supporting multi-systems such as GPS, GLONASS, and Galileo, with a positioning accuracy of horizontal ±1 cm + 1 ppm and vertical ±1.5 cm + 1 ppm.

[0079] The inertial navigation components are used to improve the real-time operating state of the drone during dynamic flight. Even when the GNSS signal of the GNSS signal receiver is lost or interfered, the inertial navigation components can still maintain the stable positioning of the drone; the inertial navigation components use an ADIS16470 MEMS inertial measurement unit, with a triaxial gyroscope range of ±2000° / s and a zero-bias stability of 0.8° / hr; the triaxial accelerometer range is ±16g and the zero-bias stability is 30μg; and a redundant IMU is adopted, specifically: two ADIS16470 modules, synchronously collected through the SPI interface, and data cross-validation is performed.

[0080] The visual sensor is used to collect image or video data of forest areas, and identify and locate fire hazards through computer vision technology; it is also used to identify fire sources and smoke at a long distance to provide auxiliary data for fire location; the visual sensor is equipped with a FLIR Vue Pro R 640 thermal imaging camera (resolution 640×512, thermal sensitivity <50mK) and a Sony α7R IV full-frame RGB camera (61 million pixels), and supports real-time image acquisition and identification of fire source thermal radiation;

[0081] The obstacle avoidance sensor is used to help the UAV platform avoid obstacles through real-time point cloud processing; the obstacle avoidance sensor uses a Livox Mid-40 lidar (horizontal field of view angle 38.4°, vertical field of view angle 28.8°), and supports real-time point cloud processing;

[0082] The data fusion module is used to perform real-time processing on the data of the GNSS signal receiver, inertial navigation components and visual sensor through the Kalman filtering algorithm, generate high-precision positioning information, and perform real-time identification and marking of the fire hazard location; the data fusion module is based on the NVIDIA Jetson AGX Xavier embedded platform, integrates a multi-threaded data fusion algorithm, supports real-time synchronous processing of GNSS signal receiver / inertial navigation components / visual data, and the processing frequency ≥100Hz. The data fusion module can exist in the UAV itself or can be remotely linked to transmit the data in real time and use more powerful computing power for operation in the background.

[0083] As shown in the appendix Figure 2 As shown, the UAV platform receives satellite signals through GNSS, the IMU monitors acceleration and angular velocity in real time, and the visual sensor collects fire source images; then, the multi-source data is fused through the data processing module, and the geographical coordinates of the fire source are output and transmitted to the ground control center; specifically as follows:

[0084] (1) Takeoff and positioning initialization: After the UAV platform takes off, it performs initial positioning through the GNSS signal receiver to obtain the current position;

[0085] (2) Data fusion and fire hazard monitoring: During the flight of the UAV platform, the GNSS signal receiver, inertial navigation components and visual sensor continuously collect data. After being processed by the data fusion module, precise three-dimensional positioning information is generated; at the same time, the visual sensor monitors the forest area through image recognition technology to detect fire sources and smoke; during the flight, the lidar and visual sensor cooperate to construct an obstacle map. If the detected obstacle distance <10m, an emergency obstacle avoidance is immediately triggered (lateral offset ≥5m);

[0086] (3) Switching mechanism when the satellite signal is lost, specifically as follows:

[0087] Trigger condition: Continuous loss of GNSS signal ≥ 5 seconds or positioning error > 10 meters;

[0088] Switching action: Activate the vision sensor and load the pre-stored environmental feature point library; The inertial navigation components provide the initial pose, and the vision data updates the UAV position in real time through the ORB-SLAM3 algorithm; Meanwhile, the thermal imaging camera continuously monitors the change of the heat radiation of the fire source, and corrects the fire source coordinates in combination with the IMU data.

[0089] By analyzing the collected image data in real time, the system can quickly identify the fire source and mark its location. The recognition result will be transmitted to the ground control center in real time through the wireless communication module to assist in command and dispatch, as follows:

[0090] (1) UAV platform: Equipped with a multi-spectral camera, IMU and GNSS receiver, supporting autonomous flight path planning and obstacle avoidance.

[0091] (2) Multi-sensor cooperation logic: GNSS provides the global positioning reference, IMU compensates for the dynamic attitude error, and the vision sensor corrects the local positioning through feature matching. The data fusion module adopts a federated filtering architecture and dynamically allocates weights according to the sensor confidence (GNSS: 50%, IMU: 30%, vision: 20%).

[0092] (3) Real-time guarantee mechanism: Real-time guarantee: The data processing module uses TensorRT to accelerate inference, and the fire recognition delay < 200ms.

[0093] (4) False alarm handling mechanism: Introduce time continuity verification, trigger an alarm only when the fire source is detected in 5 consecutive frames of images, and exclude the marked fixed heat sources (such as factories) by comparing the GPS coordinates with the historical fire database.

[0094] Meanwhile, to ensure the stable flight of the UAV in complex environments, the system adopts the following technical solutions:

[0095] ① Wind resistance control strategy: Based on the IMU, the wind speed is monitored in real time (the airflow disturbance is calculated through the accelerometer), and the parameters of the PID controller are dynamically adjusted. When the wind speed > 10m / s, the wind resistance mode is activated (increase the motor output torque and reduce the flight speed).

[0096] ② Obstacle avoidance and path planning: Integrate the Livox Mid-40 lidar (detection distance 260m, accuracy ±2cm), construct a 3D environmental map in combination with the RGB-D camera, and use the RRT* algorithm to plan the obstacle avoidance path in real time, with a replanning frequency ≥ 10Hz.

[0097] ③ Redundant design and fault tolerance mechanism: Dual IMU redundant configuration (main - backup switch delay < 50ms), automatically switches to the backup unit when the main IMU data is abnormal. After the GNSS signal is lost, visual SLAM and IMU pre - integrated data are fused through federated Kalman filtering to ensure the positioning error < 1m / minute.

[0098] ④ High - temperature environment adaptability: The drone shell is made of high - temperature - resistant composite materials (temperature tolerance ≥ 300°C), and heat - insulation layers are installed on key electronic components. When the ambient temperature > 80°C is detected above the fire site, the flight altitude is automatically increased to a safe area (default ≥ 100m).

[0099] The positioning principle of this embodiment is to use the drone to perform positioning operations at different positions in the air. The drone determines its approximate position by receiving signals from global navigation satellite system satellites. The drone can observe the fire area from different angles, and through information exchange and data fusion among each other, improve the accuracy of determining the position of the fire point.

[0100] Based on multi - drone joint positioning, inertial navigation and visual positioning means are introduced. The drone can be equipped with devices such as GNSS receivers, inertial devices, cameras, etc. In the real environment, the drone can first determine its position through satellite positioning. On this basis, it can combine satellite positioning through inertial devices to optimize the situation where the satellite signal condition is poor. Secondly, it can use visual / inertial integrated navigation to make up for the drone positioning when some satellite signals are lost. Finally, it can receive the fire source point information through image data, measure the distance between the drone and the ground target (including possible fire points) through visual ranging, and then calculate the coordinates of the unknown point through resection by distance. By using multiple positioning means to correct and improve the positioning accuracy, reduce the positioning inaccuracy problems caused by factors such as satellite signal errors and drone self - positioning errors.

[0101] Embodiment 3:

[0102] This embodiment also provides an electronic device, including: a memory and a processor;

[0103] Among them, the memory stores computer - executable instructions;

[0104] The processor executes the computer - executable instructions stored in the memory, so that the processor executes the GNSS - INS - vision combined fire detection method based on the drone carried in any embodiment of the present invention.

[0105] The processor can be a central processing unit (CPU), or can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0106] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by calling the data stored in the memory. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include high-speed random access memory, and can also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash memory card, at least one magnetic disk storage period, flash memory device, or other volatile solid-state storage devices.

[0107] Embodiment 4:

[0108] This embodiment also provides a computer-readable storage medium, which stores multiple instructions. The instructions are loaded by the processor to make the processor execute the GNSS-INS vision combined fire detection method based on an unmanned aerial vehicle according to any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided. Software program codes for implementing the functions of any one of the above embodiments are stored on the storage medium, and the computer (or CPU or MPU) of the system or device reads and executes the program codes stored in the storage medium.

[0109] In this case, the program code read from the storage medium itself can implement the functions of any one of the above embodiments. Therefore, the program code and the storage medium storing the program code constitute a part of the present invention.

[0110] Embodiments of the storage medium for providing program codes include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code can be downloaded from a server computer via a communication network.

[0111] In addition, it should be clear that not only can part or all of the actual operations be completed by executing the program code read by a computer, but also by an operating system operating on the computer based on the instructions of the program code, thereby implementing the functions of any one of the above embodiments.

[0112] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU etc. installed on the expansion board or the expansion unit execute part and all of the actual operations, thereby implementing the functions of any one of the above embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fire detection method based on GNSSINS visual combination carried by unmanned aerial vehicles, characterized in that: The method is as follows: Thermal imaging data analysis: Collect temperature data through thermal imaging cameras, set dynamic temperature thresholds, and calibrate the thresholds in real time based on ambient temperature; then use temperature gradient analysis to identify the trend of fire source spread and eliminate isolated high-temperature points; Visible light image processing: The image is collected through the RGB camera and converted to the HSV color space, the flame features are extracted, and the flame shape is detected in real time through the YOLOv5 model; Smoke recognition: Detect smoke areas based on the U-Net segmentation network and output them as binary masks. The binary masks are spatially and temporally aligned with the wind speed and direction of the meteorological data to predict the direction of fire spread. Multimodal decision fusion: DS evidence theory is used to fuse thermal imaging, visible light and smoke detection results. When the confidence of thermal imaging, visible light and smoke detection results is greater than 80%, it is determined that there is a fire.

2. The GNSSINS visual combined fire detection method based on unmanned aerial vehicle according to claim 1 is characterized in that: The specific formula for real-time calibration threshold based on ambient temperature is as follows: Fire source threshold = background temperature + ΔT; Among them, the background temperature is obtained through the ambient temperature sensor; ΔT represents the historical fire data training result.

3. The GNSSINS visual combined fire detection method based on an unmanned aerial vehicle according to claim 1 or 2, characterized in that: Smoke identification is as follows: Meteorological data encoding: convert the real-time wind speed and wind direction data into vector wind speed scalar + wind direction angle, and encode them into feature vectors through the fully connected layer; Spatiotemporal fusion network: The smoke mask sequence and meteorological feature vector output by U-Net are used through the ConvLSTM network to learn the spatiotemporal dependency of smoke diffusion and predict the direction and range of smoke diffusion in the future period. Optimization of smoke diffusion direction and range: Combined with the dynamic changes of meteorological data, adaptive Kalman filtering is used to make real-time corrections to the prediction results.

4. The fire detection method based on GNSSINS visual combination carried by unmanned aerial vehicle according to claim 3 is characterized in that: The thermal imaging camera uses a FLIR Vue Pro R thermal imaging camera; the RGB camera uses a Sony α7R IV RGB camera.

5. A GNSSINS visual combined fire detection system based on an unmanned aerial vehicle, characterized in that: The system includes a UAV platform, GNSS signal receiver, inertial navigation components, visual sensors, obstacle avoidance sensors and data fusion modules; Among them, the UAV platform is used to collect positioning and video image data related to fire risks by building GNSS signal receivers, inertial navigation components and visual sensors; GNSS signal receiver users receive positioning signals provided by satellite navigation systems and provide high-precision positioning services in open environments; Inertial navigation components are used to improve the real-time operation status of the UAV in dynamic flight. Even if the GNSS signal of the GNSS signal receiver is lost or interfered, the inertial navigation components can still maintain the stable positioning of the UAV; Visual sensors are used to collect images or video data of forest areas and identify and locate fires using computer vision technology. They are also used to identify fire sources and smoke at long distances and provide auxiliary data for fire location. Obstacle avoidance sensors are used to help drone platforms avoid obstacles through real-time point cloud processing; The data fusion module is used to process the data of GNSS signal receivers, inertial navigation components and visual sensors in real time through the Kalman filtering algorithm, generate high-precision positioning information, and perform real-time identification and marking of fire risk locations.

6. The GNSSINS visual combined fire detection system based on unmanned aerial vehicle according to claim 5 is characterized in that: The drone uses a DJI Matrice 300RTK six-rotor drone with a load capacity of ≥2.5kg, a flight time of ≥55 minutes, and supports RTK centimeter-level positioning; The GNSS signal receiver is equipped with U-blox ZED-F9P dual-frequency receiver, which supports GPS, GLONASS, and Galileo systems, with a positioning accuracy of ±1cm+1ppm horizontally and ±1.5cm+1ppm vertically. The inertial navigation components use the ADIS16470 MEMS inertial measurement unit, with a three-axis gyroscope range of ±2000° / s and a zero bias stability of 0.8° / hr; a three-axis accelerometer range of ±16g and a zero bias stability of 30μg; and a redundant IMU is used, specifically: dual ADIS16470 modules, synchronous acquisition through the SPI interface, and data cross-verification.

7. The GNSSINS visual combined fire detection system based on unmanned aerial vehicle according to claim 5 is characterized in that: The visual sensor is equipped with a FLIR Vue Pro R 640 thermal imaging camera (resolution 640×512, thermal sensitivity <50mK) and a Sony α7R IV full-frame RGB camera (61 million pixels), which supports real-time image acquisition and fire source thermal radiation identification; The obstacle avoidance sensor uses the Livox Mid-40 laser radar, which supports real-time point cloud processing; The data fusion module is based on the NVIDIA Jetson AGX Xavier embedded platform, integrates a multi-threaded data fusion algorithm, supports real-time synchronous processing of GNSS signal receivers / inertial navigation components / visual data, and has a processing frequency of ≥100Hz.

8. The GNSSINS visual combined fire detection system based on an unmanned aerial vehicle according to any one of claims 5 to 7, characterized in that: The working process of the system is as follows: (1) Takeoff and positioning initialization: After the UAV platform takes off, it performs initial positioning through the GNSS signal receiver to obtain the current position; (2) Data fusion and fire risk monitoring: During the flight of the UAV platform, the GNSS signal receiver, inertial navigation components and visual sensors continuously collect data, which are processed by the data fusion module to generate accurate three-dimensional positioning information. At the same time, the visual sensor monitors the forest area through image recognition technology to detect fire sources and smoke. During the flight, the lidar and visual sensors work together to build an obstacle map. If an obstacle is detected to be less than 10m away, emergency obstacle avoidance is immediately triggered. (3) The switching mechanism when the satellite signal is lost is as follows: Trigger conditions: GNSS signal is lost continuously for ≥5 seconds or positioning error is greater than 10 meters; Switching action: Start the visual sensor and load the pre-stored environmental feature point library; the inertial navigation components provide the initial posture, and the visual data updates the drone position in real time through the ORB-SLAM3 algorithm; at the same time, the thermal imaging camera continuously monitors the changes in the heat radiation of the fire source and corrects the coordinates of the fire source in combination with the IMU data.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the GNSSINS visual combined fire detection method based on the drone as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the GNSSINS visual combined fire detection method based on a drone as described in any one of claims 1 to 4.

Images