Unmanned aerial vehicle airborne forest fire detection method based on dual light

Abstract

The invention discloses an unmanned aerial vehicle airborne forest fire detection method based on double light (visible light and infrared light). The accuracy and reliability of fire detection are remarkably improved through a multiple confirmation mechanism. According to the method, the high maneuverability of the unmanned aerial vehicle and the multi-mode imaging capability of the dual-light camera are combined, so that the accuracy and efficiency of forest fire detection are remarkably improved. According to the method, a dual-light camera carried by an unmanned aerial vehicle is used for cruising in a forest area, and visible light and infrared light images are collected at the same time. The visible light image is used for capturing detail information of a fire scene, such as flame color and smog form, and the infrared light image can penetrate smog to directly detect heat distribution of a fire area, so that the problem of smog shielding is effectively solved. Through combination of an advanced image processing algorithm and a machine learning model, the method can automatically analyze image data, and through combination of visible light and infrared light imaging, defects of a single imaging mode are made up, and comprehensiveness of fire detection is improved.

Description

Technical Field

[0001] The invention belongs to the field of aviation technology, and in particular relates to a method for an unmanned aerial vehicle to realize autonomous inspection of forest fires. Background Art

[0002] With the rapid development of drone technology, the performance of drones has been significantly improved, including flight stability, endurance, load capacity, and intelligence level, etc. This enables drones to perform more complex and sophisticated tasks, including autonomous inspection of forest fires.

[0003] Traditional forest fire prevention methods mainly rely on ground patrols and observation towers, which have the disadvantages of low efficiency, limited coverage, and great influence of terrain and weather. However, forest fires are characterized by suddenness, rapid spread, and wide range of damage, which puts higher demands on fire prevention. Therefore, a more efficient, comprehensive, and real-time forest fire prevention method is needed, and drone autonomous inspection technology has emerged.

[0004] Compared with traditional forest fire prevention methods, autonomous drone inspections have the following advantages: 1. High efficiency: drones can quickly cover large forest areas and improve inspection efficiency; 2. Real-time: by equipping with high-definition cameras, thermal imagers and other equipment, drones can transmit forest images and temperature data in real time, enabling fire management departments to understand forest conditions in a timely manner; 3. Accuracy: combined with advanced AI recognition algorithms, drones can automatically identify abnormal conditions such as fire sources and smoke; 4. Safety: drones can conduct inspections in complex terrain and harsh environments, reducing the risk of people entering dangerous areas.

[0005] In the field of autonomous inspection by drones, there are generally two types of detection: airborne detection and ground-based detection. Airborne detection refers to real-time data collection using various sensors, cameras, LiDAR, infrared thermal imagers and other equipment carried on drones, and the real-time images collected by drones are processed and analyzed by deploying airborne edge processors on drones. The advantage of this method is that it can directly observe the target area at close range and with high resolution, without the need for personnel to enter dangerous or difficult-to-reach areas, greatly improving the safety and efficiency of inspections. Ground-based detection focuses on the reception, processing, analysis and storage of drone inspection data. Although it does not directly participate in data collection, it is a key link to ensure the effective application of inspection results. This detection method has high detection accuracy, but poor real-time performance and is limited by communication quality.

[0006] However, the existing technology has problems such as poor detection accuracy, high false detection rate, and difficulty in fully utilizing multi-spectral sensor cameras when performing airborne detection. In view of the above problems, the present invention proposes a dual-light-based forest fire detection method for drones. A variety of airborne visual sensors are used to obtain different image information (visible light and infrared), and the airborne edge processing equipment is used to quickly process the various image information collected in real time. Image processing technology and lightweight deep convolutional networks are used to complete the detection and identification of forest fires. This can ensure the real-time nature of the detection and a high accuracy rate, thereby realizing autonomous forest fire detection on the drone airborne side. Summary of the invention

[0007] The purpose of the present invention is to provide a real-time forest fire detection method based on dual-light UAV airborne terminal, which solves the existing problems such as low detection accuracy, high false detection rate and difficulty in fully utilizing multi-spectral sensor cameras.

[0008] The specific implementation steps of the present invention are as follows:

[0009] Step 1: registering the bi-optical image, including scaling the bi-optical image, calculating the scaling factor and the offset value, and completing the registration of the bi-optical image.

[0010] Step 2: The onboard processor plans the path for the target plot. Regular plots are directly inspected using the ox-plowing method. Irregular plots are divided into regular plots before inspection.

[0011] Step 3: Use the dual-light image algorithm to detect the real-time acquired image. The infrared detection algorithm uses an adaptive threshold segmentation algorithm to separate the fire target from the background, while the visible light detection algorithm uses the improved MobileNetv2 algorithm for fire detection. Both parts of the detection are run on the onboard processor.

[0012] Step 4: After detecting a fire, in order to further reduce the false detection rate, a multiple detection strategy is constructed to analyze the detection results of the two detectors and take corresponding actions to further reduce the false detection rate and missed detection rate of the airborne end.

[0013] Step 5: After completing the fire detection, the specific image of the fire point is transmitted back and the inspection task is resumed until the inspection task is completed.

[0014] The present invention is also characterized in that:

[0015] Step 1 is implemented according to the following steps:

[0016] Step 1.1: First, scale the two images and unify their sizes to the same size for easy subsequent processing.

[0017] Step 1.2, then calculate the scaling factor and the offset in the XY direction. After scaling the two images to the same size, find three points at the corresponding same positions of the two images to obtain the offset values ​​and scaling factors of the two images in the XY direction. The calculation formula for the scaling factor in the X direction is:

[0018]

[0019] Among them, δ X Indicates the scaling factor in the X direction, A VX Indicates the pixel position in the X direction of point A in the visible light image, A IRX represents the pixel position in the X direction of point A in the infrared image, and B represents the pixel position in the Y direction of point B. Similarly, the scaling factor in the Y direction is equal to the pixel length in the Y direction of the visible light image divided by the pixel length in the Y direction of the infrared image.

[0020] The calculation formula of the X-direction offset is shown in formula (2):

[0021] λ X =A VX -A IRX (2)

[0022] Among them, λ X Indicates the offset in the X direction.

[0023] Step 1.3: Complete image registration based on the calculated scaling factor and offset value. After obtaining the offset in the XY direction, scale and cut the visible light image again according to the length and width of the scaled infrared image to obtain a registered visible light image of the same scene as the infrared image.

[0024] Step 2 is implemented as follows:

[0025] Step 2.1, divide the land to be inspected into regular parts. The shape of the forest area is usually irregular, and is usually an irregular polygon. The irregular polygon can be simplified into a regular rectangle through the circumscribed rectangle. The entire rectangle can completely contain the irregular land, and its shape is regular (rectangular or square). Inside the circumscribed rectangle, according to the inspection requirements and the characteristics of the land, it is further divided into multiple regular small plots (such as small rectangles, squares, etc.).

[0026] Step 2.2: Input the coordinate information of the regular plots into the full coverage path planning algorithm to generate the inspection path. Different path planning algorithms are selected according to the importance of the inspection plots. Basic plots are planned using the ox-tillage method, and important plots are inspected using the spiral coverage method. The total length of the inspection path planned using the ox-tillage method is:

[0027]

[0028] The total length of the inspection path planned by the spiral covering method is:

[0029]

[0030] In the formula, the forest area to be inspected is a regular plot with a length of L and a width of W, and the field of view of the UAV platform is d.

[0031] Step 2.3: Input the planned path points into the onboard edge processor, optimize the generated path, generate the specific flight route of the drone, select the appropriate take-off point and entry angle, and control the drone to execute the inspection command.

[0032] Step 3 is implemented as follows:

[0033] Step 3.1, construct an infrared threshold segmentation algorithm. By setting two thresholds, the pixels in the image are divided into two parts: the target and the background. And because the color adjustment mode of the infrared image plays a key role in image processing and analysis, different color adjustment modes can highlight different features in the infrared image. After obtaining the infrared image with a specific color adjustment mode, the color space change and threshold segmentation algorithm are used to segment the target image, and then the morphological operation is used to suppress the noise, and finally the sliding window method is used to find the location of the fire point.

[0034] Step 3.2, build an airborne visible light forest fire detection algorithm. Aiming at the problems of poor real-time performance and low classification accuracy of existing lightweight image classification models, a method for forest fire classification of UAV airborne visible light images based on improved MobileNet v2 is proposed, and a lightweight feature extraction network with MobileNet v2 as the core is constructed. The number of model parameters is reduced by adjusting the width factor; channel and spatial attention modules are embedded in the model to enhance the network's ability to refine features; a multi-scale feature fusion structure is designed to enhance the network's adaptability to scale; and the model parameters are optimized using transfer learning to further improve the model accuracy.

[0035] Step 4 is implemented according to the following steps:

[0036] Step 4.1. Construct an airborne multiple confirmation strategy. In order to further reduce the false detection rate and missed detection rate of airborne forest fire detection, design a system that integrates multi-dimensional data, multi-sensor integration and image semantic context environment. First, detect the fire based on a single data source, and then fuse the detection results of different data sources. By setting thresholds and weights, multiple confirmations are performed.

[0037] Step 4.2: After multiple confirmations, for different results, the detection results will be further verified to exclude false detections in the corresponding areas. When the detection results of different images are different, the multiple confirmation mechanism will use two different verification strategies to further verify the results: for visible light results, considering the high false detection rate of small target detection, the camera magnification is adjusted, and the visible light magnification is restored after the verification is completed; for infrared detection results, since thermal radiation is directly used for fire judgment, considering the high initial threshold, it is difficult to effectively detect sheet fires, so the segmentation threshold is appropriately lowered during the verification stage, so that sheet fires can be effectively detected.

[0038] Step 5 is implemented according to the following steps:

[0039] Step 5.1: After confirming the fire, the drone hovers and controls the gimbal to shoot the fire site from multiple angles and focal lengths. By shooting from multiple angles, a comprehensive view of the fire scene is obtained, including the direction of the fire spread and the size of the affected area. Shooting with different focal lengths can take into account both macro and micro information of the fire scene.

[0040] Step 5.2: Send back the captured fire images. The computing power of the drone onboard is weak, and it is difficult to process high-resolution, high-quality image data in real time. Therefore, the onboard end compresses the captured images and sends the compressed image data back to the ground station to provide decision support for fire rescue.

[0041] Step 5.3: After completing the image transmission, the UAV system automatically resumes the inspection mission and continues to inspect the forest area. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the dual-light based airborne forest fire detection method of the present invention;

[0043] Figure 2 This is a schematic diagram of the dual-light image matching method of the present invention:

[0044] Figure 3 This is a schematic diagram of the path planning for cattle plowing method according to the present invention:

[0045] Figure 4 The present invention is directed to the spiral method path planning diagram:

[0046] Figure 5 This is a flow chart of the infrared image segmentation algorithm of the present invention:

[0047] Figure 6 This is the network structure diagram of forest fire detection based on improved MobileNetv2 in the present invention: Figure 7This is a schematic diagram of the RFB structure in the forest fire detection network based on the improved MobileNetv2 of the present invention:

[0048] Figure 8 This is a schematic diagram of the improved multi-branch convolution structure in the forest fire detection network based on the improved MobileNetv2 of the present invention:

[0049] Fig. 9 This is a schematic diagram of an improved linear bottleneck structure based on CBAM in a forest fire detection network based on improved MobileNetv2 in the present invention:

[0050] Fig.10 This is a schematic diagram of the CBAM attention of the forest fire detection network based on the improved MobileNetv2 in the present invention:

[0051] Fig.11 This is a schematic diagram of the channel attention calculation of the forest fire detection network based on the improved MobileNetv2 in the present invention:

[0052] Fig.12 This is a schematic diagram of the spatial attention calculation of the forest fire detection network based on the improved MobileNetv2 in the present invention:

[0053] Fig.13 This is a schematic diagram of the multiple confirmation mechanism for the airborne end of the present invention: Fig.14 It is the experimental diagram of simulation trajectory and planning trajectory; Fig.15 This is the GPS trajectory diagram of the experimental flight; Fig.16 This is the result of the dual-light detection of the onboard processor DETAILED DESCRIPTION

[0054] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] The present invention is based on the dual-light UAV airborne forest fire autonomous detection method, the flow chart is as follows Figure 1 As shown, the specific implementation steps are as follows:

[0056] Step 1: Register the bi-optical image, including scaling the bi-optical image, calculating the scaling factor and the offset value, and completing the registration of the bi-optical image.

[0057] Combination Figure 2 , step 1 is implemented according to the following steps:

[0058] Step 1.1: First, scale the two images to unify their sizes to reduce the computational cost of subsequent images and facilitate the calculation of the scaling factor. Transform the visible light image to the same size as the infrared image, 640*512.

[0059] Step 1.2, calculate the scaling factor and the offset in the XY direction. After scaling the two images to the same size, find three points at the corresponding same positions of the two images, so that a pixel length in the XY direction of each image can be obtained. The scaling factor in the X direction is equal to the pixel length in the X direction of the visible light image divided by the pixel length in the X direction of the infrared image. The calculation formula of the scaling factor in the X direction is expressed as:

[0060]

[0061] Among them, δ X Indicates the scaling factor in the X direction, A VX Indicates the pixel position in the X direction of point A in the visible light image, A IRX represents the pixel position of point A in the infrared image in the X direction, and B represents point B. Similarly, the scaling factor in the Y direction is equal to the pixel length in the Y direction of the visible light image divided by the pixel length in the Y direction of the infrared image. By calculation, the scaling factor in the X direction is 0.47, and the scaling factor in the Y direction is 0.65, so that the target object in the infrared image can be scaled to the same size as the target object in the visible light image.

[0062] The offset in the X and Y directions is calculated by respectively calculating the difference in the pixel coordinates X and Y of a certain point in the two images. Here, the visible light image is still based on 640*512, and the infrared image is scaled by the scaling factor to 300*332. The calculation of the offset in the X direction can be expressed as formula (6). Where, λ X It represents the offset in the X direction, and the offset in the Y direction can be obtained in the same way. The offset in the X direction is 162, and the offset in the Y direction is 95. It should be noted that the scaling factor of the infrared image should be calculated based on the visible light image, and then the offset in the X and Y directions should be calculated.

[0063] λ X =A VX -A IRX (6)

[0064] Step 1.3, complete the registration. After obtaining the offset in the XY direction, according to the length and width of the scaled infrared image, a registered visible light image of the same scene as the infrared image can be found on the 640*512 visible light image, and the size is also 300*332. In this way, the image after visible light and infrared registration is obtained.

[0065] Step 2: The onboard processor plans the path for the target plot. Regular plots are directly inspected using the ox-plowing method. Irregular plots are divided into regular plots before inspection.

[0066] Step 2 is implemented as follows:

[0067] Step 2.1. Obtain a detailed map or satellite image of the forest area, and use GIS (Geographic Information System) software and other tools to accurately depict the boundaries of the forest area and generate a minimum bounding rectangle that can completely cover the irregular polygonal plot. This rectangle should be as compact as possible to reduce unnecessary blank areas for inspection. The bounding rectangle can be determined by finding the minimum x coordinate (X min ), the maximum x coordinate (X max ), minimum y coordinate (Y min ) and the maximum y coordinate (Y max ) to achieve this. The coordinates of the four corners of the circumscribed rectangle are: (X min ,Y min ), (X max ,Y min ), (Y max ,Y max ), (X min ,Y max ).

[0068] Step 2.2: To ensure that the equipment can fully and completely cover the target area, a bow-shaped path pattern is used for edge plots to achieve full coverage of the area through a series of parallel and overlapping line segments. Figure 3 As shown. First, divide the target area into several small units, usually regular quadrilaterals, and then select a suitable starting point and flight direction, usually the edge of the area or an easily accessible location. Start from the starting point and fly in the given direction until you reach the edge of the area, then turn 90 degrees to continue the inspection, and repeat the process until the entire area is inspected. On this basis, in order to further improve the inspection efficiency and reduce the overlap between paths, the turning radius can be adjusted according to the field of view of the drone to minimize the operation time and resource consumption to complete the inspection task.

[0069] The forest area to be inspected is a regular plot with a length of L and a width of W. The field of view of the UAV platform is d. Then, the total length of the inspection path is as shown in formula (7):

[0070]

[0071] Among them, ceil represents rounding up, and the number of turns N of the UAV platform is shown in formula (8):

[0072]

[0073] For central plots and more important plots, a spiral covering algorithm is used for path planning. The core idea is to start from a fixed point and traverse the entire target area in the form of a spiral line to ensure that all areas that need to be searched or operated are covered without omission. The advantage of spiral covering path planning is that it can cover the entire area in a continuous and orderly manner, improving work efficiency and coverage. At the same time, due to the characteristics of the spiral line, this planning method can also reduce path overlap and repeated operations to a certain extent.

[0074] The schematic diagram of the spiral covering algorithm is as follows Figure 4 As shown in the figure, according to the different take-off points, it can be divided into external thread method and internal thread method. After the drone takes off from the starting point, it conducts inspection operations in a clockwise or counterclockwise direction until the full coverage inspection of the target forest area is completed. Assuming that d is the width of the drone's field of view, the forest area to be inspected is still a regular plot with a length of L and a width of W. The total length of the drone inspection path planned by the spiral covering algorithm is:

[0075]

[0076] The number of turns N of the UAV platform is shown in formula (4-4):

[0077]

[0078] Step 2.3: Input the planned path point information (latitude and longitude coordinates and altitude information) into the onboard edge processor through the API interface. Verify the input path points to ensure the validity and rationality of the data, and use an algorithm (polynomial interpolation) to smooth the original path points to reduce sharp turns and unnecessary flight distance during the flight of the drone. According to the drone's battery status, wind speed and other environmental factors, adjust the flight speed and altitude to minimize energy consumption and extend the flight time. By analyzing the positional relationship between the take-off point and the first path point, calculate the optimal entry angle of the drone to ensure that the drone can enter the preset path smoothly and efficiently.

[0079] Combination Figure 5-11 , step 3 is implemented according to the following steps:

[0080] Step 3.1: After acquiring the infrared image, the target image is segmented using the YCrCb color space and threshold segmentation algorithm, and then the morphological operation is used to suppress the noise, and finally the sliding window method is used to find the location of the fire point. Considering that it is difficult to directly separate the chromaticity and brightness information in the RGB color space, the image is converted from the RGB color space to the YCrCb color space. The conversion formula is shown as follows:

[0081] Y=0.299·R+0.587·G+0.114·B#(11

[0082] Cr=(RY)×0.713+delta#(12)

[0083] Cb=(BY)×0.564+delta#(13)

[0084] The value of delta is:

[0085]

[0086] At YC r C b In the color space, Y represents brightness information, and chromaticity information is stored in C r and C b Among them, C r Indicates the red component information, C b Indicates the blue component information. In the Arctic isothermal color adjustment mode, the higher the temperature, the brighter the target of the infrared image. Therefore, the infrared image can be segmented from the fire point and the background by using the brightness channel of the converted image to perform threshold processing on the infrared image.

[0087] Among them, g(x,y) is the grayscale value of the pixel at (x,y) in the image after segmentation, T is the segmentation threshold, which is obtained by experiment, and f(x,y) is the Y channel value of the image before segmentation.

[0088] After the image is thresholded, discrete pixels will appear around the segmented target. The opening operation in morphology can act as a filter to eliminate the interfering noise in the image. The operation of first erosion and then expansion can smooth the contour of the segmented target, disconnect narrow discontinuities and eliminate small protrusions.

[0089] The sliding window method is used to traverse the image after the thresholding operation. By traversing the local area (i.e., window) in the image, the pixel values ​​in each window are counted, and then the non-maximum suppression algorithm (Non-Maximum Suppression) is used to eliminate the remaining windows.

[0090] Step 3.2, MobileNetv2 network is an efficient, lightweight and high-precision neural network classification model, which is particularly suitable for deployment on mobile devices and embedded devices to complete visual recognition tasks. In order to further improve the detection accuracy of the model, the improved multi-branch convolution structure is used to improve the linear bottleneck with a step size of 2. By processing data in parallel with convolution kernels of different sizes, different scales and context information can be captured. Secondly, the spatial channel attention module is embedded in a specific network layer so that the model can pay more attention to the extraction of small target features, thereby improving the recognition accuracy of small targets by the improved network. Finally, the expansion factor is used to further reduce the computational complexity of the improved network. The overall schematic diagram of the algorithm is shown below. Figure 6 shown.

[0091] 1) Figure 7 It is the standard structure of RFB. The overall structure draws on the idea of ​​multi-branch convolution. The main difference is the introduction of three dilated convolution layers (for example, the size is 3×3, and the dilation rate is 1). This is also the main architecture of this structure that can increase the receptive field. The improved multi-branch convolution structure is shown in the figure below. Figure 8 As shown in the figure, the input feature map is first upgraded using 1×1 convolution, and then the feature map after the upgraded dimension is extracted using deep convolution. Finally, 1×1 convolution is used to reduce the dimension to make the input and output feature map dimensions consistent. The feature map is processed using the Relu6 activation function during the dimension upgrade and feature extraction operations, and the feature map is operated using the linear activation function during the dimension reduction operation, which can improve the expressive power of the improved multi-branch convolution structure.

[0092] 2) The CBAM attention mechanism effectively combines channel attention and spatial attention. The dual attention mechanism greatly enhances the network's ability to extract key features. In actual use, this module is embedded in the linear bottleneck structure. The specific structure is as follows: Fig. 9 As shown, the specific structure of the module is as follows Fig.10 As shown,

[0093] Fig.11 As shown in the figure, the specific calculation process of the channel attention module is shown. The input feature map is F(H×W×C). After the feature map is processed by the MaxPool layer and the AvgPool layer, two target feature maps of size 1×1×c are obtained. The feature map after the pooling operation is input into the MLP (multi-layer perceptron structure). The MLP has two layers, the first layer consists of C / r (r is the reduction rate) neurons, and the second layer consists of C neurons. The activation function of the MLP is the Relu function. Then the two output features are weighted fused to obtain the final output M c (F):

[0094]

[0095] Among them, M c (F) is the final channel attention feature map, and They represent average pooling features and maximum pooling features, respectively. W0 and W1 are the weight parameters of the first and second layers of MLP, respectively. The channel attention module uses the average pooling layer to obtain aggregated spatial information, and the maximum pooling layer is used to check the importance of features. Through these two parallel pooling layers, the model can automatically learn the weight of each channel, so that it can focus on the most important channels.

[0096] Fig.12 Represents the specific calculation process of the spatial attention module, where the input feature map is M c (F), the input features will first be average pooled and max pooled in a channel dimension to extract feature information at different positions, and then the pooled information will be fused to learn the correlation between different spatial positions. Finally, SAM will generate a spatial attention feature map, which has the same spatial size as the input feature map, and the value in the feature map indicates the importance of the position. Formula (16) is the calculation formula for the spatial attention feature map:

[0097]

[0098] Among them, M s (F) is the final spatial attention feature map, and Represent the average pooling feature and the maximum pooling feature, respectively, 7×7 Represents a standard convolution operation of size 7 × 7. The spatial attention module enables the model to better learn the weights of different spatial positions, so that the network can focus on the most important positions.

[0099] The difference between the improved network architecture and the original network is that the improved MobileNetv2 network uses the RFB-s module to replace the first layer of standard convolution in the original network; the second to eighth layers of the improved network are also composed of linear bottleneck structures, but the depth-separable structure in the linear bottleneck structure with a step size of 2 is replaced by the improved multi-branch convolution module. In the use of the attention mechanism, through ablation experiments, the channel space attention mechanism is only embedded in the inverted residual structure of the second and seventh layers. Then, the convolution layer and the global average pooling layer are used to expand the target features output by the eighth layer of linear bottleneck in space. Finally, the fully connected layer is used to map the expanded features to image classification results, thereby completing the forest fire recognition of drone images.

[0100] Step 4 is implemented according to the following steps:

[0101] Step 4.1. Construct an airborne multiple confirmation strategy. In order to further reduce the false detection rate and missed detection rate of airborne forest fire detection, a system that integrates multi-dimensional data, multi-sensor integration and image semantic context environment is designed. The drone platform inputs the real-time captured dual-light image data into two different detection algorithms for fire detection. Each sensor independently runs a specially designed fire detection algorithm. The visible light camera uses an image recognition algorithm to detect the color and shape of the flame; the infrared thermal imager identifies the fire based on the temperature abnormality area; Fig.12 It can be seen that the visible light image and infrared image captured at the same time will enter different detection algorithms for processing, and the two classifiers will process different images. Then, the detection results of different data sources will be fused, and multiple confirmations of the detection results will be performed by setting thresholds and weights.

[0102] When the detection results of the two sensors are the same: 1. If both detection algorithms detect a fire point, it is determined that there is a fire point in the area, and the UAV platform stops the inspection task and starts to locate the fire point to obtain the precise location of the fire point; 2. If neither of the two detection algorithms detects a fire point, it is determined that there is no fire point in the image and the image continues to be detected; When the results are different, the detection results will be further verified to exclude false detections in the corresponding area. When the detection results of different images are different, the multiple confirmation mechanism will use two different verification strategies to further verify the results: For visible light results, considering the high false detection rate of small target detection, the camera magnification is adjusted. After the verification is completed, regardless of the result, the camera automatically returns to the original magnification to continue monitoring a wider area. For infrared detection results, since thermal radiation is directly used for fire judgment, considering the high initial threshold, it is difficult to effectively detect sheet fires. Therefore, the segmentation threshold is appropriately lowered in the verification stage, and the detected abnormal thermal radiation area is further analyzed, such as shape, temperature distribution and other features, to distinguish the real fire area from other heat sources (such as the ground directly exposed to the sun, animal body temperature, etc.).

[0103] Step 5 is implemented according to the following steps:

[0104] Step 5.1: Once the fire is confirmed, the drone needs to quickly fly over the fire scene and hover stably. At this time, the drone needs to determine the hovering height based on the visible light detection results. If the detection results cannot detect the fire, the drone's hovering height is raised. The final drone hovering height must ensure that the drone is not directly affected by flames and smoke, and that the captured images are clear and comprehensive. Control the gimbal to shoot the fire site from multiple angles and focal lengths. Use multi-angle shooting to obtain a comprehensive view of the fire scene, including the direction of fire spread and the size of the affected area. Using different focal lengths for shooting can take into account both macro and micro information of the fire scene.

[0105] Step 5.2: Send back the captured fire images. The computing power of the drone onboard is weak, and it is difficult to process high-resolution, high-quality image data in real time. The direct transmission of uncompressed high-resolution image data between the drone and the ground station will take up a lot of bandwidth. Therefore, the onboard end compresses the captured images, compresses and transmits the acquired image data using the ROS operating system, adds timestamps, focal length and other data to the compressed image code and sends it back to the ground station to provide decision support for fire rescue.

[0106] Step 5.3: After the image is transmitted back, the drone system immediately starts its built-in automatic recovery process and seamlessly connects to the original inspection task. First, through the communication module, it is confirmed that the image data has been safely and completely transmitted to the ground control station. The drone uses its built-in GPS navigation system to compare with the pre-planned inspection route, automatically corrects the current position and heading, and ensures that it returns to the inspection route without error. In the process of resuming the inspection task, the drone will regularly check its own status, such as battery power, flight stability, sensor performance, etc., to ensure the safety and efficiency of the entire inspection task.

[0107] Example 1

[0108] The present invention is based on the dual-light UAV airborne forest fire detection method, the flow chart is as follows Figure 1 As shown, the specific implementation steps are as follows:

[0109] Step 1: registering the bi-optical image, including scaling the bi-optical image, calculating the scaling factor and the offset value, and completing the registration of the bi-optical image.

[0110] Step 2: The onboard processor plans the path for the target plot. Regular plots are directly inspected using the ox-plowing method. Irregular plots are divided into regular plots before inspection.

[0111] Step 3: Use the dual-light image algorithm to detect the real-time acquired image. The infrared detection algorithm uses an adaptive threshold segmentation algorithm to separate the fire target from the background, while the visible light detection algorithm uses the improved MobileNetv2 algorithm for fire detection. Both parts of the detection are run on the onboard processor.

[0112] Step 4: After detecting a fire, in order to further reduce the false detection rate, a multiple detection strategy is constructed to analyze the detection results of the two detectors and take corresponding actions to further reduce the false detection rate and missed detection rate of the airborne end.

[0113] Step 5: After completing the fire detection, the specific image of the fire point is transmitted back and the inspection task is resumed until the inspection task is completed.

[0114] Example 2

[0115] In order to better verify the application of the present invention in the field of forest fire detection, two typical examples are selected for simulation to verify the effect of the present invention.

[0116] Experiment 1 is a flight experiment of the present invention in a fire-free environment, and the experimental parameters are set as follows: In this simulation and actual flight experiment, taking into account the actual flight time of the UAV, a rectangular plot of 50 meters long and 20 meters wide was selected for the inspection task experiment. The UAV's flight altitude is 20 meters, the pitch angle of the gimbal is -60°, and the UAV's field of view width is 5 meters. In order to further improve the inspection efficiency, while ensuring full coverage and the total path length remains unchanged, the OSDK is used to fix the nose direction, which can avoid the UAV platform's turning action during the inspection process and reduce the situation where the UAV suddenly changes direction or posture. The simulation results in the simulator are as follows Figure 4-9 As shown,

[0117] Experiment 1 is a flight experiment of the present invention in a fire environment. The experimental parameters are set as follows: In this simulation and actual flight experiment, the length of the plot to be inspected is 50 meters and the width is 12 meters. A fire point of 0.14*0.14 meters is placed in the plot to be inspected. The flight altitude of the UAV platform is 15 meters, the flight speed is 2m / s, and the initial angle of the gimbal is -60°. The GPS trajectory of the UAV platform for inspection when there is a fire point is different from the actual flight trajectory. Fig.15 As shown in the figure, after detecting a fire, the UAV platform stops patrolling, starts taking photos of the fire, and transmits the corresponding images back to the ground station. Fig.16 shown.

Claims

1. A dual-light based UAV-borne forest fire detection method, characterized in that: Follow the steps below to implement it: Step 1: registering the bi-optical image, including scaling the bi-optical image, calculating the scaling factor and the offset value, and completing the registration of the bi-optical image. Step 2: The onboard processor plans the path for the target plot. Regular plots are directly inspected using the ox-plowing method. Irregular plots are divided into regular plots before inspection. Step 3: Use the dual-light image algorithm to detect the real-time acquired image. The infrared detection algorithm uses an adaptive threshold segmentation algorithm to separate the fire target from the background, while the visible light detection algorithm uses the improved MobileNetv2 algorithm for fire detection. Both parts of the detection are run on the onboard processor. Step 4: After detecting a fire, in order to further reduce the false detection rate, a multiple detection strategy is constructed to analyze the detection results of the two detectors and take corresponding actions to further reduce the false detection rate and missed detection rate of the airborne end. Step 5: After completing the fire detection, the specific image of the fire point is transmitted back and the inspection task is resumed until the inspection task is completed.

2. The dual-light based UAV airborne forest fire detection method according to claim 1 is characterized in that: The step 1 is specifically implemented according to the following steps: Step 1 is implemented according to the following steps: Step 1.1: First, scale the two images and unify their sizes to the same size for easy subsequent processing. Step 1.2, then calculate the scaling factor and the offset in the XY direction. After scaling the two images to the same size, find three points at the corresponding same positions of the two images to obtain the offset values ​​and scaling factors of the two images in the XY direction. The calculation formula for the scaling factor in the X direction is: Among them, δ X Indicates the scaling factor in the X direction, A VX Indicates the pixel position in the X direction of point A in the visible light image, A IRX represents the pixel position in the X direction of point A in the infrared image, and B represents the pixel position in the Y direction of point B. Similarly, the scaling factor in the Y direction is equal to the pixel length in the Y direction of the visible light image divided by the pixel length in the Y direction of the infrared image. The calculation formula of the X-direction offset is shown in formula (2): l X =A VX -A IRX (18) Among them, λ X Indicates the offset in the X direction. Step 1.3: Complete image registration based on the calculated scaling factor and offset value. After obtaining the offset in the XY direction, scale and cut the visible light image again according to the length and width of the scaled infrared image to obtain a registered visible light image of the same scene as the infrared image.

3. The dual-light based UAV airborne forest fire detection method according to claim 3 is characterized in that: The step 2 is specifically implemented according to the following steps: Step 2.1, divide the land to be inspected into regular parts. The shape of the forest area is usually irregular, and is usually an irregular polygon. The irregular polygon can be simplified into a regular rectangle through the circumscribed rectangle. The entire rectangle can completely contain the irregular land, and its shape is regular (rectangular or square). Inside the circumscribed rectangle, according to the inspection requirements and the characteristics of the land, it is further divided into multiple regular small plots (such as small rectangles, squares, etc.). Step 2.2: Input the coordinate information of the regular plot into the full coverage path planning algorithm to generate an inspection path. Different path planning algorithms are selected according to the importance of the inspection plots. The basic plots are planned using the ox-tillage method, and the important plots are inspected using the spiral covering method. The total length of the inspection path planned using the ox-tillage method is: The total length of the inspection path planned by the spiral covering method is: In the formula, the forest area to be inspected is a regular plot with a length of L and a width of W, and the field of view of the UAV platform is d. Step 2.3: Input the planned path points into the airborne edge processor, optimize the generated path, generate the specific flight route of the drone, select the appropriate take-off point and entry angle, and control the drone to execute the inspection command.

4. The dual-light based UAV airborne forest fire detection method according to claim 4 is characterized in that: The step 3 is specifically implemented according to the following steps: Step 3.1, construct an infrared threshold segmentation algorithm. By setting two thresholds, the pixels in the image are divided into two parts: the target and the background. And because the color adjustment mode of the infrared image plays a key role in image processing and analysis, different color adjustment modes can highlight different features in the infrared image. After obtaining the infrared image with a specific color adjustment mode, the color space change and threshold segmentation algorithm are used to segment the target image, and then the morphological operation is used to suppress the noise, and finally the sliding window method is used to find the location of the fire point. Step 3.2, build an airborne visible light forest fire detection algorithm. Aiming at the problems of poor real-time performance and low classification accuracy of existing lightweight image classification models, a method for forest fire classification of UAV airborne visible light images based on improved MobileNet v2 is proposed, and a lightweight feature extraction network with MobileNet v2 as the core is constructed. The number of model parameters is reduced by adjusting the width factor; channel and spatial attention modules are embedded in the model to enhance the network's ability to refine features; a multi-scale feature fusion structure is designed to enhance the network's adaptability to scale; and the model parameters are optimized using transfer learning to further improve the model accuracy.

5. The dual-light based UAV airborne forest fire detection method according to claim 5 is characterized in that: The step 4 is specifically implemented according to the following steps: Step 4.

1. Construct an airborne multiple confirmation strategy. In order to further reduce the false detection rate and missed detection rate of airborne forest fire detection, design a system that integrates multi-dimensional data, multi-sensor integration and image semantic context environment. First, detect the fire based on a single data source, and then fuse the detection results of different data sources. By setting thresholds and weights, multiple confirmations are performed. Step 4.2: After multiple confirmations, for different results, the detection results will be further verified to exclude false detections in the corresponding areas. When the detection results of different images are different, the multiple confirmation mechanism will use two different verification strategies to further verify the results: for visible light results, considering the high false detection rate of small target detection, the camera magnification is adjusted, and the visible light magnification is restored after the verification is completed; for infrared detection results, since thermal radiation is directly used for fire judgment, considering the high initial threshold, it is difficult to effectively detect sheet fires, so the segmentation threshold is appropriately lowered during the verification stage, so that sheet fires can be effectively detected.

6. The dual-light based UAV airborne forest fire detection method according to claim 6 is characterized in that: The step 5 is specifically implemented according to the following steps: Step 5.1: After confirming the fire, the drone hovers and controls the gimbal to shoot the fire site from multiple angles and focal lengths. By shooting from multiple angles, a comprehensive view of the fire scene is obtained, including the direction of the fire spread and the size of the affected area. Shooting with different focal lengths can take into account both macro and micro information of the fire scene. Step 5.2: Send back the captured fire images. The computing power of the drone onboard is weak, and it is difficult to process high-resolution, high-quality image data in real time. Therefore, the onboard end compresses the captured images and sends the compressed image data back to the ground station to provide decision support for fire rescue. Step 5.3: After completing the image transmission, the UAV system automatically resumes the inspection mission and continues to inspect the forest area.

Images