Initial forest fire target detection method in unmanned aerial vehicle scene

Abstract

The invention discloses an initial forest fire target detection method in an unmanned aerial vehicle scene, and the method comprises the steps: firstly constructing a YOLOv8n model, replacing a common convolution in a C2f module of a Neck part with a deformable convolution DCN, obtaining a C2fDCN module, enlarging the effective receptive field of the model, and improving the detection capability of the model for small target flames and smog in an initial forest fire; and then the up-sampling of the YOLOv8n model is replaced by a DySample module, so that the detection capability of the model for complex conditions such as illumination variation and shielding is improved. And inputting the screened initial forest fire image into the improved YOLOv8n model, and carrying out model training. And finally, deploying the trained model on an unmanned aerial vehicle, inputting a shot picture into the model when the unmanned aerial vehicle executes forest fire inspection, generating a visual detection result according to the output of the model, and marking the initial forest fire target position existing in the input picture.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and relates to target detection and analysis of unmanned aerial vehicle (UAV) images, and specifically to an initial forest fire target detection method in a UAV scenario. Background Art

[0002] Most forest fires occur in wild environments. The uncertainty and unpredictability after a fire increases the difficulty of extinguishing the fire source. Therefore, in the early stages of a forest fire, when the flames are small and spread slowly, it is the best time to extinguish the fire. Traditional forest fire detection methods mainly include manual patrols, thermal sensor-based methods, and satellite remote sensing-based methods. Among them, manual patrols require forest rangers to patrol the forest continuously, and the patrol area is limited, the efficiency is extremely low, and it is difficult to achieve all-weather monitoring, and it is difficult to detect early forest fires. The thermal sensor-based method has a high accuracy rate in monitoring forest fires and a low false alarm rate. However, its monitoring range is limited and the coverage cost is high. The satellite remote sensing-based method has the advantages of wide coverage and large detection area. However, its cost is high, and it has limitations in flexibility and timeliness, making it difficult to monitor early forest fires.

[0003] With the development of drones and deep learning technology, forest fire detection methods based on drones and target detection technology have attracted more and more attention. They are low-cost, safe, fast and highly flexible. They can also perform long-term, boring and repetitive tasks without human constraints. However, the current target detection algorithms cannot be directly used on drone platforms due to their large amount of calculations and parameters.

[0004] YOLOv8n is a lightweight version of YOLOv8, which is more suitable for drone platforms. However, the effective receptive field of the model is small, and it is easy to ignore the small-scale flames and smoke targets of early forest fires, resulting in poor detection of early forest fires. In addition, when dealing with complex scenes such as lighting changes and occlusion, the detection accuracy is significantly reduced. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a method for detecting early forest fire targets in a drone scenario. According to the target characteristics of early forest fires, a feature fusion module C2f_DCN based on deformable convolution is proposed to increase the receptive field of the model. The upsampling module of the model is improved so that it can dynamically select continuous feature maps generated by bilinear interpolation according to the input features, better cope with complex situations such as illumination changes and occlusion, and reduce the amount of calculation and parameters of the model.

[0006] A method for detecting early forest fire targets in a drone scenario, the specific steps are as follows: Step 1: Build an initial forest fire dataset Select images of early forest fires, annotate them with corresponding information, and construct a training set.

[0007] Step 2: Build the C2f_DCN module Build a YOLOv8n model and replace the C2f module in the Neck part with the C2f_DCN module based on the deformable convolution DCN. ,The C2f_DCN module first divides it into two parts, one part is passed through n Bottleneck_DCN modules for feature extraction, and the other part is directly spliced ​​with the output of n Bottleneck_DCN modules, and the spliced ​​result is input into CBS for calculation, and the output of the C2f_DCN module is obtained.

[0008] The Bottleneck_DCN module replaces the ordinary convolution of the Bottleneck part in the C2f module with DCN.

[0009] The core operator of the deformable convolution DCN is a dynamic sparse convolution with a window size of 3×3: (1) in, is the given input feature of the deformable convolution DCN, is the current pixel position, N is the total number of aggregation groups, represents the position-independent projection weight in the nth aggregation group, It is The modulation scalar of the kth sampling point in the group, represents the sliced ​​input feature map, For the Group grid sampling positions The corresponding offset.

[0010] Step 3: Build the DySample module The original upsampling module of YOLOv8n is replaced by the DySample module based on the dynamic upsampling method. The DySample module generates offsets related to the input features through learning, dynamically adjusts according to the local structure of the input features, guides bilinear interpolation to select new sampling positions outside the original grid positions, and generates more accurate high-resolution feature maps.

[0011] Step 4: Model training The training set data is input into the YOLOv8n model improved by steps 2 and 3 to obtain the initial forest fire target detection model.

[0012] Step 5: Initial forest fire target detection The model trained in step 4 is deployed on the drone. When the drone performs forest fire inspections, the captured images are input into the model, and visual detection results are generated based on the output of the model, marking the initial forest fire target locations in the input images.

[0013] The present invention has the following beneficial effects: 1. Taking into account the characteristics of small flames and smoke targets in the early stage of forest fires, the DCN based on deformable convolution is used to improve the model feature extraction and feature fusion module C2f to increase the effective receptive field of the model, thereby enhancing the model's ability to detect small target flames and smoke in the early stage of forest fires.

[0014] 2. Use the DySample method based on dynamic upsampling to improve the upsampling part in YOLOv8n. The upsampling process of the DySample module can be dynamically adjusted according to the local structure of the input features, especially when processing early forest fire images with weak firelight and small-scale smoke features, thereby generating more accurate high-resolution feature maps for the subsequent feature fusion process, ultimately improving the model's detection performance for forest fires and improving the robustness of the model.

[0015] 3. The C2f_DCN module enhances feature extraction and fusion so that the input feature map contains richer and more accurate information, while the DySample module can dynamically adjust the upsampling process according to the input features and retain more information during the upsampling process. The two modules act on different parts of the model and jointly improve the overall understanding ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of traditional convolution operation; Figure 2 Schematic diagram of deformable convolution DCN; Figure 3 Schematic diagram of the structure of the C2f_DCN module constructed in the embodiment; Figure 4 Schematic diagram of the Bottleneck_DCN module structure constructed in the embodiment; Figure 5 An image of an initial forest fire that is input to the model in the embodiment; Figure 6 It is a heat map of the detection results output by the YOLOv8n model in the embodiment; Figure 7 This is a heat map of the detection results output by the YOLOv8n model improved by the C2f_DCN module in the embodiment; Figure 8 Schematic diagram of the structure of the sampling point generator module constructed in the embodiment; Fig. 9 A schematic diagram of a DySample module constructed in the embodiment; Fig.10 Schematic diagram of the improved YOLOv8n model structure in the embodiment. DETAILED DESCRIPTION

[0017] The present invention will be further explained below with reference to the accompanying drawings; A method for detecting early forest fire targets in a drone scenario, the specific steps are as follows: Step 1: Get the initial forest fire dataset: This embodiment uses the D-Fire dataset as the data source for the training set and the test set. The images in the D-Fire dataset are mainly from the Internet and the surveillance cameras of the Put out the Fire! ("Apaga o Fogo!") website, as well as some synthetic images of long-distance fires. The dataset has great diversity in smoke type (color, shape, density, size), fire source (forest, industry, building), ignition source (impact, electric spark, smoking articles, fuel ignition articles), camera perspective (perspective of drones and fire-fighting aircraft, perspective of environmental monitoring towers, perspective of ordinary citizens), confusing elements and noisy environments (clouds, fog, flashing objects).

[0018] In order to conduct experiments on early forest fire detection, this embodiment selected 7537 images as early forest fire images based on the size of flames and smoke relative to the background, and randomly divided them into training set, validation set and test set in a ratio of 6:2:2.

[0019] Step 2: Build the C2f_DCN module Build a YOLOv8n model. In the YOLOv8n model, the feature extraction and fusion module C2f uses the following Figure 1 In order to increase the effective receptive field of the model and enhance the model's ability to detect small target flames and smoke in the early stages of forest fires, this application proposes a module C2f_DCN based on deformable convolution DCN (Deformable Convolution):

[0020] Step 2.1: Figure 2 As shown in Figure 1, the core operator of the deformable convolution-based DCN is a dynamic sparse convolution with a window size of 3×3. Figure 2 As can be seen from the figure, DCN has a flexible sampling offset and can dynamically learn the appropriate receptive field from the given data, thus effectively improving the detection capability of small target flames and smoke.

[0021] like Figure 3As shown in FIG. 1 , the C2f_DCN module includes a separation operation, n consecutive Bottleneck_DCN modules, a concatenation operation and a CBS module. ,The C2f_DCN module first divides it into two parts, one part is passed through n Bottleneck_DCN modules for feature extraction, and the other part is directly spliced ​​with the output of n Bottleneck_DCN modules, and the spliced ​​result is input into CBS for calculation, and the output of the C2f_DCN module is obtained.

[0022] like Figure 4 As shown in the figure, the Bottleneck_DCN module performs two consecutive deformable convolution operations on the input features and then concatenates them with the input features. The core operator of the deformable convolution DCN is a dynamic sparse convolution with a window size of 3×3:

[0023] (1) in, is the given input feature of the deformable convolution DCN, is the current pixel position, N is the total number of aggregation groups, represents the position-independent projection weight in the nth aggregation group, It is The modulation scalar of the kth sampling point in the group, represents the sliced ​​input feature map, For the Group grid sampling positions The corresponding offset.

[0024] In order to prove the effectiveness of the C2f_DCN module, the YOLOv8n model before and after improvement is used to test the Figure 5 The early forest fire image shown in the figure is tested. The results are as follows Figure 6 , 7 As shown in the figure, the original YOLOv8n model pays less attention to small target flames and smoke, while the improved YOLOv8n model has a larger effective receptive field and focuses more on the center point of the target, making the detected bounding box more accurate and filtering out noise interference on irrelevant backgrounds.

[0025] Step 3: Build the DySample module The original upsampling module of the YOLOv8n model is replaced by the DySample module based on the dynamic upsampling method to further improve the model's detection accuracy and robustness for early forest fires: Step 3.1, construct Figure 8The sampling point generator module SPG (Sampling Point Generator) shown in the figure. For a feature of size C×H×W Figure X , given an upsampling scale factor s and a grouping sampling factor g. Features Figure X First, through two input channels, C and output channels, 2gs 2 The linear layer , ,in The results were 0.5 times After operation, Multiply the output of 2 ×H×W feature map, and then reshaped into an offset O of size 2g×sH×sW after Pixel Shuffle:

[0026] (2) Given the original sampling grid G ​​of size 2g×sH×sW, after adding the offset O, we get the sampling set S: (3) In this embodiment, the upsampling scale factor s is set to 2, and the group sampling factor g is set to 4.

[0027] Step 3.2: Fig. 9 As shown, in the DySample module, for the input feature X, after the sampling point generator module SPG obtains the sampling set S, the input feature X is grid sampled and the upsampled feature Y is output: (4) The upsampling process of the DySample module can be dynamically adjusted according to the local structure of the input features, providing more accurate high-resolution feature maps for the subsequent feature fusion process of the YOLOv8n model, thereby improving the model's detection performance for forest fires and the robustness of the model.

[0028] Step 4: Model training like Fig.10 As shown in the figure, the training set data is input into the YOLOv8n model improved by steps 2 and 3, the training rounds are set to 300 rounds, the batch size of each training input is 32, and the image size is uniformly reshaped to 640 × 640. After the training is completed, the trained weights are saved to obtain the initial forest fire target detection model.

[0029] Step 5: Initial forest fire target detection In order to verify the beneficial effects of the improved network model, an ablation experiment was conducted using the test set data. The results are shown in Table 1:

[0030] Table 1 In Table 1, P represents precision; R represents recall; mAP 50 Represents the average precision at the 50% intersection over union (IoU) threshold; mAP 75 Represents the average precision under the 75% intersection-over-union threshold; mAP 50-95 The average precision calculated in the range of 50% to 95% IoU threshold with a step size of 0.05; Params indicates the number of model parameters in millions; FLOPs indicates the number of floating point operations in billions. (↑) indicates that the higher the value, the better, and (↓) indicates that the lower the value, the better.

[0031] As can be seen from Table 1, after using the C2f_DCN module to improve the YOLOv8n model, mAP 50 、mAP 75 The indicators have increased significantly, especially mAP 75 After using the DySample module to improve the YOLOv8n model, the number of parameters and the amount of calculation of the model remained basically unchanged, and except for a slight decrease in the accuracy index, all other indicators were improved. After using the C2f_DCN module and the DySample module to improve the YOLOv8n model, the number of parameters and the amount of calculation were reduced, making the model have a faster detection speed, and all other indicators were improved, including accuracy, mAP, etc. 50 、mAP 75 、mAP 50-95 The improvements were 1.5%, 1.3%, 3.2% and 1.1% respectively, making the model have better detection accuracy.

[0032] In addition, in order to verify the advantages of the proposed method in early forest fire detection, the improved model of this method is also compared with YOLOv8n, RTMDet, YOLOv6-v3.0, and YOLOv7. The specific results are shown in Table 2: Table 2 As can be seen from Table 2, the improved model has achieved a good balance between detection accuracy and recall. In terms of accuracy (P) and recall (R), this method is only inferior to YOLOv6-v3.0 in recall, but all other indicators are much better than YOLOv6-v3.0. In terms of locating the fire location, the improved model has a better detection accuracy at an IoU threshold of 50% (mAP 50 )、75%(mAP 75 ) and between 50%-95% (mAP 50-95 ) are higher than all the comparison models, which shows the accuracy of this method in locating the fire location. In terms of model parameter quantity (Params) and computational complexity (FLOPs), this method is superior to all other models in terms of parameter quantity and second only to YOLOv7 in terms of computational complexity. However, by comparing the experimental results, it can be seen that this method is much higher than YOLOv7 in all other indicators.

Claims

1. A method for detecting early forest fire targets in a drone scenario, selecting images of early forest fires, annotating corresponding information, and constructing a training set, characterized by: Build a YOLOv8n model, replace the ordinary convolution in the C2f module of the Neck part with deformable convolution DCN, and replace the upsampling module with the DySample module; use the training set data to train the improved YOLOv8n model; deploy the trained model on a drone, and when the drone performs forest fire inspections, the captured images are input into the model, and visual detection results are generated based on the output of the model, marking the target locations of early forest fires in the input images.

2. The method for detecting an initial forest fire target in a drone scenario as claimed in claim 1, characterized in that: We use the D-Fire dataset as a source of training data, from which we select images representing early forest fires based on the size of the flames and smoke relative to the background.

3. The method for detecting an initial forest fire target in a drone scenario as claimed in claim 1, characterized in that: Replace the ordinary convolution in the C2f module of the Neck part with the deformable convolution DCN to construct the C2f_DCN module; For input features ,The C2f_DCN module first divides it into two parts, one part is passed through n Bottleneck_DCN modules for feature extraction, and the other part is directly spliced ​​with the output of n Bottleneck_DCN modules, and the spliced ​​result is input into CBS for calculation, and the output of the C2f_DCN module is obtained.

4. The method for detecting an initial forest fire target in a drone scenario as claimed in claim 3, characterized in that: The core operator of the deformable convolution DCN is a dynamic sparse convolution with a window size of 3×3: (1) in, is the given input feature of the deformable convolution DCN, is the current pixel position, N is the total number of aggregation groups, represents the position-independent projection weight in the nth aggregation group, It is The modulation scalar of the kth sampling point in the group, represents the sliced ​​input feature map, For the Group grid sampling positions The corresponding offset.

5. The method for detecting an initial forest fire target in a drone scenario as claimed in claim 1, characterized in that: The DySample module first generates a sampling set S for the input feature X through the sampling point generator module SPG, then performs grid sampling on the input feature X and outputs the upsampled feature Y: (2)。 6. The method for detecting an initial forest fire target in a drone scenario as claimed in claim 5, characterized in that: The sampling point generator module SPG has an upsampling range factor of s and a group sampling factor of g. For a feature map X of size C×H×W, it first passes through two input channels of C and output channels of 2gs. 2 The linear layer , ,in The results were 0.5 times After operation, Multiply the output of 2 ×H×W feature map, and then reshaped into an offset O of size 2g×sH×sW after pixel reorganization: (3) Given the original sampling grid G ​​of size 2g×sH×sW, after adding the offset O, we get the sampling set S: (4)。 7. The method for detecting an initial forest fire target in a drone scenario as claimed in claim 6, characterized in that: The upsampling scale factor s is set to 2 and the grouping sampling factor g is set to 4.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images