Unmanned aerial vehicle cruise forest fire detection method based on binary cooperative feedback

Abstract

The application provides a UAV cruise forest fire detection method based on binary cooperative feedback, which comprises two parts of a binary detection network and a cooperative feedback mechanism based on a smoke-fire relationship. In the binary detection network, the FCOS network is improved, and a regression range adaptive training determination method based on the FPN layer is proposed. The regression range of each layer of the FPN is adaptively adjusted according to the semantic characteristics of the target, so that each type of target can learn gradient reversal in the appropriate FPN layer during the training stage. In the cooperative feedback mechanism, the correlation characteristics between smoke and fire are fully utilized as knowledge priori to continuously optimize and strengthen the proposed binary cooperative network, so that the continuous and accurate detection and tracking of the fire situation can be realized in the scene of rapid change of smoke and flame. In the forest fire inspection scene, the technical scheme of the application can realize accurate identification and positioning of the fire position, and greatly reduces the input of manpower and material resources in forest fire management.

Description

Technical Field

[0001] This invention relates to a method for detecting forest fires by drone patrol based on binary collaborative feedback, belonging to the fields of target recognition, target identification and computer vision. Background Technology

[0002] Due to the variable environment in forests, the causes of fires are diverse, posing significant challenges to forest fire prevention and detection. Early forest fire detection relied primarily on manual labor, requiring substantial investment of manpower, resources, and funds, and also presenting issues of efficiency and safety. With advancements in target detection technology and reduced costs of drones, drone-based forest fire patrols have become the primary method. However, traditional target detection models struggle to effectively detect and track smoke and flames due to their different dynamic and spatial characteristics in aerial video sequences, as well as the significant overlap and occlusion between the two targets.

[0003] In conclusion, how to accurately and effectively detect smoke and flames in aerial forest images using existing technologies has become a pressing issue for industry professionals. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method for detecting forest fires by drone inspection using binary collaborative feedback. This method solves the problem of low target recognition accuracy in drone inspection images caused by large differences in target scale and mutual occlusion between smoke and flame targets during drone inspection.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A method for detecting forest fires using unmanned aerial vehicles (UAVs) based on binary collaborative feedback includes the following steps:

[0007] Step 1: Obtain target area images from drone aerial photography, and label the obtained target area images with smoke and flame respectively to construct smoke training sets and flame training sets;

[0008] Step 2: Using the smoke and flame training sets from Step 1, construct and train a binary detection network based on the features of smoke and flame in the target area image; wherein the binary detection network includes a smoke detection branch and a flame detection branch based on the improved FCOS network respectively.

[0009] Step 3: Use the trained binary detection network to detect the current frame of the target area video acquired in real time by the UAV aerial photography, and output the detection results of the smoke detection branch and the flame detection branch after fusion.

[0010] Step 4: Repeat step 3 to detect the next frame of the target area video until the target area video ends.

[0011] Furthermore, in step 2, the improved FCOS network is used as a benchmark. The construction process of the improved FCOS network is as follows: an attention module of CBAM is added between C3 and C4 and between C4 and C5 of the backbone network ResNet; deformable convolutions are used to replace the original convolution kernels in the regression branch.

[0012] Furthermore, the improved FCOS network employs an FPN layer regression adaptive training method during training, the specific process of which is as follows:

[0013] Step 2.1: Create the FPN hierarchical regression vector set , Total number of label categories; It is a 7-tuple, with the first 5 positions being the default regression range of the FPN level, the 6th position being the label category, and the 7th position being the range modification flag;

[0014] Step 2.2: For a pixel in the current frame with a size of... Given a target, determine its default level based on the default regression scale range. ;

[0015] Step 2.3: Determine the minimum loss value for different FPN levels using the following formula:

[0016]

[0017] in, To minimize the loss between levels, This represents the index parameters of FPN from level 3 to level 7. , , , , Representing levels , , , , , The loss function for the FPN layer;

[0018] Step 2.4, if with hierarchy If the loss values ​​are consistent, the 7th position of the hierarchical regression vector corresponding to the given target category will be set to 1, and subsequent different input scales for this given target will be directly learned by gradient reversal according to the default regression scale range.

[0019] Step 2.5, if with hierarchy or If the loss value is consistent with the given target category, then set the 7th position of the hierarchical regression vector to 0, and modify the default regression scale range as follows, then return to step 2.2:

[0020] like with hierarchy If the loss values ​​are consistent, then the hierarchy will be... The corresponding default regression scale range is expanded, that is... From the hierarchy Extracted from the corresponding default regression scale range and incorporated into the hierarchy. Within the corresponding default regression scale range;

[0021] like with hierarchy If the loss values ​​are consistent, then the hierarchy will be... The corresponding default regression scale range is expanded, that is... From the hierarchy The corresponding default regression scale range is extracted and incorporated into the hierarchy. Within the corresponding default regression scale range;

[0022] Step 2.6, if with hierarchy or hierarchy If the loss values ​​are consistent, the 7th position of the hierarchical regression vector corresponding to the given target category is set to -1, and for this type of target, the loss of each level of FPN is used to automatically select the level for gradient reversal learning.

[0023] Furthermore, the fusion in step 3 specifically involves: the location box information and the identified target probability in the detection results of the smoke detection branch and the flame detection branch are both marked at the corresponding positions in the current frame.

[0024] Furthermore, in step 4, before detecting the next frame of the target region video, the detection result of the current frame is used as prior knowledge to select between two feedback mechanisms: collaborative optimization feedback and collaborative reinforcement feedback. The selection criteria are as follows:

[0025] If neither the smoke detection branch nor the flame detection branch detects the target in the detection results of the current frame, i.e., there is no smoke or fire, then cooperative reinforcement feedback is selected.

[0026] If the smoke detection branch detects the target but the flame detection branch does not detect the target in the detection results of the current frame, which is a case of smoke but no fire, then the collaborative optimization feedback is selected.

[0027] If the smoke detection branch does not detect the target, but the flame detection branch does, which indicates a fire but no smoke, then the coordinated enhanced feedback is selected.

[0028] If both the smoke detection branch and the flame detection branch detect the target, the smallest smoke target area detected in the smoke detection branch is denoted as... The smallest flame target area detected in the flame detection branch is denoted as ,according to , and the set threshold Choose based on the size relationship:

[0029] like In the case of a situation where there is a lot of smoke but little fire, both collaborative optimization feedback and collaborative reinforcement feedback should be selected simultaneously.

[0030] like In cases where the fireworks are of equal quality, then collaborative optimization feedback is selected.

[0031] like In the case of a small amount of smoke and a large amount of fire, the appropriate response is to strengthen the feedback mechanism.

[0032] Furthermore, the specific process of the collaborative optimization feedback is as follows:

[0033] 1) Based on the current frame The detection information of smoke and flame in the detection results is used to predict the next frame using the Kalman filter method. The target areas of smoke and flame are respectively denoted as the smoke target area set. and fire target area set ,in, and They are The number of target areas of smoke and flame in the detection results;

[0034] 2) Construct the first pixel discrimination function and remove... and A new set of smoke target regions is obtained by identifying the pixels whose first pixel discrimination function value is 0 within the smoke and flame target regions. and fire target area set The expression for the first pixel discrimination function is as follows:

[0035]

[0036]

[0037] in, and These are pixel discrimination functions for flames and smoke, respectively. Represents pixel coordinates, , and Representing pixels Values ​​in the HSI spatial channel;

[0038] 3) Using the inter-frame difference method, for middle and The targets outside the smoke and flames are located as follows:

[0039] S31, obtain respectively and Not included Image and and does not include Image and ;

[0040] S32, Obtain the frame difference region and :

[0041] ;

[0042] S33, regarding the obtained and Construct a second pixel discrimination function to remove... and The pixel with a second pixel discrimination function value of 0 is obtained as the deviation. or Areas of smoke and fire and The calculation process of the second pixel discrimination function is as follows:

[0043]

[0044]

[0045] Where T is the threshold for determining the motion region. and Deviation and Areas of smoke and flames;

[0046] S34, in China respectively Region and The area is masked to obtain Corresponding smoke mask image Images of flame mask removal ;

[0047] 4) If If there is smoke but no fire, then proceed with... During the test, As input to the flame detection branch in the binary detection network, As input to the smoke detection branch;

[0048] like When the smoke is plentiful but the fire is small, then proceed with... During the test, As the input to the smoke detection branch in the binary detection network, As input to the flame detection branch;

[0049] like When the situation is similar to that of fireworks, then proceed with... During the test, As the input to the smoke detection branch in the binary detection network, As input to the flame detection branch.

[0050] Furthermore, based on the current frame The detection information of smoke and flame in the detection results is used to predict the next frame using a velocity-corrected Kalman filter method. In the target area containing smoke and flames, the calculation process for the Kalman filter velocity correction is as follows:

[0051]

[0052] in, Indicates the speed of the drone. and Indicates the width and height of the target region image. This indicates the receptive field diameter of the drone's aerial camera lens. This represents the scaling parameter. This indicates the altitude difference of the drone during ascent or descent. This indicates the time spent by the drone during its ascent and descent. and for The width and height of the target detection bounding box. For frame intervals, , , , Then, represents the center coordinates of the target detection box and the velocity values ​​of the width and height of the detection box at time t, respectively.

[0053] Furthermore, the specific process of the collaborative reinforcement feedback is as follows:

[0054] (1) If In the absence of smoke and fire, then proceed with... During detection, the feature weights of each layer of the FPN in the smoke detection branch and the flame detection branch of the binary detection network are adjusted. Adaptive adjustment is performed, and the adjusted weights are:

[0055]

[0056] in, This is a hierarchical weight adjustment factor. , This represents the current drone aerial photography altitude. This represents the average drone aerial altitude corresponding to the training of the binary detection network. This indicates the current level of the FPN. The total number of FPN layers that require feature map fusion;

[0057] (2) If If there is fire but no smoke, then proceed with... During detection, the feature weights of each layer of the FPN in the flame detection branch of the binary detection network are adjusted. To perform adaptive adjustments, the specific steps are as follows:

[0058] First of all, The detection probabilities of multiple flame targets are ranked, and the flame target with the lowest detection probability is denoted as... The corresponding recognition probability is denoted as ;

[0059] Secondly, based on the current target scale regression range of the smoke, Perform FPN layer scale localization, and record the localization to the FPN layer as ;

[0060] Subsequently, using recognition probability and setting the expected probability of the target The feature weights of each layer in the FPN of the flame detection branch of the binary detection network are corrected as follows:

[0061] ;

[0062] (3) If When the smoke is plentiful but the fire is small, then proceed with... During detection, the feature fusion weights of each layer of the FPN in the smoke detection branch and the flame detection branch of the binary detection network are adjusted. Gaussian weighted function The standard deviations are adjusted accordingly, and the specific adjustment steps are as follows:

[0063] In the smoke detection branch Correction measures: obtained through collaborative reinforcement feedback The regional standard deviation is used to replace the original standard deviation, thereby achieving the desired effect. Corrections;

[0064] In the smoke detection branch Correction measures: First, identify the smoke target with the lowest recognition probability. Corresponding probability and perform corresponding FPN layers Positioning; when At this time, no weight adjustment is required, if The following formula is used to adjust the weights:

[0065]

[0066] Finally, using the revised And the adjusted fusion weights of each layer of the FPN layer Adjust the smoke detection branch;

[0067] Based on the same adjustment steps as the smoke detection branch, the flame detection branch... and Make adjustments;

[0068] (4) If When the smoke is small and the fire is large, then proceed with... During detection, the weights of the smoke detection branch in the binary detection network are fused hierarchically. According to the flame detection branch under the condition of fire without smoke The modification method has been adjusted, and the flame detection branch has been modified accordingly. According to the flame detection branch under the condition of large smoke and small fire The modification method has been adjusted.

[0069] The present invention provides a computer-readable storage medium for storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the method described above.

[0070] The present invention also provides a drone-based forest fire detection device based on binary collaborative feedback, comprising one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the methods described above.

[0071] The binary collaborative feedback-based UAV forest fire detection method described in this invention, compared with existing technologies, has the following technical advantages:

[0072] 1. The improved FCOS model proposed in this invention has better recognition performance for smoke and flame targets;

[0073] 2. The binary detection network built by this invention based on the improved FCOS model treats smoke detection and flame detection as two independent problems and detects them separately to avoid the problems of missed detection and inaccurate positioning of smoke and flame due to mutual adhesion and occlusion.

[0074] 3. This invention improves the robustness of the binary detection network under different patrol altitudes and different forest fire inspection scenarios by proposing two collaborative feedback mechanisms. Attached Figure Description

[0075] Figure 1 This is a flowchart of a method for forest fire detection using a binary collaborative network;

[0076] Figure 2 Improved regression branching based on deformable convolution;

[0077] Figure 3 It is an improved FCOS network. Detailed Implementation

[0078] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0079] The binary collaborative feedback-based UAV patrol forest fire detection method described in this invention has the following overall structure: Figure 1 As shown, the specific steps are as follows:

[0080] Step 1: Since the FCOS network is not good at detecting smoke and flame targets, this invention improves the backbone network, regression branch and training strategy of the FCOS model based on the target characteristics of smoke and flame.

[0081] The FOCS algorithm employs a pixel-by-pixel regression strategy. For a given point on the FPN feature map... According to step size The coordinates mapped to the input image are . On point If the mapped point falls within any of the ground truth bounding boxes labeled in the input image, then that point... If a sample point is not found, it is considered a positive sample. Otherwise, it is marked as a negative sample. Map back to input image points If a point falls within the real bounding box, then that point is within the real bounding box. The distances to the four sides (top, bottom, left, and right) are denoted as the target regression offset. The calculation is as follows:

[0082]

[0083] like If a point falls within multiple ground truth bounding boxes, it will be labeled as an ambiguous sample. To avoid this, FCOS mitigates the overlap of detection boxes by adding bounding box regression range constraints to the feature layers at each scale, as follows:

[0084] (1) Calculate the regression objective in the current level ;

[0085] (2) Judgment or Does it meet the requirements?

[0086] (3) If satisfied, no regression prediction is performed on this bounding box.

[0087] in, The maximum regression range of the current feature layer is, in order: , , , , This constraint ensures that objects of different sizes are assigned to different feature layers for regression learning, thereby avoiding the generation of too many blurry samples.

[0088] The improved FCOS network overall structure diagram is as follows: Figure 3 As shown, the specific improvements to the FCOS network are as follows:

[0089] A CBAM attention module is added between C3 and C4, and between C4 and C5 in the ResNet backbone network, to enhance the feature extraction capability for small-scale smoke or flame targets. Secondly, deformable convolutions are used instead of conventional convolution kernels in the regression branch to address the inaccurate localization of some flame targets, resulting in better target detection performance. The specific structure is as follows: Figure 2 As shown.

[0090] Regarding the training strategy, a method for determining adaptive training of FPN layer regression is proposed, and the relevant steps are as follows:

[0091] Step 1.1: Create hierarchical regression vectors , The total number of label categories. It is a 7-tuple, where the first 5 positions are the default regression range of the FPN level, the 6th position is the label category, and the 7th position is the range modification flag, which defaults to NULL;

[0092] Step 1.2: For a pixel of size in the current frame Given a target, determine its default level based on the scale range of the default regression. .

[0093] Step 1.3: Determine the minimum loss value for different FPN levels using the following formula:

[0094]

[0095] in, To minimize the loss between levels, This represents the index parameters of FPN from level 3 to level 7. , , , , Representing levels , , , , , This is the loss function for the FPN layer.

[0096] Among them, hierarchy The loss is calculated as follows:

[0097]

[0098]

[0099]

[0100]

[0101] in, , and Representing levels Upper position The predicted classification score, predicted bounding box location, and predicted centrality are given above. The number of positive samples. For classification purposes, The coordinates of the annotation box The centerness of the true bounding box. This is the sample discrimination function, which is 1 for positive samples and 0 for negative samples. For a given target pixel At the level Total loss value, For classification loss, Focal loss is used to address the imbalance between positive and negative samples. For center-ness loss, binary cross-entropy is used. The regression loss is represented using the crossover ratio loss.

[0102] Step 1.4, if with hierarchy If the loss values ​​are consistent, it is considered that the semantic feature extraction performance of the target at different FPN levels is strongly correlated with the regression range of different FPN levels. Therefore, the regression vector corresponding to the input target category is set to... The 7th position is set to 1, and subsequent gradient reversal learning is directly performed according to the FPN regression level range for different input scales of this target.

[0103] Step 1.5, if with hierarchy If the loss values ​​are inconsistent, they need to be judged separately:

[0104] when with hierarchy or If the loss values ​​are consistent, then the target is considered to be weakly correlated with the regression at different levels of the FPN, and the corresponding hierarchical regression vector of the target category is... The 7th position is set to 0, and the scale limit range is modified as follows:

[0105] like and If the loss values ​​are consistent, then... The corresponding regression range is expanded, that is... from Extracted from the regression range and incorporated into middle;

[0106] like and If the loss values ​​are consistent, then... The corresponding regression range is expanded, that is... from Extracted from the regression range and incorporated into middle.

[0107] Step 1.6, if with hierarchy or If the loss values ​​are consistent, meaning the interval between the feature layer with the minimum loss and the current feature layer is greater than 1, then the hierarchical scaling regression strategy of FPN is considered unsuitable for this type of target. Therefore, the hierarchical regression vector corresponding to the target category is changed. The 7th position is set to -1, and for this type of target, the loss of each layer of FPN is used to automatically select the layer for gradient reversal learning.

[0108] Step 1.7: For targets that are determined to be weakly correlated (Step 1.5), repeatedly adjust the scale range according to Step 1.2, Step 1.3 and Step 1.5 until the situations in Step 1.4 and Step 1.6 occur, and then the iteration terminates.

[0109] Step 2: Using the improved FCOS network as a benchmark, construct and train a binary detection network: The smoke detection branch and the flame detection branch of the binary detection network are respectively an improved FCOS network described in Step 1; In terms of training, the smoke detection branch is only trained on the smoke training set, that is, after training, only smoke detection is performed. Similarly, the flame detection branch is only trained on the flame training set, that is, after training, only flame detection is performed.

[0110] Step 3: Use the trained binary detection network to detect the current frame, and input the target box information and class probability obtained from the two detection branches into Step 4 and Step 5 for subsequent processing.

[0111] Step 4: Based on the relationship between smoke and flame targets obtained in Step 3, select one of two feedback mechanisms: collaborative optimization feedback and collaborative reinforcement feedback. Collaborative optimization feedback uses the positional information of multiple smoke or flame frames obtained in Step 3 as prior knowledge, combining velocity-corrected Kalman filtering and inter-frame difference methods to predict the approximate positions of smoke and flame in the next frame. Depending on the detection branch of the binary collaborative network, the predicted approximate smoke or flame target positions are masked in advance in the next frame, thereby improving the detection accuracy of the binary detection network. Collaborative reinforcement feedback, as a post-processing operation, mainly adjusts the feature map weights in the detection branch of the binary network, thereby using the results passed from Step 3 to enhance the detection accuracy of the same type of target in the next frame.

[0112] Regarding the choice between the two feedback mechanisms: if neither the smoke detection branch nor the flame detection branch in the binary detection network detects the target (i.e., no smoke or fire), cooperative reinforcement feedback is selected; if the smoke detection branch detects the target but the flame detection branch does not (i.e., smoke present but no fire), cooperative optimization feedback is selected; if the smoke detection branch does not detect the target but the flame detection branch does, (i.e., fire present but no smoke), cooperative reinforcement feedback is selected; if both the smoke detection branch and the flame detection branch detect the target, the smallest smoke target area detected by the smoke detection branch is denoted as... Similarly, the smallest flame target area detected in the flame detection branch is denoted as... ,according to , and the set threshold The size relationship is determined as follows:

[0113] like This refers to a situation where there is a lot of smoke but little fire, in which case both feedback mechanisms are selected.

[0114] like This means that the situation is similar to that of fireworks, in which case collaborative optimization feedback is selected;

[0115] like This refers to a situation where there is little smoke but a large fire, in which case collaborative reinforcement feedback should be selected.

[0116] The collaborative optimization feedback mainly uses the positional information of multiple smoke or flame frames detected in the current frame as prior knowledge. It combines velocity-corrected Kalman filtering and inter-frame difference methods to jointly predict the approximate positions of smoke and flames in the next frame. Furthermore, depending on the different detection branches of the binary collaborative network, it pre-masks the predicted approximate smoke or flame target positions in the next frame, thereby improving the accuracy of the binary detection network in detecting smoke or flames. The specific steps are as follows:

[0117] Step 1: Construct a velocity-corrected Kalman filter model to predict the possible location of smoke or flame targets in the next frame.

[0118] Kalman filtering models the target motion as uniform motion, and the target's motion state is represented as... ,in Indicates the target position at time t. express The velocity of each parameter and the target state can be represented in the following vector form:

[0119]

[0120] In the formula, , This represents the center coordinates of the target detection box at time t; Indicates the width of the detection frame; Indicates the height of the detection frame; , , , These represent their respective velocity changes.

[0121] However, in actual forest fire patrols, drones often experience sudden acceleration and deceleration. Therefore, using a uniform velocity model would result in significant deviations in the predicted bounding box. This paper utilizes the current drone aerial photography speed and altitude change rate to... The various speed parameters are corrected, and the relevant calculations are as follows:

[0122]

[0123] In the formula, Indicates the speed of the drone. and This indicates the width and height of the aerial image. This indicates the receptive field diameter of the drone's aerial camera lens. This represents the scaling parameter. This indicates the altitude difference of the drone during ascent or descent. This indicates the time spent by the drone during its ascent and descent. and This represents the width and height of the target detection bounding box from the previous frame. The frame interval.

[0124] After velocity correction, the Kalman filter is used to predict the target state and its covariance as follows:

[0125]

[0126] In the formula, express The predicted state of the target at any given time. express The optimal estimate at time -1 The state transition matrix represents the target's motion parameter matrix. for The motion noise at time -1 is usually represented by Gaussian white noise with zero mean. , , They are respectively , and The covariance.

[0127] In the trajectory state update process of Kalman filtering, based on the detection at the current moment, the associated trajectory state is corrected to obtain a more accurate state estimate. The state update equation is as follows:

[0128]

[0129] In the formula, express Optimal estimate of the target at any given time. for Detect the mean vector at any time. Represents the observation transition matrix, The Kalman gain is calculated as follows:

[0130]

[0131] In the formula, The noise matrix of the detector is a... The diagonal matrix is ​​given, with the values ​​on the diagonal representing the center point coordinates and the noise levels (width and height). The Kalman filter uses the above formula to estimate the optimal transition state of the current target motion estimate and updates it. covariance matrix This process is repeated sequentially to estimate the target's position in the next moment based on the target detection information at the current moment.

[0132] This invention records the image of the current frame as The image of the next frame is denoted as Based on velocity-corrected Kalman filtering, The predicted sets of smoke and flame target areas are denoted as follows: and ,in and They are in The number of smoke and flame targets detected. However, unlike rigid targets such as trees, smoke and flames are non-rigid targets that are likely to undergo large deformations in a short period of time. That is, some smoke or flame targets may deviate from the trajectory predicted by the Kalman filter due to non-rigid deformation. Therefore, it is necessary to refine and compensate the predicted area. The specific details are shown in steps two and three.

[0133] Step 2: Based on the color space features of smoke and flame, construct the first pixel discrimination function to refine the predicted position obtained by Kalman filtering. The specific steps are as follows:

[0134] 2.1) Define the number of discrimination functions for the first pixel, as follows:

[0135] In forest scenes, the color characteristics of smoke and flames are prominent. Smoke typically appears as white, gray, and black, while flames are generally brownish-red or orange-red. Therefore, based on the color space characteristics of smoke and flames, this invention proposes a first-pixel discrimination function for targets in smoke and flames, specifically calculated as follows:

[0136]

[0137]

[0138] In the formula, and These are the discrimination functions for flame targets and smoke targets, respectively. Represents pixel coordinates, , and These represent the values ​​of the pixel in the HSI spatial channel.

[0139] 2.2) Traversing the set of regions and Using the first pixel discrimination function in step 2.1, the pixels in each region are judged and discarded. and The new smoke and fire target area is obtained by identifying the pixels whose first pixel discrimination function value is 0 within the smoke and flame target area. and .

[0140] Step 3: Use the inter-frame difference method to analyze the deviation from the output frame. or The location of some smoke or flame targets in the area is as follows:

[0141] 3.1) Based on the results obtained in step two and , obtain and Not included Image of the region and and does not include Image of the region and .

[0142] 3.2) Obtain the frame difference region and The relevant calculations are as follows:

[0143]

[0144] 3.3) Based on the obtained and By using threshold judgment and the pixel judgment rule proposed in step 2.1, a second pixel discrimination function is constructed to eliminate pixels. and The pixel with a second pixel discrimination function value of 0 is obtained as the deviation. or Areas of smoke and fire and The calculation process of the second pixel discrimination function is as follows:

[0145]

[0146]

[0147] In the formula, T is the threshold for judging the motion region. and Deviation and The area was filled with smoke and flames.

[0148] Step Four: In right area or The area is masked to obtain Corresponding desmoked image Or remove the image of flame. .

[0149] The collaborative optimization feedback process described above is applied in the following ways when there is smoke but no fire, a lot of smoke but little fire, or a balance between smoke and fire:

[0150] (1) There is smoke but no fire. This situation indicates that... The system detected only smoke targets but not flame targets, a situation that does exist in real-world scenarios. However, smoke interference can also cause flame targets to be missed. To avoid this, during the... During image detection, As input to the flame detection branch in the binary network, while the smoke detection branch remains... The image input remains unchanged.

[0151] (2) The smoke is thick and the fire is small. This situation indicates that... The system simultaneously detected both smoke and flame targets, with the smoke target occupying a larger area than the flame target, consistent with the smoke-fire relationship in real-world scenarios. To ensure subsequent detection stability, further steps were taken... During image detection, As the input to the smoke detection branch in a binary network, As input for the flame detection branch.

[0152] (3) The fireworks are comparable, this situation indicates that in The system simultaneously detected both smoke and flame targets, but the area occupied by the smoke target was not significantly different from that of the flame target. This situation does exist in real-world scenarios, but it can also lead to inaccurate smoke target localization. To improve the accuracy of smoke target recognition and ensure the stability of subsequent flame target detection, further measures were taken... The image detection process is handled in the same way as when there is a large amount of smoke and a small fire.

[0153] Collaborative reinforcement feedback, as a post-processing operation, primarily adjusts the feature map weights in the detection branch of the binary network, thereby using the results from the current frame to correct potential errors in the next frame for the same type of target. During target detection, the FCOS network performs weighted fusion of multiple FPN feature maps, employing a Gaussian-weighted feature fusion method. The specific fusion process is as follows:

[0154]

[0155] In the formula, For the fused feature maps, The total number of merged FPN layers, for Layer in The feature vector at that location, It is a trainable weight, deterministically generated through a training strategy, used to control the importance of each feature map. It is a Gaussian weighting function used to weight each feature map. The Gaussian weighting function has the following form:

[0156]

[0157] In the formula, and yes The center position of the layer feature map in the spatial dimension. It is the standard deviation of a Gaussian distribution. This function represents the distance from location. Feature maps that are further apart will be assigned smaller weights, while feature maps that are closer together will be assigned larger weights.

[0158] The specific applications of collaborative reinforcement feedback in four scenarios are as follows: no smoke and no fire, fire but no smoke, lots of smoke and small fire, and little smoke and big fire:

[0159] (1) If If the situation is determined to be smokeless and flameless, then proceed with... During image detection, it is necessary to adjust the feature weights of each layer of the FPN in the smoke detection branch and the flame detection branch of the binary detection network, which are generated during network training. The drone's inspection altitude is adjusted adaptively, as follows:

[0160] First, define a hierarchical weight adjustment factor based on height variation. The specific calculation process is as follows:

[0161]

[0162] in, This represents the current drone aerial photography altitude. This represents the average altitude collected by the drone during training using a binary collaborative network. This indicates the current level of the FPN.

[0163] Then, based on what was obtained The weights for hierarchical feature fusion in the smoke detection and flame detection branches of the binary detection network. The weights are reassigned, and the specific calculations are as follows:

[0164]

[0165] in, The total number of FPN layers that require feature map fusion.

[0166] Finally, the weighted binary detection network is used for smoke or flame target detection in subsequent frames until subsequent frames show situations such as fire without smoke, large smoke and small fire, or small smoke and large fire, at which point the weights are adjusted again.

[0167] (2) If When a situation is judged to be a fire without smoke, because it does not match the actual smoke and fire relationship in the scene, then... During image detection, the flame branch in the binary detection network needs to be enhanced based on the detection results of the current flame target. The relevant steps are as follows:

[0168] First, the recognition probabilities of the detected flame targets are ranked, and the flame target with the lowest recognition probability is denoted as... The corresponding recognition probability is denoted as ;

[0169] Secondly, based on the regression range of the smoke target scale... Perform FPN layer scale localization, and record the localization to the FPN layer as ;

[0170] Subsequently, using recognition probability and target expected probability The weights are then adjusted, and the relevant calculations are as follows:

[0171]

[0172] Finally, the corrected FPN hierarchical fusion weights are used. This replaces the weights in the original flame detection branch, thereby enhancing the flame detection branch's ability to detect flame targets in the next frame.

[0173] (3) If If the situation is judged to be characterized by more smoke than fire, then proceed with... During image detection, it is necessary to adjust the feature fusion weights of each layer of the FPN in the smoke detection branch and the flame detection branch of the binary detection network. Gaussian weighted function The standard deviations were adjusted accordingly, as follows:

[0174] In the smoke detection branch Correction measures: First, the collaborative reinforcement feedback module is used to obtain... Areas where smoke may appear in the frame Then, based on Regional standard deviation Calculate, and finally Replace the original standard deviation Thus achieving Corrections.

[0175] In the smoke detection branch Correction measures: First, similar to the case of fire without smoke, identify the smoke target with the lowest recognition probability. Corresponding probability and perform corresponding FPN layers Location; however, based on expected probability and recognition probability For weights The corrections are different, when When this occurs, it means that even in the worst-case scenario of smoke recognition, the probability has reached or exceeded the expected probability. This indicates that the fusion weights of each layer in the FPN layer are suitable and no further adjustments are needed. , indicating the current feature layer The feature map fusion effect still needs to be improved, so the following formula is used for adjustment:

[0176]

[0177] Finally, using the modified The weights of each layer of the FPN layer are merged. Adjust the smoke detection branch.

[0178] Similarly, modify the flame detection branch in the same way as described above. and Make the necessary modifications.

[0179] (4) If When a situation is judged to be characterized by little smoke and a large fire, this usually occurs due to inaccurate smoke localization, i.e., incomplete detection. Therefore, further investigation is needed. During image detection, it is necessary to fuse the weights at different levels in the smoke detection branch of the binary detection network. According to the flame detection branch under the condition of fire without smoke The modification method has been adjusted, and the flame detection branch has been modified accordingly. According to the flame detection branch under the condition of large smoke and small fire The modification method has been adjusted.

[0180] Step 5: Fuse the detection results from the two detection branches obtained in Step 3, and output the detection category and corresponding probability of the current frame. Then, repeat Step 3 to Step 5 in subsequent frames until the video ends.

[0181] Based on the same technical solution, the present invention also discloses a computer-readable storage medium storing one or more programs, the one or more programs including instructions, which, when executed by a computing device, cause the computing device to perform the UAV patrol forest fire detection method based on binary cooperative feedback as described above.

[0182] The present invention also provides a drone-based forest fire detection device based on binary collaborative feedback, comprising one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the drone-based forest fire detection method based on binary collaborative feedback as described above.

[0183] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for detecting forest fire by UAV cruising based on binary cooperative feedback, characterized in that, The binary synergistic feedback is selected according to the synergistic optimization feedback and the synergistic reinforcement feedback two feedback mechanisms according to the smoke and the flame target relationship condition; wherein, the synergistic optimization feedback is to take the position information of a plurality of smoke or flames obtained in advance as knowledge prior, combine the Kalman filter based on speed correction and the interframe difference method to jointly predict the position of smoke and flame in the next frame image, and in view of different detection branches of the binary detection network, the predicted smoke or flame target position is masked in advance in the next frame, so as to improve the detection accuracy of the binary detection network for smoke or flame; the synergistic reinforcement feedback is to adjust the feature map weight in different detection branches of the binary detection network, so as to realize the use of knowledge prior to strengthen the detection accuracy of the next frame for the same type of target; specifically comprising the following steps: Step 1, the target area image obtained by the unmanned aerial vehicle aerial photography, and the target area image obtained is labeled for smoke and flame respectively, and a smoke training set and a flame training set are constructed; Step 2, using the smoke training set and the flame training set in step 1, the features of smoke and flame in the target area image are detected, and the binary detection network is built and trained; wherein, the binary detection network includes a smoke detection branch and a flame detection branch which respectively take the improved FCOS network as a benchmark; Step 3, using the trained binary detection network to detect the current frame of the target area video obtained by the unmanned aerial vehicle aerial photography in real time, and outputting after fusing the detection results of the smoke detection branch and the flame detection branch; Step 4, repeating step 3 to detect the next frame of the target area video until the end of the target area video; In step 2, the improved FCOS network is taken as a benchmark, wherein the improved FCOS network is constructed as follows: a CBAM attention module is added between C3 and C4 and between C4 and C5 of the backbone network ResNet; a deformable convolution is used instead of the original convolution kernel in the regression branch; The improved FCOS network adopts the FPN layer regression self-adaptive training determination method during training, and the specific process is as follows: Step 2.1, Create FPN hierarchy regression vector set , is the total number of label categories; is a seven-tuple, the first 5 positions are the default regression range of FPN hierarchy, the 6th position is the label category, and the 7th position is the range modification flag bit; Step 2.2, for a given target with a pixel size of in the current frame, determine its default level according to the default regression scale range; Step 2.3, the minimum value of the loss value of different FPN layers is judged, and the judgment formula is as follows: ， wherein, is the minimum value of the loss degree between the levels, represents the index parameter of FPN from 3 to 7 levels, , , , , respectively represent the level , , , , , is the loss function of the FPN level; Step 2.4, if with the loss value of the hierarchy consistent, the 7th position of the hierarchy regression vector corresponding to the given target category is set to 1, and subsequent different input scales about this given target directly learn gradient inversion according to the default regression scale range. Step 2.5, if with the loss value of the hierarchy or corresponding to the given target class is set to 0, and after modifying the default regression scale range as follows, return to Step 2.2: like with hierarchy If the loss values ​​are consistent, then the hierarchy will be... The corresponding default regression scale range is expanded, that is... From the hierarchy Extracted from the corresponding default regression scale range and incorporated into the hierarchy. Within the corresponding default regression scale range; If the loss value of the level is consistent with the loss value of the level , the default regression scale range corresponding to the level is expanded, that is, the loss value of the level is stripped from the default regression scale range corresponding to the level and is incorporated into the default regression scale range corresponding to the level . Step 2.6, if with the hierarchical or the loss value of the hierarchical consistent, the 7th position of the hierarchical regression vector corresponding to the given target class is set to -1, and the loss degree of each level of the FPN is directly used for automatic selection of the level for gradient inversion learning for this target.

2. The method according to claim 1, wherein, In step 3, the fusion is specifically: the position box information and the recognized target probability in the detection results of the smoke detection branch and the flame detection branch are labeled in the corresponding position of the current frame.

3. The UAV cruising forest fire detection method based on binary collaborative feedback according to claim 1, characterized in that, In step 4, before detecting the next frame of the target area video, the detection results of the current frame are taken as prior knowledge, and the selection of the synergistic optimization feedback and the synergistic reinforcement feedback two feedback mechanisms is selected, and the selection basis is as follows: If the smoke detection branch and the flame detection branch in the detection results of the current frame do not detect the target, that is, there is no smoke and no fire, then the synergistic reinforcement feedback is selected; If the smoke detection branch detects the target in the detection results of the current frame, and the flame detection branch does not detect the target, that is, there is smoke and no fire, then the synergistic optimization feedback is selected; If the smoke detection branch does not detect the target, and the flame detection branch detects the target, that is, there is fire and no smoke, then the synergistic reinforcement feedback is selected; If both the smoke detection branch and the flame detection branch detect the target, the smallest smoke target area detected in the smoke detection branch is denoted as... The smallest flame target area detected in the flame detection branch is denoted as ,according to , and the set threshold Choose based on the size relationship: If , i.e. the case of large smoke and small fire, then the synergistic optimization feedback and synergistic reinforcement feedback are selected simultaneously; If , i.e. the pyrotechnic equivalent case, then the synergistic optimization feedback is selected; If If the fire is small and the smoke is large, then the synergistic reinforcement feedback is selected.

4. The UAV cruising forest fire detection method based on binary collaborative feedback according to claim 3, characterized in that, The specific process of the synergistic optimization feedback is as follows: 1) According to the current frame The detection information of smoke and flame in the detection result, using Kalman filtering method, predicts the next frame The target area of smoke and flame in the next frame is recorded as smoke target area set And flame target area set Where, And Respectively The number of target areas of smoke and flame in the detection result; 2) constructing a first pixel discrimination function to eliminate and pixels with a value of 0 in the first pixel discrimination function in the smoke and flame target regions, to obtain a new smoke target region set and a flame target region set wherein the expression of the first pixel discrimination function is as follows: ， ， wherein and are pixel discrimination functions for fire and smoke, respectively, denotes a pixel coordinate, , and denote the values of the pixel on the HSI space channels, respectively. 3) Using interframe differencing to locate in and out smoke and flame targets as follows: S31, respectively acquire and not containing image and , and not containing image and ; S32, acquire frame difference region and : ； S33, regarding the obtained and Construct a second pixel discrimination function to remove... and The pixel with a second pixel discrimination function value of 0 is obtained as the deviation. or Areas of smoke and fire and The calculation process of the second pixel discrimination function is as follows: ， ， wherein T is a motion region judgment threshold value, and are smoke and flame regions deviating from and respectively. S34、in each of the regions and regions is masked, thereby obtaining the corresponding smoke mask image and the flame mask image ; 4) if for the case of smoke without fire, then the process of detection, the signals are used as inputs to the fire detection branch of the binary detection network, and the signals are used as inputs to the smoke detection branch. If For the case of smoke large and fire small, then in the process of In the detection, the As the input of the smoke detection branch in the binary detection network, the As the input of the flame detection branch; If For pyrotechnic equivalent case, then in the process In the detection, the As the input of the smoke detection branch in the binary detection network, the As the input of the flame detection branch.

5. The UAV cruising forest fire detection method based on binary collaborative feedback according to claim 4, characterized in that, The specific process of the synergistic reinforcement feedback is as follows: (1) If In the case of no smoke and no fire, the smoke detection branch and the fire detection branch in the binary detection network are adjusted adaptively. The feature weights of each layer of the FPN in the smoke detection branch and the fire detection branch are adjusted adaptively during detection. The adjusted weights are: ， wherein, is a hierarchical weight adjustment factor, , represents the current UAV aerial height, represents the average UAV aerial height corresponding to the binary detection network training, represents the number of layers of the current FPN, is the total number of FPN layers that need to be fused. (2) If If there is fire but no smoke, then proceed with... During detection, the feature weights of each layer of the FPN in the flame detection branch of the binary detection network are adjusted. To perform adaptive adjustments, the specific steps are as follows: First, the recognition probabilities of the detected multiple flame targets are sorted, and the flame target with the smallest recognition probability is recorded as , and its corresponding recognition probability is recorded as . ; Secondly, according to the current smoke target scale regression range FPN layer scale positioning is performed, and the positioning to the FPN layer is recorded as ; Subsequently, the recognition probability and the target expected probability are set, and the feature weight of each layer of the FPN in the flame detection branch of the binary detection network is corrected, and the corrected weight is as follows: ； (3) If the smoke is large and the fire is small, then the FPN feature fusion weights of each layer in the smoke detection branch and the fire detection branch in the dual detection network are adjusted during detection, and the standard deviations of the Gaussian weighting functions are adjusted, and the specific adjustment steps are as follows: ​​​ In the smoke detection branch Correction measures: obtained through collaborative reinforcement feedback The regional standard deviation is used to replace the original standard deviation, thereby achieving the desired effect. Corrections; In the smoke detection branch Correction aspect: first, the smoke target with the minimum recognition probability is obtained , the corresponding probability , and the corresponding FPN layer positioning is carried out; when , weight adjustment is not required, and if , weight adjustment is carried out by using the following formula: ， Finally, the modified and adjusted FPN layer fusion weights for each layer Adjust the smoke detection branch; Based on the same adjustment steps as for the smoke detection branch, the flame detection branch is adjusted and ; (4) If When the smoke is small and the fire is large, then proceed with... During detection, the weights of the smoke detection branch in the binary detection network are fused hierarchically. According to the flame detection branch under the condition of fire without smoke The modification method has been adjusted, and the flame detection branch has been modified accordingly. According to the flame detection branch under the condition of large smoke and small fire The modification method has been adjusted.

6. The UAV cruising forest fire detection method based on binary collaborative feedback according to claim 4, characterized in that, According to the current frame The detection information of smoke and fire in the detection result is predicted for the next frame using a Kalman filtering method based on speed correction The smoke and fire target area in the middle, wherein the calculation process of Kalman filtering speed correction is as follows: ， wherein, denotes the speed of the UAV, and denotes the width and height of the target region image, denotes the field of view diameter of the UAV aerial shot, denotes the zoom parameter, denotes the height difference of the UAV during the process of ascending or descending, denotes the time spent by the UAV during the process of ascending and descending, and are the width and height of the target detection frame in the middle, is the frame interval, , , , denote the center coordinates of the target detection frame and the speed values of the width and height of the detection frame at the t-th moment, respectively.

7. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that are executable to: The instructions, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 6.

8. A UAV cruising forest fire detection device based on binary cooperative feedback, characterized in that, A computing device comprising one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any one of claims 1 to 6.

Images