Unmanned aircraft smart cruise detection method based on binary cooperation feedback
The smart cruise detection method for unmanned aircraft uses a binary cooperative feedback system with improved FCOS networks and adaptive training to enhance smoke and flame detection accuracy, addressing the challenges of occluded targets in forest fire detection.
Patent Information
- Application Number
- JP2023131096
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-04-27
- Filing Date
- 2023-08-10
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2043-08-10
AI Technical Summary
Traditional forest fire detection methods using unmanned aerial vehicles struggle with low accuracy in distinguishing and tracking smoke and flames due to their intertwined or occluded nature in aerially captured video sequences, leading to inefficiencies and safety concerns.
A smart cruise detection method for unmanned aircraft utilizing a binary cooperative feedback system, which includes a binary detection network with improved FCOS networks for smoke and flame detection, enhanced by CBAM attention modules and deformable convolution, and adaptive training using FPN layer regression, combined with cooperative optimization and reinforcement feedback mechanisms to enhance target identification.
The method significantly improves the accuracy of smoke and flame detection by preventing occlusion and enhancing robustness across varying cruise heights and forest fire scenarios, ensuring effective and precise target identification.
Smart Images

Figure 2024159396000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a smart cruise detection method for unmanned aerial vehicles based on binary cooperative feedback, which belongs to the fields of target identification and computer vision. [Background technology]
[0002] Forests are a rapidly changing environment, and the causes of fires are diverse, making forest fire prevention and detection extremely difficult. Traditionally, forest fire detection has been mainly done manually, requiring a lot of manpower, materials and funds, and also has problems with detection efficiency and safety. With the development of target detection technology and the reduction in the price of unmanned aerial vehicles, patrol monitoring of forest fires using unmanned aerial vehicles has become the main method. However, smoke detection and flame detection in forest fires have different dynamic and spatial characteristics in aerial video sequence images, and the two targets are mutually entangled and occluded, so traditional target detection models cannot effectively detect and track smoke and flames.
[0003] In view of the above, how to accurately and effectively detect smoke and flames in aerial forest images based on the conventional technology has become an urgent problem for those skilled in the art. Summary of the Invention [Problem to be solved by the invention]
[0004] The problem to be solved by the present invention is to provide a smart cruise detection method for unmanned aerial vehicles based on binary cooperative feedback, which solves the problem in the prior art unmanned aerial vehicle patrol monitoring process that the accuracy rate of target identification in unmanned aerial vehicle patrol monitoring images is low due to the relatively large difference in target scale and the mutual occlusion of smoke and flame targets.
[0005] In order to solve the above problems, the present invention employs the following technical solutions.
[0006] A method for smart cruise detection of an unmanned aerial vehicle based on binary cooperative feedback, comprising: Step 1: acquiring images of a target area by an unmanned aerial vehicle, and attaching smoke tabs and flame tabs to the acquired images of the target area to construct a smoke training set and a flame training set; Step 2: Using the smoke training set and the flame training set in step 1, a binary detection network is constructed and trained for two target features of smoke and flame in the image of the target area, where the binary detection network includes a smoke detection branch and a flame detection branch based on the improved FCOS network, respectively; Step 3: using the trained binary detection network to detect a current frame of the video of the target area captured by the unmanned aerial vehicle in real time, and fusing and outputting the detection results of the smoke detection branch and the flame detection branch; and repeating step 3 until the moving image of the target area ends, and detecting the next frame of the moving image of the target area.
[0007] Furthermore, taking the improved FCOS network in step 2 as a reference, the construction process of the improved FCOS network is to add one CBAM attention module between C3 and C4 of the backbone network ResNet, and between C4 and C5, respectively, and replace the original convolution kernel with a deformable convolution in the regression branch.
[0008] In addition, the improved FCOS network adopts an adaptive training decision method of regression in the FPN layer during training, and the specific process is as follows: The regression vector set of the FPN layer, Ψ = [Ψ 1 ,Ψ 2 ,…Ψ i ,…Ψ N[, N, N = 1, Ψi ... For a given target with a pixel size of k × k in the current frame, the default layer P l’ Step 2.2 to determine For the loss values of different FPN layers, the minimum value is judged. The judgment formula is as follows:
number
number
number
number
number
number
number
number
[0009] Furthermore, in the fusion in step 3, specifically, the position frame information and the identified target probability in the detection results of the smoke detection branch and the flame detection branch are both tabulated to the corresponding positions in the current frame.
[0010] In addition, before detecting the next frame of the video of the target region in step 4, the detection result of the current frame is used as prior knowledge to select two feedback mechanisms, namely, collaborative optimization feedback and collaborative reinforcement feedback, and the selection is based on the following reasons: If no target is detected in either the smoke detection branch or the flame detection branch in the detection result of the current frame, it is a smokeless and fireless situation, and the cooperative reinforcement feedback is selected; If a target is detected in the smoke detection branch in the detection result of the current frame, while a target is not detected in the flame detection branch, it is a smokeless / fireless situation, and a collaborative optimization feedback is selected; If no target is detected in the smoke detection branch, but a target is detected in the flame detection branch, this is a fire and smokeless situation, and the cooperative reinforcement feedback is selected; When targets are detected in both the smoke detection branch and the flame detection branch, the minimum target smoke area detected in the smoke detection branch is
number
number
number
number
number
number
[0011] In addition, the specific process of the collaborative optimization feedback is as follows: 1) Current Frame I t-1 Based on the smoke and flame detection information in the (x,y) detection result, the next frame I t Predict the target regions of smoke and flame at (x,y) and define the smoke target region set
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
number
[0012] Furthermore, the specific process of the collaboration reinforcement feedback is as follows: (1)I t-1 If (x,y) is a smokeless and fireless situation, I t When detecting (x, y), the feature weights ω of each layer of the FPN in the smoke detection branch and the flame detection branch of the binary detection network are l Then, adaptive adjustment is performed on the weights,
number
number
number
number
number
number
number
number
number
number
[0013] The present invention provides a computer-readable storage medium having stored thereon one or more programs, the one or more programs including commands which, when executed by a computing device, cause the computing device to perform the method described above.
[0014] The present invention further includes a smart cruise detection apparatus for an unmanned aerial vehicle based on binary cooperative feedback, comprising one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and wherein the one or more programs include commands for performing the methods described above. Effect of the Invention
[0015] The smart cruise detection method for unmanned aerial vehicles based on binary collaborative feedback described in the present invention adopts the above technical solution, and compared with the prior art, it has the following technical advantages:
[0016] 1. The improved FCOS model of the present invention has better discrimination effect on smoke targets and flame targets.
[0017] 2. The present invention uses an improved binary detection network based on the FCOS model to detect smoke and flame as two independent entities, thereby avoiding the problems of missed targets and low positioning accuracy caused by the mutual entanglement or obscuration of smoke and flame.
[0018] 3. Through the two proposed cooperative feedback mechanisms, the present invention improves the robustness of the binary detection network in different cruise heights and different forest fire patrol monitoring scenes. [Brief description of the drawings]
[0019] [Figure 1] 1 is a flow chart of a method for forest fire detection using a binary cooperative network. [Diagram 2] This is an improvement of the regression branch based on deformable convolution. [Diagram 3] An improved FCOS network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0020] The technical solution of the present invention will be further described below in conjunction with the drawings and embodiments.
[0021] The overall structure of the unmanned aerial vehicle smart cruise detection method based on binary cooperative feedback described in the present invention is shown in FIG. 1, and the specific steps are as follows: Step 1: Because the effect of FCOS network in detecting smoke and fire targets is not good, the present invention improves the backbone network, regression branch and training policy of FCOS model based on the target features of smoke and fire.
[0022] The FOCS algorithm employs a pixel-wise regression policy. For a point (x,y) in the FPN feature map, the corresponding coordinates that are mapped to the input image based on a step size s are
number
number
number
[0023] where m Fare the maximum regression ranges of the current feature layer, which are (-1,64), (64,128), (128,256), (256,512), and (512,∞), respectively. These constraints ensure that different feature layers are provided for learning regression among objects of different sizes, thereby avoiding the generation of too many blurred samples. The overall configuration diagram of the improved FCOS network is shown in FIG. 3, and the improvements of the FCOS network are specifically as follows:
[0024] One CBAM attention module is added between C3 and C4 of the backbone network ResNet, and between C4 and C5, respectively, to improve the feature extraction ability for smoke or flame targets with relatively small scales. Then, in the regression branch, the traditional convolution kernel is replaced with a deformable convolution to solve the problem of inaccurate positioning of some flame targets, thereby improving the target detection effect. The specific configuration is shown in Figure 2.
[0025] For the training policy, we propose an FPN layer regression adaptive training decision method, and the related steps are as follows:
[0026] Step 1.1: Layer regression vector Ψ = [Ψ 1 ,Ψ 2 ,…Ψ i ,Ψ N ], N is the total number of tab categories, and Ψ i is a 7-tuple, where the first 5 positions are the default regression range for the FPN layer, the 6th position is the tab category, and the 7th position is a range modification flag bit, which defaults to NULL.
[0027] Step 1.2: For a given target with a pixel size of k × k in the current frame, we calculate the default layer P based on the scale range of the default regression. l’ Determine.
[0028] Step 1.3: For the loss values of different FPN layers, a minimum value judgment is performed. The judgment formula is as follows:
number
number
number
[0029] Here, layer P l The calculation of the loss is as follows:
number
number
number
number
number
number
[0030] Step 1.4:
number
[0031] Step 1.5:
number
[0032]
number
[0033]
number
number
[0034] Step 1.6:
number
[0035] Step 1.7: For targets determined to be weakly relevant (step 1.5), iteratively revise the scale range according to steps 1.2, 1.3 and 1.5, and terminate the iterations after the situations of steps 1.4 and 1.6 are resolved.
[0036] Step 2: Taking the improved FCOS network as a reference, construct and train a binary detection network. The smoke detection branch and the flame detection branch of the binary detection network are respectively the improved FCOS network described in step 1. For training, the smoke detection branch is trained only on the smoke training set, i.e., only performs smoke detection after completing training; similarly, the flame detection branch is trained only on the flame training set, i.e., only performs flame detection after completing training.
[0037] Step 3: Use the trained binary detection network to detect the current frame, and the target frame information and category probabilities obtained in the two detection branches are input into step 4 and step 5, respectively, for further processing.
[0038] Step 4: Based on the relationship between the smoke target and the flame target obtained in step 3, two feedback mechanisms are selected: cooperative optimization feedback and cooperative reinforcement feedback. Here, the cooperative feedback optimization mainly uses the position information of the multiple smoke or flame frames obtained in step 3 as prior knowledge, combines both the speed-based modified Kalman filter and the frame-to-frame difference method to predict the approximate positions of the smoke and flame in the image of the next frame, and performs pre-masking on the approximate positions of the predicted smoke or flame targets in the next frame for the different detection branches of the binary cooperative network, thereby improving the detection accuracy of the binary detection network for smoke or flame. Meanwhile, the cooperative feedback reinforcement, as a post-processing operation, mainly adjusts the weights of the feature maps in the detection branches of the binary network, thereby realizing the use of step 3 to transfer the results and improve the detection accuracy of the same type of target in the next frame.
[0039] Regarding the selection of the two feedback mechanisms, when no target is detected in either the smoke detection branch or the flame detection branch in the binary detection network, i.e., the situation is smokeless and fireless, the cooperative reinforcement feedback is selected; when a target is detected in the smoke detection branch, while no target is detected in the flame detection branch, i.e., the situation is smokeless and fire, the cooperative optimization feedback is selected; when no target is detected in the smoke detection branch, while a target is detected in the flame detection branch, i.e., the situation is fire and fire, the cooperative reinforcement feedback is selected; when a target is detected in both the smoke detection branch and the flame detection branch, the minimum target smoke area detected in the smoke detection branch is
number
number
number
[0040]
number
number
number
[0041] The cooperative optimization feedback mainly uses the position information of multiple smoke or flame frames detected in the current frame as prior knowledge, combines both the speed-based modified Kalman filter and the frame difference method to predict the approximate position of smoke and flame in the image of the next frame, and performs pre-masking on the approximate smoke or flame target position predicted in the next frame for the different detection branches of the binary cooperative network, thereby improving the detection accuracy of the smoke or flame of the binary detection network. The specific steps are as follows:
[0042] Step 1: Construct a modified Kalman filter model based on the velocity to predict the likely positions of the smoke or flame targets in the next frame.
[0043] The Kalman filter models the target motion as a uniform motion, and the target motion state is (p t ,v t ), where p t denotes the target position at time t, and v t pt and the target state can be expressed as a vector form: Gamma t = (x tc , y tc , w t , h t , v tx , v ty , v tw , v th ) T In the formula, x tc , y tc indicates the center coordinate of the detection frame of the target at time t, and w t indicates the width of the detection window, and h t indicates the height of the detection window, and v tx , v ty , v tw , v th indicates the change in each velocity.
[0044] However, in the actual forest fire patrol monitoring process, the unmanned aerial vehicle may sometimes accelerate or decelerate rapidly, so if a constant speed model is used, a relatively large deviation will occur in the target prediction frame. Therefore, in this paper, we use the current aerial shooting speed and height change rate of the unmanned aerial vehicle to calculate the v t The kinetic parameters in are modified and the associated calculations are as follows:
number
[0045] After correcting for the velocity, the prediction equation for the target state and the covariance of the target state using the Kalman filter is:
number
number
number
number
[0046] In the trajectory state update process of the Kalman filter, based on the detection of the current time, the relevant trajectory state is calibrated to obtain a more accurate state estimate, and the state update equation is as follows:
number
number
number
number
[0047] The present invention converts the image of the current frame into I t-1 (x,y), and the image of the next frame is I t (x,y), and after modifying the Kalman filter based on the velocity, I t The predicted smoke target and flame target area sets at (x,y) are
number
number
[0048] Step two: For the color space features of smoke and flame, construct a discriminant function of the first pixel, and subdivide the predicted position obtained by the Kalman filter. The specific steps are as follows:
[0049] 2.1) First, define the number of discriminant functions of the pixel, specifically as follows: In forest scene, the color features of smoke and fire are relatively prominent, smoke generally appears in white, gray and black, while fire generally appears in red-brown, orange. Therefore, the present invention puts forward a first pixel discrimination function for fire and smoke targets based on the color space features of smoke and fire, and the specific calculation is as follows:
number
number
[0050] 2.2) Region Set
number
number
[0051] 3.1) A' obtained from step 2 sm and A' fir Based on I t (x, y) and I t-1 A' at (x,y) sm Image containing region
number
number
[0052] 3.2) Frame Difference Area D sm (x, y) and D fir Taking (x,y), the relevant calculations are:
number
[0053] 3.3) Obtained D sm (x, y) and D fir Based on (x,y), construct a second pixel discriminant function by the threshold judgment and the decision rule of the pixel submitted in step 2.1, D sm (x, y) and D fir The pixel dots in (x, y) whose function value of the second pixel discrimination is 0 are removed, and A' sm Or A' fir Smoke and flame area U sm and U fir where the calculation process of the discriminant function of the second pixel is as follows:
number
[0054] Step 4: t In (x,y)
number
number
number
[0055] The specific applications of the above-described collaborative optimization feedback process in the cases of smoke and no fire, large smoke and small flame, and approximately the same smoke and flame are as follows.
[0056] (1) In the case of smoke and fire, such a situation is I t-1 (x,y) indicates that only the smoke target was detected, but the flame target was not detected. This situation may certainly exist in a real scene, but it is also possible that the flame target is not detected due to interference from the smoke. To avoid the above situation, I t When detecting an (x,y) image,
number
[0057] (2) When the smoke is large and the flame is small, this situation is I t-1 In (x,y), smoke and flame targets are detected simultaneously, and the area of the smoke target is larger than that of the flame target. This corresponds to the relationship between smoke and flame in the actual scene. In order to ensure the stability of the subsequent detection, I tWhen detecting an image of (x,y),
number
number
[0058] (3) When smoke and flames are almost the same, such a situation is I t-1 In (x, y), a smoke target and a flame target are detected at the same time, but there is little difference between the area occupied by the smoke target and the area of the flame target. This situation certainly exists in an actual scene, but it may occur that the positioning of the smoke target is not accurate. In order to improve the accuracy of identifying the smoke target and ensure the stability of the subsequent detection of the flame target, I t When detecting the (x,y) image, you can process it in the same way as when there is a lot of smoke and a small flame.
[0059] Collaborative reinforcement feedback, as a post-processing operation, mainly adjusts the weights of feature maps in the detection branch of the binary network, thereby realizing the use of the propagation of the results of the current frame to correct possible miss results for the same type of target in the next frame. In the target detection process, the FCOS network performs weighted fusion on multiple FPN feature maps and adopts a feature fusion method based on Gaussian weighting. The specific fusion process is as follows:
number
number
number
number
number
[0060] The specific use of cooperative feedback reinforcement in four cases, namely, no smoke and no fire, fire and no smoke, large smoke and small flame, and small smoke and large flame, is as follows:
[0061] (1)I t-1 If (x,y) is judged to be a smokeless and fireless situation, I t When detecting an (x,y) image, the feature weights ω of each layer of the FPN generated by network training in the smoke detection branch and the flame detection branch of the binary detection network are l However, adaptive adjustment is required according to the height of the unmanned aerial vehicle's patrol surveillance, and the specific adjustment is as follows:
[0062] First, a layer weighting adjustment factor that varies based on height
number
number
number
[0063] And it was obtained
number
number
number
[0064] Finally, the binary detection network with adjusted weights is used to detect smoke or fire targets in subsequent frames, and the weights are adjusted again until subsequent frames have fire and no smoke, heavy smoke, small fire, and small smoke and large fire situations.
[0065] (2)I t-1 If (x,y) is judged to be a fire / smokeless situation, it does not match the relationship between smoke and flames in the actual scene, so I tWhen detecting an image of (x,y), feedback enhancement needs to be performed on the flame branch in the binary detection network based on the current flame target detection result, and the related steps are as follows:
[0066] First, the detected flame targets are sorted according to their identification probability, and the flame target with the smallest identification probability is selected as S fir Let the corresponding classification probability be P fir year,
[0067] Then, based on the scale regression range of the smoke target, S fir Scale the FPN layer to the
number
number
[0068] Finally, the fusion weights of the modified FPN layers, ω' l Then, the weights in the original flame detection branch are replaced, thereby improving the detection ability of the flame detection branch for the flame target in the next frame.
[0069] (3)I t-1 If (x,y) is judged to be a situation where there is a lot of smoke and a small flame, I t When detecting an image (x,y), the feature fusion weights ω of each layer of the FPN in the smoke detection branch and the flame detection branch of the binary detection network are l and the Gaussian weighting function G x,y,l It is necessary to make adjustments for each of the standard differences, and the specific adjustments are as follows:
[0070] Smoke detection branch G x,y,l Regarding the correction of I, first, the collaborative feedback reinforcement module t Possible smoke area in the (x,y) frame
number
number
[0071] Smoke detection branch ω l Regarding the correction of, first, as in the case of fire and smokeless, the smoke target S with the smallest discrimination probability is sm , and the corresponding probability P sm and the corresponding FPN layer
number
number
number
[0072] Finally, the revised G x,y,l and the fusion weights ω' of each layer of the FPN layer l is used to adjust the smoke detection branch.
[0073] Similarly, according to the above-mentioned modification method of the smoke detection branch, G in the flame detection branch x,y,l and ω' l Fix.
[0074] (4)I t-1 If (x,y) is determined to be a situation where the smoke is small and the flame is large, this situation occurs because the smoke positioning is not accurate, i.e., the detection is incomplete, and I t When detecting an image (x,y), the layer fusion weights ω of the smoke detection branch in the binary detection network are l In contrast, ω in the flame detection branch in the case of fire and smokeless l while ω in the flame detection branch needs to be adjusted according to the correction scheme for l In contrast, when the smoke is large and the flame is small, ω l Adjustments are made according to the correction method for
[0075] Step 5: Perform fusion on the detection results in the two detection branches obtained by step 3, output the detection category and corresponding probability of the current frame, and then cycle through the operations of steps 3 to 5 for subsequent frames until the video ends.
[0076] Based on the same technical proposal, the present invention further discloses a computer-readable storage medium on which one or more programs are stored, the one or more programs including commands which, when executed by a computing device, cause the computing device to perform the smart cruise detection method for an unmanned aerial vehicle based on binary cooperative feedback described above.
[0077] The present invention further provides an apparatus for detecting smart cruise of an unmanned aerial vehicle based on binary cooperative feedback, comprising one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and arranged to be executed by the one or more processors, and the one or more programs comprise commands for executing the smart cruise detection method for an unmanned aerial vehicle based on binary cooperative feedback described above.
[0078] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the present invention is not limited to the above-described embodiments, and various modifications may be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.
Claims
1. A method for smart cruise detection of an unmanned aerial vehicle based on binary cooperative feedback, comprising: Step 1: acquiring images of a target area by an unmanned aerial vehicle, and attaching smoke tabs and flame tabs to the acquired images of the target area to construct a smoke training set and a flame training set; Step 2: Using the smoke training set and the flame training set in step 1, construct and train a binary detection network for two target features of smoke and flame in the image of the target area, where the binary detection network includes a smoke detection branch and a flame detection branch based on the improved FCOS network, respectively; Step 3: using the trained binary detection network to detect the current frame of the video of the target area captured by the unmanned aerial vehicle in real time, and fusing and outputting the detection results of the smoke detection branch and the flame detection branch; and repeating step 3 until the video of the target area ends, and detecting the next frame of the video of the target area.
2. The method for smart cruise detection of unmanned aerial vehicles based on binary collaborative feedback as claimed in claim 1, characterized in that the improved FCOS network in step 2 is taken as a reference, and the construction process of the improved FCOS network is to add one CBAM attention module between C3 and C4 of the backbone network ResNet, and between C4 and C5, respectively, and replace the original convolution kernel with a deformable convolution in the regression branch.
3. The improved FCOS network adopts an adaptive training decision method of regression of FPN layer during training, and the specific process is as follows: The regression vector set of the FPN layer Ψ = [Ψ 1 , Ψ 2 , …Ψ i , …, Ψ N ], where N is the total number of tab categories, and Ψ i Step 2.1, where is a 7-tuple, where the previous 5 positions are the default regression range of the FPN layer, the 6th position is the tab category, and the 7th position is the range modification flag bit; For a given target with a pixel size of k×k in the current frame, the default layer P l’ and a step 2.2 of determining For the loss values of different FPN layers, a minimum value judgment is performed, and the judgment formula is as follows: [Equation 112] Where: [Equation 113] is the minimum value of the inter-layer loss degree, l denotes the index parameter from layer 3 to layer 7 of the FPN, and L all (l-2), L all (l-1), L all (l), L all (l+1), L all (l+2) are layers P l-2 , P l-1 , P l , P l+1 , P l+2 , [Equation 114] Step 2.3, which shows the loss function where [Number 115] is layer P l’ Step 2.4: if the loss value of the layer regression vector corresponds to the category of the given target, set the seventh position of the layer regression vector to 1, and then perform gradient reversal learning directly according to the default regression scale range for different input scales of the given target; [Equation 116] is layer P l’-1 Or P l’+1 If it matches the loss value of , set the 7th position of the layer regression vector corresponding to the category of the given target to 0, and return to step 2.2 after making the following modifications to the range of the default regression scale: [Equation 117] is layer P l’-1 If the loss value of layer P l’-1 Expand the range of the default regression scale corresponding to (2 8(l’-3) , k) into layer P l’ From the range of the default regression scale corresponding to l’-1 into the range of the default regression scale corresponding to [Number 118] is layer P l’+1 If the loss value of layer P l’+1 Extend the range of the default regression scale corresponding to (k, 2 8(l’-2) ) to layer P l’ From the range of the default regression scale corresponding to l’+1 Step 2.5 of fitting the range of the default regression scale corresponding to [Equation 119] is layer P l’-2 Or Layer P l’-2 and setting the seventh position of the layer regression vector corresponding to the category of the predetermined target to -1 if the loss value of the layer regression vector matches the loss value of the predetermined target, and for such a target, automatically selecting a layer and directly performing gradient inversion learning using the loss degree of each layer of the FPN.
4. The smart cruise detection method for unmanned aerial vehicles based on binary collaborative feedback as described in claim 1, characterized in that in the fusion in step 3, the position frame information and the identified target probability in the detection results of the smoke detection branch and the flame detection branch are both tabbed to the corresponding positions in the current frame.
5. In step 4, before detecting the next frame of the video of the target region, the detection result of the current frame is used as prior knowledge to select one of two feedback mechanisms, namely, collaborative optimization feedback and collaborative reinforcement feedback, and the reasons for the selection are as follows: If no target is detected in either the smoke detection branch or the flame detection branch in the detection result of the current frame, it is a smokeless and fireless situation, and the cooperative reinforcement feedback is selected; If a target is detected in the smoke detection branch in the detection result of the current frame, while a target is not detected in the flame detection branch, it is a smokeless / fireless situation, and a collaborative optimization feedback is selected; If no target is detected in the smoke detection branch, but a target is detected in the flame detection branch, this is a fire and smokeless situation, and the cooperative reinforcement feedback is selected; When targets are detected in both the smoke detection branch and the flame detection branch, the minimum target smoke area detected in the smoke detection branch is [Number 120] The minimum target flame area detected in the flame detection branch is [Number 121] year, [Number 122] The threshold value η min Select based on the magnitude relationship of [Number 123] That is, when the smoke is large and the flame is small, the collaborative optimization feedback and the collaborative reinforcement feedback are selected simultaneously; [Number 124] That is, when the smoke and flame are almost the same, select the collaborative optimization feedback; [Number 125] That is, when the smoke is small and the flame is large, the cooperative reinforcement feedback is selected.
6. The specific process of the collaborative optimization feedback is as follows: 1) Current Frame I t-1 Based on the smoke and flame detection information in the detection result of (x, y), the next frame I t Predict the smoke and flame target regions at (x, y) and use the smoke target region set [Number 126] and flame target area set [Number 127] where: [Number 128] are I t-1 is the number of smoke and flame target regions in the (x,y) detection result, 2) Construct a discriminant function for the first pixel, A sm and A fir The pixel dots in the smoke and flame target regions in which the discriminant function value of the first pixel is 0 are removed, and a new smoke target region set [Number 129] and flame target area set [Number 130] where the discriminant function for the first pixel is: [Number 131] Where: [Number 132] are the discrimination functions for flame and smoke pixels, respectively, (x, y) indicate pixel coordinates, and f H (x, y), f S (x, y) and f I (x, y) respectively indicate the value in the HSI space channel of pixel (x, y), 3) Using the frame difference method, I t A' in (x, y) sm and A' fir Locating smoke and flame targets other than those listed below: S31: I t (x, y) and I t-1 In (x, y), A' sm Images that do not contain [Number 133] and A' fir Images that do not contain [Number 134] and S32: Frame difference area D sm (x, y) and D fir Obtain (x, y), [Number 135] S33: Acquired D sm (x, y) and D fir Construct a second pixel discriminant function for (x, y) and define D sm (x, y) and D fir In (x, y), pixel dots whose second pixel has a discriminant function value of 0 are removed, and A' sm Or A' fir Smoke and flame area U sm and U fir where the calculation process of the discriminant function of the second pixel is as follows: [Number 136] where T is the threshold for determining the motion region, and U sm and U fir are A', respectively. sm and A' fir smoke and flame areas that deviate from the S34: I t In (x, y) [Number 137] Then, masking is performed on each of I t Smoke-elimination masked image corresponding to (x, y) [Number 138] and flame erasing masking images [Number 139] Get 4) I t-1 If (x, y) is a smokeless / fireless situation, then I t When performing (x, y) detection, [Number 140] Let I be the input of the flame detection branch in the binary detection network, and I t Let (x,y) be the input of the smoke detection branch, I t-1 (x, y) is I when the smoke is large and the flame is small t When performing (x, y) detection, [Number 141] Let be the input of the smoke detection branch in the binary detection network, [Equation 142] is the input of the flame detection branch, I t-1 (x, y) is I when smoke and flame are almost the same. t When performing (x, y) detection, [Number 143] Let be the input of the smoke detection branch in the binary detection network, [Number 144] The method for smart cruise detection of unmanned aerial vehicles based on binary cooperative feedback according to claim 5, characterized in that:
7. The specific process of the collaborative reinforcement feedback is as follows: (1) I t-1 If (x, y) is a smokeless and fireless situation, then I t When performing (x, y) detection, the feature weights ω of each layer of the FPN in the smoke detection branch and the flame detection branch in the binary detection network are l Then, adaptive adjustment is performed on the weights, [Number 145] and Where: [Number 146] is the layer weight adjustment factor, [Number 147] h uav indicates the current aerial height of the unmanned aerial vehicle, [Number 148] Let,denote the average aerial height of the corresponding unmanned aerial vehicle during binary detection network training,,l,denotes the number of layers of the current FPN,, [Number 149] is the total number of FPN layers that need to perform feature map fusion, (2) I t-1 If (x, y) is a fire and smokeless situation, I t When detecting (x, y), the feature weights ω of each layer of the FPN of the flame detection branch in the binary detection network are l For this, adaptive adjustment is performed. The specific steps are as follows: First, I t-1 The identification probability of multiple flame targets detected at (x, y) is sorted, and the flame target with the smallest identification probability is selected as S fir Let the corresponding classification probability be P fir year, Next, based on the scale regression range of the current smoke target, S fir Perform FPN layer scale positioning for [Number 150] year, Then, the discrimination probability P fir and the desired probability P of the set target E Using the above, the feature weights of each layer of the FPN of the flame detection branch in the binary detection network are modified, and the modified weights are as follows: [Number 151] (3) I t-1 (x, y) is I when the smoke is large and the flame is small t When performing (x, y) detection, the feature fusion weights ω of each layer of the FPN of the smoke detection branch and the flame detection branch in the binary detection network are l and the Gaussian weighting function G x,y,l The standard deviations are adjusted accordingly. The specific adjustment steps are as follows: G in smoke detection branch x,y,l In terms of correction, the results obtained by enhanced collaborative feedback are [Number 152] By replacing the original standard difference with the standard difference of the domain of G x, y,l The correction of ω in the smoke detection branch l In terms of correction, first, the smoke target S with the smallest discrimination probability is sm , the corresponding probability P sm and obtain the corresponding FPN layer [Number 153] Positioning is performed, and P sm ≧P E If P sm <P E If so, adjust the weights using the following formula: [Number 154] Based on the same adjustment steps as the smoke detection branch, the G x,y,l and ω' l Adjust the (4) I t-1 (x, y) is I when the smoke is small and the flame is large. t When performing (x, y) detection, the layer fusion weights ω of the smoke detection branch in the binary detection network are l In contrast, ω in the flame detection branch in the case of a fire-free and smoke-free situation l While the correction scheme for ω l In contrast, when the smoke is large and the flame is small, ω l The method for smart cruise detection of an unmanned aerial vehicle based on binary cooperative feedback according to claim 5, characterized in that the adjustment is made by a correction scheme for the
8. Current Frame I t-1 Based on the smoke and flame detection information in the detection result of (x, y), the next frame I is calculated based on the velocity-based modified Kalman filter method. t Predict the target area of smoke and flame at (x,y), where the calculation process of the correction of the velocity of the Kalman filter is as follows: [Number 155] Here, v uav denotes the speed of the unmanned aerial vehicle, w′ and h′ denote the width and height of the image of the target area, L denotes the diameter of the receptive field of the aerial photography lens of the unmanned aerial vehicle, ∂ denotes a scaling parameter, Δh denotes the height difference during the ascent or descent process of the unmanned aerial vehicle, and Δt h indicates the time taken for the ascent and descent process of the unmanned aerial vehicle, and w t-1 and h t-1 I t-1 are the width and height of the detection window of the target at (x, y), Δt is the interval in frames, and v tx , v ty , v tw , v th The smart cruise detection method for unmanned aerial vehicles based on binary collaborative feedback as described in claim 6, characterized in that: respectively indicate the center coordinates of the detection frame of the target at the tth time and the width and height velocity values of the detection frame.
9. A computer-readable storage medium having one or more programs stored thereon, 9. A computer-readable storage medium, comprising: one or more programs comprising commands that, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 8.
10. 9. A smart cruise detection apparatus for an unmanned aerial vehicle based on binary cooperative feedback, comprising: one or more processors; one or more memories; and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and wherein the one or more programs include commands for performing the method of any one of claims 1 to 8.
Citation Information
Cited By
Multi-unmanned aerial vehicle inspection track and mobile energy storage terminal scheduling inspection method and system
CN119937625A
