Unmanned aerial vehicle cluster fire point real-time detection and positioning method and system

Abstract

The invention provides an unmanned aerial vehicle cluster fire point real-time detection and positioning method and system. The method comprises the following steps: acquiring a visible image and an infrared image synchronously shot by a single unmanned aerial vehicle, and recorded shooting time and unmanned aerial vehicle pose data; processing the visible image and the infrared image based on threshold segmentation and morphological filtering to obtain a moment image pair; inputting the moment image pair into the trained double-attention segmentation network, and outputting a fire semantic map; analyzing and processing the fire semantic map based on the connected domain, outputting candidate fire points, and obtaining initial geographic coordinates of the candidate fire points based on the pose data of the unmanned aerial vehicle; based on the shooting time, tracking the initial geographic coordinates of the candidate fire points in real time through Kalman filtering, and outputting a fire point tracking and positioning result of the single unmanned aerial vehicle at the to-be-positioned moment through dynamic compensation of elevation data; and fusing the fire point tracking and positioning results of all the unmanned aerial vehicles in the unmanned aerial vehicle cluster at the to-be-positioned moment to obtain a fire point fusion positioning result at the to-be-positioned moment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of unmanned aerial vehicle fire monitoring, and particularly relates to a method and system for real-time detection and positioning of fire points by a cluster of unmanned aerial vehicles. BACKGROUND

[0002] Existing fire monitoring methods mostly use single-view image segmentation or target detection algorithms, rely on single-view visual analysis or sensor data, and when processing multi-view data from unmanned aerial vehicles, usually only perform simple feature splicing or layer-by-layer fusion, lacking a fine alignment mechanism for view angle differences, scale changes and complex backgrounds, which results in the inability to fully combine the fire field region in high-altitude overhead view and the flame details in the ground close-range image, affecting segmentation accuracy and small fire point detection rate, and causing problems such as missed detection, false alarm rate and the like in complex environments. For example, traditional unmanned aerial vehicle fire detection models based on CNN are prone to missed detection in thick smoke, low light or small fire point scenarios, because the resolution and viewing angle limitations of the camera result in “small fire spots being difficult to capture”. Moreover, most algorithms only use a single sensor, and cannot fully fuse multi-source information such as infrared and visible light, resulting in insufficient robustness. In addition, in complex terrain and side-view angles, due to factors such as terrain undulations, unmanned aerial vehicle side-view angles and sensor time synchronization, there is a significant deviation between the image position and the actual geographical position of the fire point, and existing research has less research on terrain elevation compensation, dynamic fire prediction and multi-sensor spatio-temporal fusion. In the multi-sensor scenario, the timestamps and coordinate systems of different sensors are inconsistent, which also affects data fusion and positioning accuracy.

[0003] Therefore, there is an urgent need for a new technical solution that can accurately segment and identify fire points in complex terrain and small fire point conditions, and provide information support for early detection and timely handling of fires. SUMMARY

[0004] In order to overcome the insufficient accuracy and missed detection phenomenon of fire point identification and positioning in complex terrain and small fire point conditions in the prior art, the present application provides a method and system for real-time detection and positioning of fire points by a cluster of unmanned aerial vehicles, which achieves significant improvement in small fire point segmentation accuracy, positioning accuracy, system robustness and processing efficiency through multi-view attention fusion, non-downsampling network architecture, dynamic tracking and terrain compensation, multi-source spatio-temporal fusion and real-time design, and has extremely high engineering application value and promotion potential.

[0005] According to an aspect of the present application, a method for real-time detection and positioning of fire points by a cluster of unmanned aerial vehicles is provided, comprising:

[0006] obtaining visible images and infrared images synchronously photographed by a single unmanned aerial vehicle, and recording the photographing time and unmanned aerial vehicle pose data;

[0007] The visible image and the infrared image are processed based on threshold segmentation and morphological filtering to obtain a time image pair;

[0008] The time image pair is input into the trained dual attention segmentation network to output a fire semantic map;

[0009] The fire semantic map is processed based on connected domain analysis to output a candidate fire point, and the preliminary geographic coordinates of the candidate fire point are obtained based on the pose data of the unmanned aerial vehicle;

[0010] Based on the shooting time, the preliminary geographic coordinates of the candidate fire point are tracked in real time through Kalman filtering, and the tracking positioning result of the fire point at the to-be-positioned time of a single unmanned aerial vehicle is output through dynamic compensation of the height data;

[0011] The tracking positioning results of the fire point at the to-be-positioned time of all unmanned aerial vehicles in the unmanned aerial vehicle cluster are fused to obtain the fusion positioning result of the fire point at the to-be-positioned time.

[0012] As a further technical solution, the step of obtaining the time image pair comprises:

[0013] The infrared image is processed based on an automatic threshold segmentation method to separate the high-temperature region from the background, and the minimum circumscribed rectangle of all high-temperature regions is calculated, and the infrared image and the visible light image synchronously shot are cropped and size-transformed based on the minimum circumscribed rectangle to obtain the time image pair.

[0014] As a further technical solution, the backbone network of the dual attention segmentation network adopts a residual network, the encoding part processes different types of pictures in the input time image pair through two structurally consistent encoding branches, extracts semantic feature maps of different scales, processes the highest scale semantic feature maps output by the two branches through a fusion convolution layer, and outputs a fusion feature map; the decoding part decodes and processes the fusion feature map, and finally outputs a fire semantic map.

[0015] As a further technical solution, the encoding branch extracts features of the image input by the branch layer by layer through a plurality of encoding layers, generates channel importance weights through a channel attention module after each encoding layer, generates a channel-weighted semantic feature map, and then processes the semantic feature map through a spatial attention module to obtain an attention-enhanced semantic feature map.

[0016] As a further technical solution, the step of obtaining the candidate fire point comprises:

[0017] Based on connected domain analysis, the connected regions in the fire semantic map are extracted, the connected regions in the extracted fire semantic map are removed according to a preset connected region area threshold, and the remaining connected regions are taken as main fire point regions;

[0018] The geometric center of the main fire point region is calculated through image moment, and is recorded as a candidate fire point.

[0019] As a further technical solution, the acquisition process of the fire point tracking positioning result is:

[0020] The preliminary geographic coordinates of the candidate fire points at several time points are sorted in chronological order, and based on Kalman filtering processing, a smooth fire point time sequence position sequence is generated;

[0021] Based on the elevation data compensation, the preliminary geographic coordinates of the candidate fire points at the positioning time are corrected, and the fire point time sequence position at the positioning time output by the Kalman filter is updated by secondary filtering according to the result of the correction processing. The fire point tracking positioning result at the positioning time is output.

[0022] As a further technical solution, the step of fusing the fire point positioning result includes:

[0023] According to the shooting time, the visible images and infrared images of multiple perspectives at the same time stamp are matched;

[0024] The fire point tracking positioning results of several unmanned aerial vehicles at the same time stamp are obtained, and the fire point fusion positioning result at the positioning time stamp is calculated by weighted average according to the respective filter expected covariance size, with the reciprocal of the variance as the weight.

[0025] According to another aspect of the present specification, an unmanned aerial vehicle cluster fire point real-time detection and positioning system is provided, comprising an unmanned aerial vehicle and a data processing end, the unmanned aerial vehicle is used for synchronously shooting visible images and infrared images, and recording shooting time and unmanned aerial vehicle pose data; the data processing end is used for receiving information sent by the unmanned aerial vehicle, and executing steps of a unmanned aerial vehicle cluster fire point real-time detection and positioning method.

[0026] According to another aspect of the present specification, an electronic device is provided, comprising a memory and a processor, the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute a unmanned aerial vehicle cluster fire point real-time detection and positioning method.

[0027] According to another aspect of the present specification, a non-transitory computer readable storage medium is provided, which stores computer instructions, and the computer instructions make the computer execute a unmanned aerial vehicle cluster fire point real-time detection and positioning method.

[0028] Compared with the prior art, the beneficial effects of the present application are that the present application provides a whole process technology from data acquisition, fine segmentation, dynamic tracking to terrain compensation and multi-source fusion, realizes real-time detection and high-precision positioning of unmanned aerial vehicle fire points, and has the following outstanding advantages and significant effects:

[0029] (1) Fine alignment of multi-perspective features with double attention mechanism, significantly improving the small fire point recognition rate

[0030] Traditional CNN-based flame segmentation networks usually use simple concatenation or weighted fusion when fusing multi-view images, which cannot balance the overall flame shape in overhead view and flame details in ground side view. The invention introduces a channel attention and spatial attention dual module: channel attention automatically learns the most discriminative spectral or color channel, highlighting flame-related features; spatial attention assigns weights at the pixel level, enhancing the response to small and irregular fire point areas. This mechanism can adaptively align the feature distribution difference between air and ground, solving the problems of "missing small fire points" and "background false positives".

[0031] (2) End-to-end network architecture without downsampling, preserving details to the maximum extent

[0032] Many semantic segmentation models rely on pooling or downsampling operations to expand the receptive field, but this will weaken small target features and blur boundaries. The segmentation network of the invention completely abandons the pooling layer, uses layer-by-layer convolution, double attention weighting and context enhancement module, and maintains the original resolution. This can not only ensure the capture of global features of large-scale fire field, but also maximize the preservation of small fire point details.

[0033] (3) Kalman filter and terrain compensation linkage tracking, eliminating spatial and temporal errors

[0034] The projection error caused by the pitch angle and terrain undulation of the side view of the unmanned aerial vehicle often leads to the deviation of the fire point positioning. The invention uses Kalman filter to dynamically track and predict the position of the fire point segmentation output plane coordinates, and combines with the digital elevation model (DEM) to query the ground height in real time for projection compensation. This cascading strategy not only reduces sensor noise, but also effectively corrects the deviation caused by terrain and angle changes.

[0035] (4) Multi-source space-time synchronization and weighted fusion, improving positioning stability

[0036] In the multi-sensor or multi-model parallel deployment scene, clock drift and coordinate reference differences of each data source will affect the fusion effect. The invention uses a unified time reference calibration method to achieve millisecond-level alignment of multiple images and IMU / GPS data, and maintains spatial consistency through coordinate system mapping. On this basis, the covariance weighted fusion of multiple Kalman filter results weakens single-source anomalies, and still outputs accurate positions in the scene of multiple target fire points crossing or blocking.

[0037] (5) End-to-end real-time processing framework, with high precision and high efficiency

[0038] The overall algorithm of the present application does not need post-processing scripts or tedious manual intervention, and segmentation, tracking, compensation and fusion are automatically completed in an integrated process. By using a lightweight attention module and a sparse data structure, the segmentation network inference speed is ensured to be above 25fps; the Kalman filter and DEM query operation overhead is small, and can be used for online deployment and real-time monitoring of unmanned aerial vehicles.

[0039] In summary, the method of the present application has achieved significant improvement in small fire point segmentation accuracy, positioning accuracy, system robustness and processing efficiency through multi-view attention fusion, non-downsampling network architecture, dynamic tracking and terrain compensation, multi-source spatio-temporal fusion and real-time design, and has high engineering application value and popularization potential. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 A flowchart of a UAV cluster fire point real-time detection and positioning method provided by the embodiment of the present application is shown in the figure.

[0042] Figure 2 A structure diagram of a double-attention segmentation network in the embodiment of the present application is shown in the figure.

[0043] Figure 3 A structure diagram of a UAV cluster fire point real-time detection and positioning system provided by the embodiment of the present application is shown in the figure.

[0044] Figure 4 A structure diagram of an electronic device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0045] It should be noted that:

[0046] The terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to clearly listed steps or units, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0047] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0049] like Figure 1 As shown, a method for real-time detection and localization of fire points in a drone swarm includes:

[0050] Step 1: Acquire visible and infrared images captured simultaneously by a single drone, along with the recorded capture time and drone pose data;

[0051] Step 2: Based on threshold segmentation and morphological filtering, process the visible image and infrared image to obtain time-series image pairs;

[0052] Step 3: Input the time-lapse images into the trained dual-attention segmentation network and output a fire semantic map;

[0053] Step 4: Process the fire semantic graph based on connected component analysis, output candidate fire points, and obtain the preliminary geographic coordinates of the candidate fire points based on UAV pose data;

[0054] Step 5: Based on the shooting time, the preliminary geographic coordinates of the candidate fire points are tracked in real time using Kalman filtering, and the fire point tracking and positioning results of a single UAV at the time to be located are output through dynamic compensation of elevation data.

[0055] Step 6: Merge the fire point tracking and positioning results of all UAVs in the UAV cluster at the time to be located to obtain the fire point fusion positioning result at the time to be located.

[0056] Optionally, before step 1, the actions performed by the drone include:

[0057] Simultaneously activate the visible light camera and infrared thermal imager on the drone to capture visible and infrared images, and record the timestamp of each frame;

[0058] Record the UAV's GPS / IMU data and output the UAV's pose information (latitude, longitude, altitude, pitch, yaw, roll) for each frame of image.

[0059] Preferably, step 1 further includes preprocessing of the captured visible and infrared images, including: (1) performing lens distortion correction on the visible and infrared images based on camera calibration parameters to ensure accurate geometric relationships of the images; (2) scaling the corrected images to a uniform resolution (optionally, a uniform resolution of 512×512 pixels) for subsequent network input; (3) using a filtering method (optionally, bilateral filtering or median filtering) to weaken random noise while preserving edge details; (4) adjusting the brightness and contrast of the visible image and stretching the grayscale of the infrared image to ensure optimal target contrast under different viewing angles and obtain a preprocessed image with uniform shape and color characteristics.

[0060] Step 2, the steps for obtaining time-image pairs include:

[0061] Infrared images are processed using an automatic thresholding segmentation method to separate high-temperature regions from the background. The minimum bounding rectangle of all high-temperature regions is calculated. Based on the minimum bounding rectangle, the infrared image and the simultaneously captured visible light image are cropped and resized to obtain time-series image pairs.

[0062] Specifically, step 2 is essentially the rapid localization of candidate fire points. This invention uses infrared images for preliminary localization of candidate regions before segmentation, reducing the computational load on the deep network and improving the response speed for small fire point detection. The specific process is as follows:

[0063] Step 2-1: For the preprocessed infrared image, the high-temperature region is separated from the background by an automatic thresholding method based on the image histogram to obtain a binary mask;

[0064] Step 2-2: Extract all connected regions from the binary mask obtained in Step 2-1, calculate the minimum bounding rectangle of the connected regions, construct a bounding box based on the minimum bounding rectangle, and crop and resize the visible light image and infrared image to obtain time-phase image pairs as input to the subsequent segmentation network.

[0065] Preferably, after step 2-1, an opening and closing operation is performed on the binary mask obtained in step 2-1 to remove isolated noise points and fill small holes, ensuring that the candidate regions are connected and continuous.

[0066] Preferably, after obtaining the connected regions in step 2-2, false target regions with an area smaller than a preset threshold or with an extremely long and thin shape are removed.

[0067] This step provides several candidate heat source boxes (rectangular boxes in step 2-2), providing a window of interest for the flame segmentation task, significantly reducing the network processing area and speeding up the operation.

[0068] Optionally, the candidate heat source box adopts the largest inscribed circle or the smallest circumscribed rectangle, and the edge of each candidate region is appropriately expanded by a certain number of pixels to take into account the context information.

[0069] In step 3, the backbone of the dual-attention segmentation network adopts a residual network. The encoding part processes different types of images in the input time image pairs through two structurally consistent encoding branches, extracting semantic feature maps of different scales. The highest-scale semantic feature maps output by the two branches are processed by the fusion convolutional layer to output a fused feature map. The decoding part decodes the fused feature map and finally outputs the fire semantic map.

[0070] The encoding branch extracts features from the image of the input branch layer by layer through several encoding layers. After each encoding layer, the channel importance weight is generated by the channel attention module, and a channel-weighted semantic feature map is generated. Then, it is processed by the spatial attention module to obtain an attention-enhanced semantic feature map.

[0071] Specifically, such as Figure 2 As shown, the backbone network of the dual attention segmentation network used in this embodiment adopts a residual network (such as ResNet50), which includes an encoding part and a decoding part.

[0072] The encoding part consists of two identical encoding branches and a fused convolutional layer (encoder 5).

[0073] One branch (visible light branch) is used to process visible light images, and the other branch (infrared branch) is used to process infrared images. Optionally, each branch contains four encoders (encoders 1-4), with each encoder followed by a channel attention module and a spatial attention module.

[0074] The visible light image and infrared image in the time-series image pair are respectively input into the corresponding feature extraction branch. In each branch, a channel attention module and a spatial attention module are added in sequence. The channel attention module generates channel weights through global pooling to highlight channel information related to fire semantics. The spatial attention module generates a two-dimensional weight map to emphasize the areas where flames and smoke may appear, and has a high response capability to small-scale fire points.

[0075] The features from the two branches are weighted by attention and then added or concatenated in the fusion layer. Several convolutional layers then extract the fused high-level semantic features. This dual-attention aggregation mechanism can adaptively align the differences in feature distribution between aerial and ground-based perspectives, generating a unified fire semantic map and thus achieving accurate fire area segmentation.

[0076] Specifically, in step 3, the process of the dual-attention segmentation network outputting the fire semantic map includes:

[0077] Step 3-1-1: Input the visible light and infrared images from the time-phase image pair into the corresponding encoding branches respectively;

[0078] Step 3-1-2: Extract mid-to-high-level semantic features from visible light and infrared inputs through two coding branches, and output two feature maps;

[0079] Step 3-1-3: The channel attention module performs global averaging and max pooling on the feature map, and combines two fully connected network layers to generate channel importance weights, weighting the features of different channels to highlight the semantics most beneficial to flame recognition;

[0080] Step 3-1-4: Based on the weighted feature map of the channel features, perform large-size convolution operations after concatenating the features using max pooling and average pooling to generate a spatial attention-weighted two-dimensional weight map. By weighting key locations, the response to small-scale and irregular flame regions is enhanced.

[0081] Step 3-1-5: The weighted feature maps from the two branches are concatenated along the channel dimension. Through several layers of convolution and upsampling operations, a flame probability map with the same resolution as the input is output.

[0082] Step 3-1-6: Apply a fixed threshold to the flame probability map to generate the final binary segmentation mask, which serves as the final output of the fire semantic map.

[0083] This stage integrates visible light and infrared information. Channel attention aggregates the most discriminative spectral features, while spatial attention focuses on key location features, effectively solving the problems of missed detections and false alarms caused by differences in viewing angles and complex backgrounds. This dual attention aggregation mechanism can adaptively align the differences in feature distribution between aerial and ground views, generating a unified fire semantic map, thereby achieving accurate fire area segmentation.

[0084] Step 4, the steps for obtaining candidate fire points include:

[0085] Based on connected component analysis, connected regions in the fire semantic graph are extracted. Connected regions in the extracted fire semantic graph are removed according to a preset connected region area threshold, and the remaining connected regions are used as the main fire point areas.

[0086] The geometric centroid of the main fire point region is calculated using image moments and recorded as candidate fire points.

[0087] Specifically, the steps for post-processing the fire semantic map output by the dual-attention segmentation network to obtain high-quality fire point candidates include:

[0088] 4-1. Perform connected component analysis on the fire semantic graph (which is a binary mask graph) to extract the binary connected components of all fire point regions;

[0089] 4-2. Calculate the minimum circumcenter geometric centroid of each connected component and record it as a pixel-level candidate fire point;

[0090] 4-3. Based on the UAV pose data, convert the image coordinates of each candidate fire point into preliminary geographic coordinates.

[0091] Preferably, before step 4-1, the fire semantic map output in step 3 is first opened to remove small noise points, and then closed to smooth the edge contours, remove isolated noise points in the fire semantic map and fill small holes, so as to ensure that the connected domains of the fire point area are connected and continuous.

[0092] Preferably, in step 4-2, the patch area of ​​each connected region is calculated, and based on a preset removal area threshold (less than 25 pixels), the main fire point areas are retained, and the subsequent centroid calculation steps are performed.

[0093] This stage transforms the segmentation results into a stable set of candidate points, providing input for subsequent geolocation.

[0094] Specifically, in step 4-3, based on the UAV's pose data and camera model, the centroid of each pixel is mapped to planar geographic coordinates. The process of transforming the pixel coordinates of candidate fire points to preliminary geographic coordinates is as follows:

[0095] Step 4-3-1: Obtain the UAV's global positioning and altitude information based on the UAV's pose data (London). u ,lat u ,alt u ), where lon u lat u and alt u These represent the longitude, latitude, and altitude of the drone's location, respectively.

[0096] Step 4-3-2: Based on camera intrinsic parameters and pixel offset Calculate the horizontal geographic offset under the assumption of approximate ground height:

[0097]

[0098]

[0099] In the formula, It is the horizontal geographic offset corresponding to the pixel; These are the pixel coordinates of the fire point at the geometric centroid of the fire mask in the fire semantic map; These are the coordinates of the camera's principal point; It is the camera's focal length (in pixels).

[0100] Step 4-3-3: Combine the UAV's pose coordinates with the horizontal distance offset to obtain the preliminary geographic coordinates of the candidate fire points:

[0101]

[0102]

[0103] in,( , () indicates the initial geographic coordinates of the candidate fire point.

[0104] Step 4-3-4: For each candidate fire point, record... Its timestamp provides observational data for tracking and compensation steps.

[0105] In step 5, the process of obtaining the fire point tracking and positioning results is as follows:

[0106] The preliminary geographic coordinates of candidate fire points at several times are sorted in chronological order of shooting time, and a smooth temporal sequence of fire point locations is generated based on Kalman filtering.

[0107] Based on elevation data compensation, the candidate fire points at the time of location are initially corrected in terms of geographic coordinates. Then, based on the result of the correction process, the fire point time sequence position at the time of location output by the Kalman filter is updated by secondary filtering, and the fire point tracking and location result at the time of location is output.

[0108] Specifically, the process of Kalman filtering to track the initial geographic coordinates of candidate fire points in real time first involves defining the state vector in dynamic tracking using Kalman filtering, mathematically represented as:

[0109]

[0110] Where k represents the kth time step; This represents the observation vector obtained at time step k, and represents the observation value of the candidate fire point in the planar coordinate system; The preliminary planar geographic coordinates of the candidate fire points at the k-th time step are obtained from step 4-3; Let be the fire spread rate at the k-th time step;

[0111] And the state transition matrix F, used to define the target motion model, mathematically represented as:

[0112]

[0113] in, For time step.

[0114] Based on the above settings, a Kalman filter is used to track the preliminary geographic observations in real time. The specific process is as follows:

[0115] Step 5-1-1: Taking candidate fire points as the observation objects, for the initial observation points, use the initial state vector. and the initial covariance matrix Initialization: The process noise covariance Q and measurement noise covariance R are preset based on the UAV speed and positioning accuracy;

[0116] Step 5-1-2: For each subsequent time step, first perform state prediction, then input the observation state of that time step (the preliminary geographic coordinates of the candidate fire points obtained in step 4-3-3) into the update stage to obtain the updated state;

[0117] The process of performing prediction at each time step can be mathematically expressed as follows:

[0118]

[0119]

[0120] in, This represents the predicted state at the k-th time step; This represents the update state at the (k-1)th time step; It is the predicted state covariance matrix at the k-th time step, reflecting the estimation uncertainty; It is the updated state covariance matrix at the (k-1)th time step, reflecting the prediction error; It is the process noise covariance, which describes the error of the state transition model;

[0121] Update Status: After obtaining the predicted status, the preliminary geographic coordinates of the candidate fire points obtained by processing the visible and infrared images captured at the k-th time step are used as the new fire coordinate observations at the k-th time step. A correction and update process is then performed, mathematically represented as:

[0122]

[0123]

[0124]

[0125] in, This represents the new fire coordinate observation value at the k-th time step; and These represent the longitude and latitude of the preliminary geographic coordinates of the candidate fire point corresponding to time step k, respectively. Indicates the updated status; It is the observation matrix. It is the observation noise covariance, which describes the measurement error; It is the Kalman gain, which balances the information from prediction and observation;

[0126] Step 5-1-3: At each time step, record the position after the filter update. And velocity estimation to form the trajectory of the fire point.

[0127] If no valid observation is detected at a certain time step, only the prediction phase is executed, and no update is performed, in order to maintain the continuity of the trajectory.

[0128] The Kalman filter smooths out observation noise, improves the stability of position estimation, and can maintain tracking even when fire points appear intermittently or detection is unstable.

[0129] Specifically, since terrain undulations and side-view angles introduce projection errors, this invention incorporates a digital elevation model (DEM) for dynamic terrain compensation. The steps include:

[0130] Step 5-2-1: Predict or correct the fire point location from the Kalman filter output. Query the actual ground height of the fire point from the pre-loaded elevation model. ;

[0131] Step 5-2-2: Recalculate the camera projection depth. Then, the pixel coordinates are remapped according to the formula in step 5 to obtain the corrected geographic coordinates. ;

[0132] Specifically, the horizontal geographic offset after incorporating the actual ground height is first calculated. , The calculation formula is:

[0133]

[0134]

[0135] In the formula, It is the flight altitude of the drone in the world coordinate system.

[0136] Corrected geographic coordinates .

[0137] Step 5-2-3: Use the corrected geographic coordinates as observations again and input them into the Karman wave filter for a second update to eliminate residual errors;

[0138] Step 5-2-4: Final output of the three-dimensional fire point position with elevation compensation .

[0139] This step significantly reduces lateral positioning deviations caused by changes in terrain elevation, achieving higher accuracy in ground fire point location.

[0140] Step 6, the steps for obtaining the fire point fusion localization results include:

[0141] Time alignment is performed based on the shooting time to match visible and infrared images from multiple perspectives under the same timestamp;

[0142] The fire point tracking and positioning results of several UAVs at the same timestamp are obtained, and a weighted average is calculated according to the magnitude of their respective expected covariance. The inverse of the variance is used as the weight to calculate the fire point fusion positioning result at the timestamp to be located.

[0143] Specifically, the steps of multi-UAV spatiotemporal fusion include:

[0144] 6-1. Process the visible light and infrared data of all UAVs using steps 1–5 above to generate their respective fire tracking results;

[0145] 6-2. Perform spatiotemporal fusion on the tracking results of each sensor in the world coordinate system: first correct the timestamp, then align the spatial coordinates, and use a weighted average or minimum variance fusion strategy for the same target to output the final fire point position.

[0146] Preferably, at least two drones are used to simultaneously photograph the airspace of the fire area from multiple perspectives, including at least one overhead view and at least one side view.

[0147] Preferably, in step 6-2, a spatiotemporal synchronization and preprocessing process is set up before spatiotemporal fusion: using the GNSS received by the UAV to receive precise time information, the clocks of all sensors are aligned with UTC to achieve millisecond-level time synchronization; then, the timestamps of visible light and infrared images on all UAVs are linearly interpolated to ensure that they correspond at the same time.

[0148] As a preferred embodiment, the present invention also provides a method for fusing results from multiple sensors or multiple models. In practical applications, multiple segmentation models or multiple sensors (such as visible light and infrared) may be used simultaneously for detection. To address this, the present invention proposes a weighted fusion strategy, the steps of which include:

[0149] For multiple location estimates of the same target at the same time, a weighted average is calculated based on the magnitude of the expected covariance of each filter, and the inverse of the variance is used as the weight to calculate the fused location.

[0150] Temporal smoothing of the fusion results is performed to remove abrupt outliers;

[0151] The final fused location and estimated confidence level are written into a standard format file and provided to subsequent applications or visualization systems.

[0152] The implementation of the various embodiments of the present invention is based on programmed processing through a system with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a real-time detection and location system for unmanned aerial vehicle (UAV) swarm fire points, which is used to execute a real-time detection and location method for UAV swarm fire points from the above method embodiments.

[0153] See Figure 3 The system includes: a drone and a data processing terminal. The drone is used to simultaneously capture visible and infrared images and record the capture time and drone pose data. The data processing terminal is used to receive information sent by the drone and execute the steps of a method for real-time detection and localization of drone swarm fire points.

[0154] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the modules in the above system embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above system embodiments, and on the premise of ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0155] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, embodiments of the present invention provide an electronic device, such as... Figure 4As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein the at least one processor, the communication interface, and the at least one memory communicate with each other via the communication bus. The at least one processor invokes logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0156] Furthermore, when the logical instructions in at least one of the aforementioned memories are implemented as software functional units and sold or used as independent products, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, is embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks—various media for storing program code.

[0157] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place, or distributed across multiple network units. The purpose of this embodiment is achieved by selecting some or all of the modules according to actual needs. Those skilled in the art will understand and implement this without any inventive effort.

[0158] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] Based on the same technical concept as the foregoing embodiments, the present invention provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute a method for real-time detection and localization of fire points in a drone swarm.

[0163] In summary, this invention discloses a real-time detection and localization method for unmanned aerial vehicle (UAV) swarm fire points, aiming to improve the segmentation of small fire points from multiple perspectives and high-precision geolocation capabilities. The method includes: simultaneous acquisition of visible light and infrared images by multiple UAVs and spatiotemporal alignment preprocessing; rapid localization of infrared candidate regions based on thresholding and morphological filtering; inputting paired visible light and infrared images, after channel-dimensional stitching, into a depth segmentation network containing channel attention and spatial attention modules to obtain a high-resolution flame probability mask; calculating connected components and centroids of the segmentation results and mapping them to preliminary geographic coordinates; dynamically tracking and smoothing the preliminary position state using Kalman filtering; combining digital elevation model (DEM) to query ground height, dynamically compensating for the predicted position projection depth, and updating with secondary filtering; finally, weighted fusion of multi-source or multi-model tracking results according to covariance to output the final three-dimensional fire point coordinates. The proposed method effectively preserves the features of small fire points through a non-subsampling network architecture; a dual attention mechanism enables fine alignment from multiple perspectives; Kalman filtering and DEM compensation work together to eliminate spatiotemporal errors; multi-source spatiotemporal synchronization and weighted fusion ensure the stability of the results; it combines high precision, high efficiency and strong robustness, and can be widely used in fire early warning and emergency command.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time detection and positioning of fire points by UAV cluster, characterized in that, The method comprises the following steps: acquiring visible images and infrared images synchronously photographed by a single unmanned aerial vehicle (UAV), and recording the photographing time and the UAV pose data; processing the visible images and the infrared images based on threshold segmentation and morphological filtering to obtain a time image pair; inputting the time image pair into a trained dual-attention segmentation network to output a fire semantic map; processing the fire semantic map based on connected domain analysis to output a candidate fire point, and acquiring the preliminary geographic coordinates of the candidate fire point based on the UAV pose data; based on the photographing time, tracking the preliminary geographic coordinates of the candidate fire point in real time through Kalman filtering, and dynamically compensating through elevation data to output the tracking positioning result of the fire point at the to-be-positioned time of the single UAV; fusing the tracking positioning results of the fire point at the to-be-positioned time of all the UAVs in the UAV cluster to obtain the fusion positioning result of the fire point at the to-be-positioned time.

2. The method of claim 1, wherein, The step of acquiring the time image pair comprises: processing the infrared image based on an automatic threshold segmentation method to separate the high-temperature region from the background, and calculating the minimum circumscribed rectangle of all the high-temperature regions, and performing cropping and size transformation on the infrared image and the visible image synchronously photographed based on the minimum circumscribed rectangle to obtain the time image pair.

3. The method of claim 1, wherein, The backbone network of the dual-attention segmentation network adopts a residual network, the encoding part processes different types of pictures in the input time image pair through two structurally consistent encoding branches to extract semantic feature maps of different scales, processes the highest scale semantic feature maps output by the two branches through a fusion convolution layer, and outputs a fusion feature map; the decoding part decodes and processes the fusion feature map to finally output a fire semantic map.

4. The method of claim 3, wherein, The encoding branch extracts features of the image input by the branch layer by layer through a plurality of encoding layers, generates channel importance weights through a channel attention module after each encoding layer, generates a channel weighted semantic feature map, and then processes the semantic feature map through a spatial attention module to obtain an attention-enhanced semantic feature map.

5. The method of claim 1, wherein, The step of acquiring the candidate fire point comprises: extracting the connected regions in the fire semantic map based on connected domain analysis, performing elimination processing on the connected regions in the extracted fire semantic map according to a preset connected region area threshold, and retaining the connected regions as main fire point regions; calculating the geometric center of the main fire point region through image moment, and recording the geometric center as a candidate fire point.

6. The method of claim 1, wherein, The process of acquiring the fire point tracking positioning result comprises: sorting the preliminary geographic coordinates of the candidate fire points at a plurality of times in chronological order based on the photographing time, generating a smooth fire point time sequence position sequence based on Kalman filtering processing; based on elevation data compensation, correcting the preliminary geographic coordinates of the candidate fire point at the to-be-positioned time, and performing secondary filtering update on the fire point time sequence position at the to-be-positioned time output by the Kalman filtering based on the correction processing result to output the fire point tracking positioning result at the to-be-positioned time.

7. The method of claim 1, wherein, The step of acquiring the fire point fusion positioning result comprises: aligning the time according to the photographing time, matching the visible images and the infrared images of multiple perspectives under the same timestamp; acquiring the fire point tracking positioning results of a plurality of UAVs under the same timestamp, performing weighted average on the fire point tracking positioning results according to the respective filtering expected covariance, taking the inverse of the variance as the weight, and calculating the fire point fusion positioning result under the to-be-positioned timestamp.

8. A UAV swarm fire point real-time detection and positioning system, characterized in that , including a drone and a data processing end, the drone is used to synchronize the visible image and the infrared image, and record the shooting time and the drone pose data; the data processing end is used to receive the information sent by the drone, and execute the steps of the method for real-time detection and positioning of fire points of the drone cluster according to any one of claims 1-7.

9. An electronic device, comprising: A non-transitory computer readable storage medium stores computer instructions, and the computer instructions make the computer execute the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, and the computer instructions make the computer execute the method according to any one of claims 1-7.

Images