Unmanned aerial vehicle fire source three-dimensional positioning method based on thermal field time-varying distortion compensation and inertial fusion

Abstract

This invention relates to the field of UAV fire source localization technology, and discloses a UAV fire source three-dimensional localization method based on thermal field time-varying distortion compensation and inertial fusion. The method acquires thermal infrared image sequences, inertial measurement data, and UAV pose data during UAV fire field navigation and synchronizes them in time. It extracts candidate fire source regions and stable thermal feature points from the thermal infrared image sequences; predicts the theoretical image displacement field caused by UAV motion based on inertial measurement data and UAV pose data, and obtains the actual image displacement field based on the inter-frame changes of stable thermal feature points; constructs a thermal field time-varying distortion field based on the difference between the actual and theoretical image displacement fields; uses the thermal field time-varying distortion field to perform reverse compensation on the fire source observation points, generating a compensated fire source observation ray, and outputs the three-dimensional coordinates of the fire source in the inertial fusion localization model. This invention can improve the accuracy and stability of UAV fire source three-dimensional localization.

Description

Technical Field

[0001] This invention relates to the field of UAV fire source localization technology, specifically a UAV fire source three-dimensional localization method based on thermal field time-varying distortion compensation and inertial fusion. Background Technology

[0002] Unmanned aerial vehicles (UAVs) equipped with thermal infrared cameras have become an important technical means for fire identification and emergency response in forest fires, building fires, warehouse fires, and chemical industrial park fires. Existing UAV fire source localization methods typically extract high-temperature areas from thermal infrared images and then combine them with UAV pose data, inertial measurement data, lidar data, or terrain height data to convert the fire source observation point in the thermal infrared image into a spatial observation ray. The three-dimensional coordinates of the fire source are then calculated through single-frame projection or multi-frame intersection.

[0003] However, in real fire environments, the area around the fire source experiences thermal plume disturbances, hot air refraction, smoke disturbances, and localized thermal saturation diffusion. This causes non-rigid drift over time in high-temperature regions of thermal infrared images. Existing methods typically use the high-temperature centroid, highest temperature point, or high-temperature connected region in the thermal infrared image directly as the fire source observation object, failing to distinguish between image displacement caused by the UAV's own motion and image drift caused by time-varying thermal field distortion. This leads to the fire source observation point deviating from the actual fire source location, further shifting the compensating fire source observation ray and reducing the accuracy and stability of three-dimensional fire source localization. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a UAV fire source three-dimensional positioning method based on thermal field time-varying distortion compensation and inertial fusion. This method solves the problem that existing UAV thermal infrared fire source three-dimensional positioning methods fail to effectively distinguish between UAV motion displacement and thermal field time-varying distortion displacement, leading to the offset of the fire source observation point and the fire source observation ray, thus affecting the accuracy of fire source three-dimensional positioning.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a UAV fire source three-dimensional positioning method based on thermal field time-varying distortion compensation and inertial fusion, comprising:

[0006] Acquire thermal infrared image sequences, inertial measurement data, and drone pose data during drone patrols over fire sites, and synchronize them in time.

[0007] Fire source candidate regions are extracted from the thermal infrared image sequence, and pixels that meet the positional stability condition within the fire source candidate regions are obtained as stable thermal feature points.

[0008] Based on the inertial measurement data and the UAV pose data, predict the theoretical image displacement field caused by the UAV motion between adjacent thermal infrared images;

[0009] Based on the inter-frame changes of the stable thermal feature points in adjacent thermal infrared images, the actual image displacement field is obtained;

[0010] Based on the difference between the actual image displacement field and the theoretical image displacement field, a time-varying thermal distortion field is constructed using a spatial interpolation method based on distortion confidence weights.

[0011] The reverse compensation of the fire source observation point in the fire source candidate region based on the thermal field time-varying distortion field includes: obtaining the distortion vector corresponding to the fire source observation point from the thermal field time-varying distortion field, correcting the position of the fire source observation point along the opposite direction of the distortion vector, and obtaining the compensated fire source observation point.

[0012] A compensation fire source observation ray is generated based on the aforementioned compensation fire source observation point;

[0013] Multiple frames of thermal infrared images are selected within a sliding time window. Ray intersection constraints are constructed based on the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images. Inertial constraints are constructed based on the inertial measurement data. Pose constraints are constructed based on the UAV pose data. The inertial constraints, pose constraints, and ray intersection constraints are jointly optimized in the inertial fusion positioning model to output the three-dimensional coordinates of the fire source.

[0014] Preferably, the steps of acquiring thermal infrared image sequences, inertial measurement data, and UAV pose data during UAV patrol of a fire site, and performing time synchronization, include:

[0015] The thermal infrared image sequence was acquired using a thermal infrared camera mounted on a drone;

[0016] The inertial measurement data is collected by an inertial measurement unit carried by the drone;

[0017] The drone's pose data is obtained through the positioning module mounted on the drone.

[0018] The inertial measurement data and the UAV pose data are aligned based on the acquisition time of the thermal infrared image sequence.

[0019] Preferably, the step of extracting fire source candidate regions from the thermal infrared image sequence and obtaining pixels that meet the positional stability condition within the fire source candidate regions as stable thermal feature points includes:

[0020] Temperature enhancement processing is performed on the thermal infrared image to obtain a temperature-enhanced image;

[0021] High-temperature connected regions are extracted based on the temperature-enhanced image;

[0022] The candidate ignition source region is determined based on the temperature distribution characteristics of the high-temperature connected region.

[0023] Pixels that meet the positional stability condition within consecutive frames are extracted from the fire source candidate region and used as the stable thermal feature points.

[0024] Preferably, the step of predicting the theoretical image displacement field caused by the UAV motion between adjacent thermal infrared images based on the inertial measurement data and the UAV pose data includes:

[0025] The relative motion of the UAV between adjacent thermal infrared images is calculated based on inertial measurement data between adjacent thermal infrared images;

[0026] The relative motion is corrected based on the UAV pose data to obtain the pose change of the thermal infrared camera;

[0027] Theoretical image displacement field is generated by predicting the theoretical pixel migration of pixels in the candidate fire source region based on the pose change of the thermal infrared camera.

[0028] Preferably, the step of obtaining the actual image displacement field based on the inter-frame changes of the stable thermal feature points in adjacent thermal infrared images includes:

[0029] The stable thermal feature points in adjacent thermal infrared images are tracked and matched to obtain the inter-frame pixel displacement of the stable thermal feature points.

[0030] The inter-frame pixel shifts are anomaly removed;

[0031] The actual image displacement field is generated based on the inter-frame pixel displacements after anomaly removal.

[0032] Preferably, the step of constructing the time-varying thermal distortion field using a spatial interpolation method based on distortion confidence weights, based on the difference between the actual image displacement field and the theoretical image displacement field, includes:

[0033] The actual image displacement field is compared with the theoretical image displacement field to obtain the thermal field time-varying distortion residual.

[0034] The distortion direction, distortion amplitude, and distortion confidence weight within the candidate fire source region are determined based on the time-varying distortion residual of the thermal field.

[0035] Spatial interpolation is performed based on the distortion direction, the distortion amplitude, and the distortion confidence weight to generate the time-varying distortion field of the thermal field.

[0036] Preferably, the step of performing reverse compensation on the fire source observation points in the fire source candidate region based on the time-varying distortion field of the thermal field to obtain the compensated fire source observation points includes:

[0037] The fire source observation point is determined based on the temperature distribution of the fire source candidate area;

[0038] Obtain the distortion vector corresponding to the fire source observation point from the time-varying distortion field of the thermal field;

[0039] The position of the fire source observation point is corrected along the opposite direction of the distortion vector to obtain the compensated fire source observation point.

[0040] Preferably, the steps of selecting multiple frames of thermal infrared images within a sliding time window, constructing ray intersection constraints based on the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images, constructing inertial constraints based on the inertial measurement data, constructing pose constraints based on the UAV pose data, and jointly optimizing the inertial constraints, pose constraints, and ray intersection constraints in the inertial fusion positioning model to output the three-dimensional coordinates of the fire source include:

[0041] Select multiple frames of thermal infrared images within a sliding time window, and obtain the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images;

[0042] Inertial constraints are constructed based on the inertial measurement data;

[0043] Construct pose constraints based on the UAV pose data;

[0044] A ray intersection constraint is constructed based on the compensated fire source observation ray;

[0045] The inertial constraints, pose constraints, and ray intersection constraints are jointly optimized in the inertial fusion positioning model to obtain the three-dimensional coordinates of the fire source.

[0046] This invention also provides a UAV fire source three-dimensional positioning system based on thermal field time-varying distortion compensation and inertial fusion, comprising:

[0047] The data synchronization module is used to acquire thermal infrared image sequences, inertial measurement data, and drone pose data during the drone's patrol of the fire site, and to synchronize them in time.

[0048] The stable feature extraction module is used to extract fire source candidate regions from the thermal infrared image sequence, and obtain pixels that meet the position stability condition within the fire source candidate regions as stable thermal feature points.

[0049] The theoretical displacement prediction module is used to predict the theoretical image displacement field caused by the UAV motion between adjacent thermal infrared images based on the inertial measurement data and the UAV pose data.

[0050] The actual displacement acquisition module is used to acquire the actual image displacement field based on the inter-frame changes of the stable thermal feature points in adjacent thermal infrared images;

[0051] The distortion field construction module is used to construct a thermal time-varying distortion field based on the difference between the actual image displacement field and the theoretical image displacement field by using a spatial interpolation method based on distortion confidence weight.

[0052] The observation point compensation module is used to perform reverse compensation on the fire source observation points in the fire source candidate region based on the thermal field time-varying distortion field to obtain compensated fire source observation points.

[0053] The observation ray generation module is used to generate a compensation fire source observation ray based on the compensation fire source observation point.

[0054] The fusion positioning module is used to select multiple frames of thermal infrared images within a sliding time window, construct ray intersection constraints based on the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images, construct inertial constraints based on the inertial measurement data, construct pose constraints based on the UAV pose data, and jointly optimize the inertial constraints, pose constraints and ray intersection constraints in the inertial fusion positioning model to output the three-dimensional coordinates of the fire source.

[0055] The present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the processor performs the steps of the above-described method for three-dimensional localization of UAV fire sources by thermal field time-varying distortion compensation and inertial fusion.

[0056] This invention provides a three-dimensional fire source localization method for unmanned aerial vehicles (UAVs) based on thermal field time-varying distortion compensation and inertial fusion. It has the following beneficial effects:

[0057] 1. This invention constructs a thermal time-varying distortion field by the difference between the actual image displacement field and the theoretical image displacement field. This can separate the normal image displacement caused by the movement of the UAV from the non-rigid image drift caused by the disturbance of the fire field's thermal plume, the refraction of hot air, and the disturbance of smoke. This avoids directly using the offset high-temperature area as the fire source observation object and improves the accuracy of the fire source observation point.

[0058] 2. This invention performs reverse compensation on the fire source observation point based on the thermal field time-varying distortion field. It can correct the position of the fire source observation point before generating the compensated fire source observation ray, making the compensated fire source observation ray closer to the actual fire source direction and reducing the impact of thermal field time-varying distortion on the three-dimensional intersection results.

[0059] 3. This invention selects multiple frames of thermal infrared images within a sliding time window, constructs ray intersection constraints based on the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images, constructs inertial constraints based on inertial measurement data, constructs pose constraints based on UAV pose data, and jointly optimizes the inertial constraints, pose constraints, and ray intersection constraints in the inertial fusion positioning model to output the three-dimensional coordinates of the fire source. This allows for the use of multiple frame observation results and inertial / pose constraints to improve the stability of the three-dimensional coordinate output of the fire source and reduce the impact of single-frame thermal infrared image errors on the positioning results.

[0060] 4. This invention obtains stable thermal feature points within the candidate fire source region and obtains the actual image displacement field based on the inter-frame changes of the stable thermal feature points. This can reduce the impact of thermal saturation, smoke occlusion, and random noise on the construction of the distortion field and improve the reliability of the time-varying thermal distortion field. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the method flow for the UAV fire source three-dimensional positioning method based on thermal field time-varying distortion compensation and inertial fusion according to the present invention. Detailed Implementation

[0062] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] This invention provides a UAV-based three-dimensional fire source localization method based on thermal field time-varying distortion compensation and inertial fusion. This method is applicable to scenarios such as forest fires, warehouse fires, chemical industrial park fires, and external building fire inspections. The UAV is equipped with a thermal infrared camera, an inertial measurement unit (IMU), a positioning module, and an altitude measurement module to conduct cruise observation of the fire area. The thermal infrared camera is used to acquire thermal infrared image sequences, the IMU is used to acquire angular velocity and acceleration data, the positioning module is used to obtain the UAV's position and attitude information, and the altitude measurement module is used to obtain the UAV's altitude information relative to the ground or target area. The positioning module can employ a GNSS module, an RTK module, a visual odometry module, or a combination thereof, and the altitude measurement module can employ a barometer, a laser rangefinder, a millimeter-wave altimeter, or a combination thereof.

[0064] The core of this invention lies in the following: between adjacent thermal infrared images, the theoretical image displacement field caused by the UAV's motion is first predicted using inertial measurement data and UAV pose data; then, the actual image displacement field is obtained based on the inter-frame changes of stable thermal feature points in the thermal infrared image; subsequently, the actual image displacement field is differentiated from the theoretical image displacement field to obtain the time-varying thermal distortion field caused by thermal plume disturbance, hot air refraction, and smoke disturbance; finally, the time-varying thermal distortion field is used to perform reverse compensation on the fire source observation point, and the compensated fire source observation ray is input into the inertial fusion positioning model to output the three-dimensional coordinates of the fire source.

[0065] Example 1

[0066] Reference Figure 1 This embodiment provides a three-dimensional localization method for UAV fire source based on thermal field time-varying distortion compensation and inertial fusion, including the following steps.

[0067] S1. Acquire the thermal infrared image sequence, inertial measurement data, and drone pose data during the drone's patrol of the fire site, and synchronize them in time.

[0068] Specifically, this includes: acquiring thermal infrared image sequences using a thermal infrared camera mounted on the drone; acquiring inertial measurement data using an inertial measurement unit mounted on the drone; obtaining drone pose data using a positioning module mounted on the drone; and aligning the inertial measurement data and drone pose data based on the acquisition time of the thermal infrared image sequences.

[0069] Specifically, the drone flies above the fire area along a preset or manually controlled flight path. During flight, the thermal infrared camera acquires thermal infrared images at a set frame rate, forming a thermal infrared image sequence. The inertial measurement unit acquires angular velocity and acceleration data at a sampling frequency higher than the thermal infrared camera's frame rate, forming inertial measurement data. The positioning module simultaneously outputs the drone's position, altitude, and attitude in the world coordinate system, forming the drone's pose data.

[0070] To ensure a unified time reference for thermal infrared images, inertial measurement data, and UAV pose data in subsequent calculations, this embodiment uses the acquisition time of each frame of the thermal infrared image sequence as the reference time to interpolate and align the inertial measurement data and UAV pose data. Specifically, for a given frame of thermal infrared image, inertial measurement data and UAV pose data before and after its acquisition time are selected, and linear interpolation or spline interpolation is performed based on the time difference to obtain the synchronous observation data corresponding to that frame of thermal infrared image.

[0071] In one optional implementation, the thermal infrared camera and the inertial measurement unit (IMU) undergo extrinsic parameter calibration beforehand to obtain the rotational and translational relationships of the thermal infrared camera coordinate system relative to the IMU coordinate system. The thermal infrared camera also undergoes intrinsic parameter calibration beforehand to obtain focal length, principal point, and distortion parameters. For the static distortion of the thermal infrared camera's own lens, a static correction can be performed before the thermal infrared image is input into this method. The time-varying thermal distortion field of this invention is primarily used to compensate for dynamic distortion caused by heat flow and smoke disturbance in a fire, rather than static distortion caused by ordinary lenses.

[0072] S2. Extract candidate regions of fire sources from thermal infrared image sequences and obtain stable thermal feature points within the candidate regions of fire sources.

[0073] Specifically, this includes: performing temperature enhancement processing on thermal infrared images to obtain temperature-enhanced images; extracting high-temperature connected regions based on the temperature-enhanced images; determining candidate fire source regions based on the temperature distribution characteristics of the high-temperature connected regions; and extracting pixels that meet the positional stability condition within consecutive frames from the candidate fire source regions as stable thermal feature points.

[0074] Specifically, temperature enhancement processing is performed on each frame of the thermal infrared image. This involves first converting the original grayscale values ​​in the thermal infrared image into temperature values ​​or relative temperature values, and then normalizing them based on the background temperature distribution of the current frame. For background areas in the image that have a high overall temperature but do not represent a real fire source, suppression can be achieved through local mean filtering, morphological opening operations, or background temperature estimation, thereby obtaining a temperature-enhanced image.

[0075] Subsequently, high-temperature connected regions are extracted from the temperature-enhanced image. These regions can be obtained using an adaptive temperature threshold. The adaptive temperature threshold can be determined based on the current frame's average temperature, temperature standard deviation, local maximum temperature, or historical background temperature. For the extracted high-temperature connected regions, they are filtered based on region area, average region temperature, maximum temperature, temperature gradient, and boundary continuity to remove isolated regions formed by reflections, local thermal noise, or short-term high-temperature interference, thus obtaining candidate fire source regions.

[0076] Stable thermal feature points are acquired within the candidate fire source region. Stable thermal feature points refer to thermal feature points that have high positional and temperature stability in consecutive frames. Stable thermal feature points can include lower edge points of hot spots, points with maximum temperature gradients, temperature-weighted centroids, and boundary points where the temperature change amplitude in consecutive frames is less than a set threshold.

[0077] In a fire, the heat plume above the fire source tends to drift with the wind and airflow, while the base of the fire source or the lower edge of the hot spot remains relatively stable. Therefore, in a preferred embodiment, this example prioritizes extracting stable thermal feature points near the lower edge of the fire source candidate region. Specifically, the main temperature gradient direction of the fire source candidate region is first determined, and then the boundary position where the temperature transitions from the high-temperature region to the background region is searched along the main temperature gradient direction. Points with good boundary continuity, large temperature gradients, and small positional changes within consecutive frames are selected as stable thermal feature points. If the fire source candidate region is obscured or its lower edge is incomplete, the points with maximum temperature gradients and temperature-weighted centroids can be used as supplementary stable thermal feature points.

[0078] The condition of satisfying positional stability specifically refers to: for any thermal feature point extracted from the candidate fire source region, calculating its positional stability in a continuous range. Intra-frame pixel coordinate position variance and temperature variance , Pick to ;when and If the thermal feature point is determined to be a stable thermal feature point, it is discarded otherwise.

[0079] Among them, the location threshold Based on the thermal infrared image resolution and the UAV's flight altitude, the ground sampling interval corresponding to a single pixel is set as follows: Rice time, Pick When the flight altitude changes, Adjusted inversely proportional to the square of the ground sampling interval. Temperature threshold. Take the standard deviation of the temperature of all pixels within the candidate fire source region of the current frame. times.

[0080] S3. Based on inertial measurement data and UAV pose data, predict the theoretical image displacement field caused by UAV motion between adjacent thermal infrared images.

[0081] Specifically, this includes: calculating the relative motion of the UAV between adjacent thermal infrared images based on inertial measurement data; correcting the relative motion based on the UAV pose data to obtain the pose change of the thermal infrared camera; and predicting the theoretical pixel migration amount of pixels in the candidate fire source region based on the pose change of the thermal infrared camera to generate a theoretical image displacement field.

[0082] Specifically, for two adjacent thermal infrared images, inertial measurement data is acquired between the acquisition times of the two images. Short-time integration is performed on the inertial measurement data to obtain the relative rotation and translation of the UAV between adjacent thermal infrared images. Subsequently, the short-time integration result is corrected using UAV pose data to obtain the pose change of the thermal infrared camera.

[0083] To predict the theoretical pixel migration amount of pixels in the fire source candidate region caused by the drone's movement, this embodiment back-projects the pixels in the current frame's fire source candidate region onto the camera coordinate system, and then projects them onto the next frame's image plane based on the thermal infrared camera's pose change. For a given pixel... The corresponding camera coordinates can be represented as:

[0084] ;

[0085] in, Represents pixels The 3D point corresponding to the current frame's camera coordinate system; Represents pixel coordinates, whose coordinate values ​​are ; Represents pixels The corresponding depth estimate; This represents the intrinsic parameter matrix of the thermal infrared camera.

[0086] Depth estimate It can be obtained in any of the following ways: estimated using UAV altitude and ground altitude models; estimated using digital elevation models of the fire area; obtained using lidar, binocular cameras or depth cameras carried by UAVs; or calculated based on ground altitude constraints when the candidate fire source areas are approximately located on the same ground plane.

[0087] The depth estimate The benchmark acquisition method is as follows:

[0088] (1) Obtain the ground elevation corresponding to the fire area The ground elevation is obtained from a digital elevation model or by approximating the local ground where the fire source is located as a horizontal reference surface.

[0089] (2) Obtain the UAV's flight altitude through a barometer or GNSS module Calculate the drone's altitude relative to the ground ;

[0090] (3) All pixels within the fire source candidate area Depth estimate The relative height is uniformly taken as the given height. For areas with undulating terrain, based on the real-time roll angle, pitch angle, and thermal infrared camera intrinsic parameters of the UAV, Converted to the actual depth value of each pixel along the camera's optical axis.

[0091] Let the change in pose of the thermal infrared camera between two adjacent frames be: Then the pixel The theoretical projection position in the next frame is:

[0092] ;

[0093] in, This represents the projection operation from camera coordinates to image coordinates. From this, the theoretical pixel migration for that pixel is obtained:

[0094] ;

[0095] in, This represents the theoretical image displacement field at the pixel point. The displacement vector at the location. By repeating the above calculation for multiple pixels or grid points within the fire source candidate region, the theoretical image displacement field corresponding to the fire source candidate region can be obtained.

[0096] The theoretical image displacement field reflects the image displacement caused solely by UAV motion, camera attitude changes, and spatial projection relationships, without considering thermal field disturbances.

[0097] S4. Obtain the actual image displacement field based on the inter-frame changes of stable thermal feature points in adjacent thermal infrared images.

[0098] Specifically, this includes: tracking and matching stable thermal feature points in adjacent thermal infrared images to obtain the inter-frame pixel displacement of the stable thermal feature points; removing anomalies from the inter-frame pixel displacements; and generating the actual image displacement field based on the anomaly-removed inter-frame pixel displacements.

[0099] Specifically, in adjacent thermal infrared images, stable thermal feature points are used as tracking objects for inter-frame tracking and matching to obtain the inter-frame pixel displacements of the stable thermal feature points. Tracking and matching can employ local optical flow tracking, feature template matching, local contour matching of hot spots, or multi-scale optical flow calculation methods.

[0100] In one specific implementation, a local window is established centered on the stable thermal feature point in the current frame. The location in the next frame's thermal infrared image that most closely resembles the temperature distribution of this local window is searched to obtain the matching point of the stable thermal feature point. The pixel position difference between the stable thermal feature point in the current frame and the matching point in the next frame is the inter-frame pixel displacement of the stable thermal feature point.

[0101] To mitigate the impact of smoke obscuring, thermal saturation, and random noise, this embodiment performs anomaly removal on inter-frame pixel displacements. Anomaly removal can be performed based on at least one of the following conditions: the matching confidence of a stable thermal feature point is lower than a preset threshold; the temperature change rate of the stable thermal feature point in adjacent frames exceeds a preset threshold; the displacement direction of the stable thermal feature point differs significantly from the dominant displacement direction of other stable thermal feature points in the neighborhood; or the stable thermal feature point falls into a thermal saturation region with insufficient boundary continuity. Stable thermal feature points that meet the anomaly conditions are not included in subsequent thermal field time-varying distortion field calculations.

[0102] Spatial interpolation is performed on the inter-frame pixel displacements of the retained stable thermal feature points to obtain the actual image displacement field within the fire source candidate region. The actual image displacement field reflects the true observed displacement of thermal feature points in adjacent thermal infrared images, and it simultaneously includes image displacement caused by UAV motion and non-rigid drift caused by thermal field disturbances in the fire field.

[0103] S5. Construct a thermal time-varying distortion field based on the difference between the actual image displacement field and the theoretical image displacement field.

[0104] Specifically, this includes: differentiating the actual image displacement field from the theoretical image displacement field to obtain the thermal field time-varying distortion residual; determining the distortion direction and distortion amplitude within the fire source candidate region based on the thermal field time-varying distortion residual; and generating the thermal field time-varying distortion field based on the distortion direction and distortion amplitude.

[0105] Specifically, the actual image displacement field is subtracted from the theoretical image displacement field to obtain the thermal field time-varying distortion residual. For the th For each stable thermal characteristic point, the time-varying distortion residual of its thermal field is:

[0106] ;

[0107] in, Indicates the first The time-varying distortion residual of the thermal field corresponding to each stable thermal feature point; Indicates the first The first frame A stable thermal characteristic point; This represents the actual image displacement of the stable thermal feature point; This represents the theoretical image displacement at the location corresponding to the stable thermal feature point.

[0108] The thermal field time-varying distortion residual represents the remaining pixel drift caused by factors such as fire plume disturbance, hot air refraction, and smoke disturbance after deducting the image displacement caused by UAV motion. If uncompensated fire source observation points are used directly for 3D positioning, this remaining pixel drift will cause the fire source observation ray to deviate from the actual fire source location.

[0109] To improve the reliability of the time-varying distortion field of the thermal field, this embodiment sets distortion confidence weights for the time-varying distortion residuals of the thermal field corresponding to each stable thermal feature point. The distortion confidence weights comprehensively consider temperature gradient, boundary stability, matching confidence, and thermal saturation degree, and can be specifically expressed as follows:

[0110] ;

[0111] in, Indicates the first The distortion confidence weight of a stable hot feature point; This represents the normalized value of the temperature gradient; Indicates boundary stability; Indicates the confidence level of inter-frame matching; This indicates the degree of thermal saturation. The more pronounced the temperature gradient, the more stable the boundary, the higher the matching confidence, and the lower the degree of thermal saturation, the greater the distortion confidence weight.

[0112] Subsequently, a time-varying thermal distortion field is generated based on the thermal field time-varying distortion residual and distortion confidence weight. For any pixel within the fire source candidate region... Its thermal time-varying distortion field can be expressed as:

[0113] ;

[0114] in, Represents pixels The thermal field time-varying distortion vector at the location; This indicates the number of stable thermal feature points involved in the calculation; Indicates the first The distortion confidence weight of a stable hot feature point; Indicates the first The location of a stable thermal feature point; Indicates the first The time-varying distortion residual of the thermal field corresponding to each stable thermal feature point; Indicates the smoothness of the spatial scale; This represents a small constant used to prevent the denominator from being zero.

[0115] Through the above processing, the obtained thermal time-varying distortion field has both spatial continuity and observation reliability, and can reflect the direction and amplitude of the influence of thermal plume disturbance, hot air refraction and smoke disturbance at different locations in the fire source candidate area.

[0116] S6. Based on the time-varying distortion field of the thermal field, the fire source observation points in the fire source candidate region are reverse compensated to obtain the compensated fire source observation points.

[0117] Specifically, this includes: determining the fire source observation point based on the temperature distribution of the fire source candidate area; obtaining the distortion vector corresponding to the fire source observation point from the time-varying distortion field of the thermal field; and correcting the position of the fire source observation point along the opposite direction of the distortion vector to obtain the compensated fire source observation point.

[0118] Specifically, the observation point for the fire source is determined based on the temperature distribution within the candidate fire source region. The observation point can be the temperature-weighted centroid of the candidate fire source region, a stable point at the lower edge of the candidate fire source region, or a point obtained by merging the temperature gradient maximum point with the temperature-weighted centroid.

[0119] In a preferred embodiment, to reduce the impact of thermal plume drift on localization, stable points near the lower edge of the fire source candidate region are preferentially identified as fire source observation points. When the lower edge is occluded or the boundary is incomplete, the temperature-weighted centroid is then used as the fire source observation point. The calculation of the temperature-weighted centroid can be completed based on the relative temperature weights of each pixel within the fire source candidate region, which is a conventional image geometry calculation, and the formula will not be elaborated here.

[0120] The distortion vector corresponding to the fire source observation point is extracted from the time-varying distortion field of the thermal field, and the position of the fire source observation point is corrected along the opposite direction of the distortion vector to obtain the compensated fire source observation point. Specifically, the compensated fire source observation point can be represented as:

[0121] ;

[0122] in, Indicates compensation for fire source observation points; This indicates the fire source observation point before compensation; This represents the time-varying distortion vector of the thermal field at the fire source observation point; This represents the compensation coefficient.

[0123] compensation coefficient The compensation coefficient can be determined based on the temporal continuity of the thermal time-varying distortion field. When the direction of the thermal time-varying distortion field is stable across multiple consecutive frames, a larger value is used; when the direction of the thermal time-varying distortion field changes abruptly or the confidence level is low, a smaller value is used. To avoid overcompensation, regional constraints can be applied to the compensated points to ensure that the compensated fire source observation points do not deviate from the fire source candidate area or its neighboring area by more than a preset distance.

[0124] S7. Generate a compensated fire source observation ray based on the compensated fire source observation point.

[0125] Specifically, a compensated fire source observation ray is generated based on the compensated fire source observation point, the intrinsic parameters of the thermal infrared camera, and the extrinsic parameters between the thermal infrared camera and the inertial measurement unit.

[0126] Specifically, the compensation fire source observation point is back-projected into a unit direction vector in the thermal infrared camera coordinate system:

[0127] ;

[0128] in, Indicates the compensated observation direction in the thermal infrared camera coordinate system; and This represents the x and y coordinates of the compensated fire source observation point in the image plane; This represents the intrinsic parameter matrix of the thermal infrared camera.

[0129] By combining the extrinsic parameters between the thermal infrared camera and the inertial measurement unit, as well as the UAV's pose in the world coordinate system, the compensated observation direction is transformed into the world coordinate system to obtain the compensated observation direction in the world coordinate system. Let the center position of the thermal infrared camera in the world coordinate system be... The compensated fire source observation ray corresponding to the current frame is represented as:

[0130] ;

[0131] in, Indicates the first The compensated fire source observation ray corresponding to the frame; This indicates the center position of the thermal infrared camera in the world coordinate system; Indicates the compensated observation direction in the world coordinate system; The parameters of the ray are represented. Using the above method, a compensated fire source observation ray can be obtained for each frame of thermal infrared image. Compared to the uncompensated fire source observation ray, the compensated fire source observation ray has reduced the impact of pixel shifts caused by thermal plume disturbances and hot air refraction on 3D positioning.

[0132] S8. Select multiple frames of thermal infrared images within the sliding time window, construct ray intersection constraints based on the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images, construct inertial constraints based on inertial measurement data, construct pose constraints based on UAV pose data, and jointly optimize the inertial constraints, pose constraints and ray intersection constraints in the inertial fusion positioning model to output the three-dimensional coordinates of the fire source.

[0133] Specifically, this includes: selecting multiple frames of thermal infrared images within a sliding time window and obtaining the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images; constructing inertial constraints based on inertial measurement data; constructing pose constraints based on UAV pose data; constructing ray intersection constraints based on compensated fire source observation rays; and jointly optimizing the inertial constraints, pose constraints, and ray intersection constraints in the inertial fusion positioning model to obtain the three-dimensional coordinates of the fire source.

[0134] Specifically, multiple frames of thermal infrared images are selected within a sliding time window, and the corresponding compensated fire source observation rays are obtained. The sliding time window can be set according to a fixed number of frames or a fixed time length, for example, selecting 5 to 30 consecutive frames of thermal infrared images, or selecting thermal infrared images within 1 to 5 seconds.

[0135] Inertial constraints are constructed based on inertial measurement data to ensure that the UAV pose changes between adjacent frames are consistent with the inertial measurement data. Pose constraints are constructed based on UAV pose data to ensure that the UAV's position and attitude in the world coordinate system do not deviate from the localization module's output. Ray intersection constraints are constructed based on compensated fire source observation rays to ensure that the fire source's 3D coordinates are as close as possible to the spatial intersection position of multiple compensated fire source observation rays. In one specific implementation, the inertial fusion localization model uses the fire source's 3D coordinates and the UAV pose within the sliding time window as the variables to be optimized. The ray intersection error is the vertical distance from the fire source's 3D coordinates to the compensated fire source observation rays. The optimization objective can be expressed as:

[0136] ;

[0137] in, This represents the three-dimensional coordinates of the fire source to be solved; This represents the set of UAV pose variables within the sliding time window. Indicates a sliding time window; Indicates the first Frame-compensated ray observation weights for fire source observation rays; Represents the identity matrix; Indicates the compensated observation direction in the world coordinate system; This indicates the center position of the thermal infrared camera in the world coordinate system; Indicates inertial constraint error; Indicates pose constraint error; and These represent the weighting coefficients for inertial constraint error and pose constraint error, respectively.

[0138] Ray observation weight The weight of the ray observation for the compensated fire source observation ray can be determined based on the time-varying distortion residual of the thermal field, the distortion confidence weight, and the stability of the candidate fire source region. If the time-varying distortion residual of the thermal field in a certain frame is too large, the degree of thermal saturation is too high, or the number of stable thermal feature points is too small, the ray observation weight of the compensated fire source observation ray in that frame is reduced; if the boundary of the candidate fire source region in a certain frame is stable, the distortion confidence weight is high, and the continuity before and after compensation is good, the ray observation weight of the compensated fire source observation ray in that frame is increased.

[0139] Taking least squares optimization as an example, the baseline implementation of the joint optimization is as follows: the inertial constraint error term is... Defined as the residual between the pre-integration result of the inertial measurement unit and the change in pose variables between adjacent frames, the pose constraint error term is included. Defined as the residual between the pose variable and the output value of the localization module, the ray intersection constraint is constructed as the vertical distance residual from the three-dimensional coordinates of the fire source to the observed ray of the compensated fire source in each frame. The objective function is iteratively optimized using the Levenberg-Marquardt algorithm or the Gauss-Newton method until convergence is obtained to obtain the three-dimensional coordinates of the fire source. The optimal estimate.

[0140] The optimization solution can employ least squares optimization, robust kernel function optimization, sliding window factor graph optimization, or extended Kalman filter optimization. After optimization, the three-dimensional coordinates of the fire source are output. The three-dimensional coordinates of the fire source can be expressed as latitude, longitude, and elevation, or as three-dimensional coordinates in a local northeast-northeast coordinate system.

[0141] While outputting the three-dimensional coordinates of the fire source, the system can also generate fire source location reliability. The fire source location reliability is determined comprehensively based on the spatial intersection error of the fire source observation rays compensated for across multiple frames, the number of effective observation frames, the average thermal field time-varying distortion residual, and the UAV's pose stability. When the fire source location reliability is lower than a set threshold, the system can prompt the UAV to change its observation angle, reduce its flight speed, or reacquire the thermal infrared image sequence; when the fire source location reliability is higher than the set threshold, the system outputs the three-dimensional coordinates of the fire source as the final location result.

[0142] Furthermore, in the above spatial interpolation method, the spatial smoothing scale... The adaptive setting method is as follows:

[0143] ;

[0144] in, The number of stable thermal feature points involved in the calculation. For the The geometric center coordinates of a stable thermal feature point.

[0145] In the reverse compensation, the compensation coefficient The method for determining it is as follows:

[0146] Calculate the time-varying distortion vector field of the thermal field in the current frame. With the distortion vector field of the previous frame The mean cosine similarity of the distortion vectors at each stable thermal feature point within the candidate fire source region. :

[0147] ;

[0148] according to Determine the compensation coefficient: ,in The function will The value is limited to Within the range.

[0149] In the inertial fusion positioning model, the ray observation weights corresponding to the compensated fire source observation rays in each frame are... The calculation method is as follows:

[0150] ;

[0151] in, This represents the number of effective stable hot feature points in the current frame. To preset the maximum number of feature points, The average modulus of the distortion residuals of all stable thermal feature points in the current frame. As the preset attenuation coefficient, take to , The mean cosine similarity is given.

[0152] Through the above steps, this embodiment can separate the image displacement caused by the movement of the UAV from the non-rigid drift caused by thermal field disturbance, and use the thermal field time-varying distortion field to compensate for the fire source observation point and the fire source observation ray, thereby improving the three-dimensional positioning accuracy of the UAV under the conditions of strong thermal plume, smoke disturbance and hot air refraction.

[0153] Example 2

[0154] This embodiment provides a UAV fire source three-dimensional positioning system with thermal field time-varying distortion compensation and inertial fusion, used to execute the method of Embodiment 1. The system includes a data synchronization module, a stable feature extraction module, a theoretical displacement prediction module, an actual displacement acquisition module, a distortion field construction module, an observation point compensation module, an observation ray generation module, and a fusion positioning module. The data synchronization module is used to acquire the thermal infrared image sequence, inertial measurement data, and UAV pose data during the UAV's fire field cruise and to synchronize them in time. The data synchronization module uses the acquisition time of the thermal infrared image sequence as a reference to interpolate and align the inertial measurement data and UAV pose data to generate synchronized observation data corresponding to each frame of thermal infrared image.

[0155] The stable feature extraction module is used to extract candidate fire source regions from the thermal infrared image sequence and obtain stable thermal feature points within these regions. The module performs temperature enhancement processing on the thermal infrared images, extracts high-temperature connected regions, and determines candidate fire source regions based on the temperature distribution characteristics of these regions. The module further extracts pixels that meet the positional stability condition within consecutive frames from the candidate fire source regions, which are then used as stable thermal feature points.

[0156] The theoretical displacement prediction module is used to predict the theoretical image displacement field caused by the UAV's motion between adjacent thermal infrared images based on inertial measurement data and UAV pose data. The module calculates the relative motion of the UAV based on the inertial measurement data between adjacent thermal infrared images, corrects the relative motion based on the UAV pose data to obtain the thermal infrared camera pose change, and predicts the theoretical pixel migration amount of pixels in the fire source candidate region based on the thermal infrared camera pose change, thus generating the theoretical image displacement field.

[0157] The actual displacement acquisition module is used to acquire the actual image displacement field based on the inter-frame changes of stable thermal feature points in adjacent thermal infrared images. The module tracks and matches stable thermal feature points in adjacent thermal infrared images to obtain the inter-frame pixel displacements of these stable thermal feature points, performs anomaly removal on the inter-frame pixel displacements, and generates the actual image displacement field based on the anomaly-removed inter-frame pixel displacements.

[0158] The distortion field construction module is used to construct a thermal time-varying distortion field based on the difference between the actual image displacement field and the theoretical image displacement field, using a spatial interpolation method based on distortion confidence weights. The module subtracts the actual image displacement field from the theoretical image displacement field to obtain the thermal time-varying distortion residual. Based on this residual, it determines the distortion direction, distortion amplitude, and distortion confidence weight within the fire source candidate region. Then, based on the distortion direction, distortion amplitude, and distortion confidence weight, it performs spatial interpolation to generate the thermal time-varying distortion field.

[0159] The observation point compensation module is used to perform reverse compensation on the fire source observation points in the fire source candidate region based on the time-varying distortion field of the thermal field, to obtain compensated fire source observation points. The observation point compensation module determines the fire source observation points according to the temperature distribution of the fire source candidate region, obtains the distortion vector corresponding to the fire source observation point from the time-varying distortion field of the thermal field, and corrects the position of the fire source observation point along the opposite direction of the distortion vector.

[0160] The observation ray generation module is used to generate a compensated fire source observation ray based on the compensated fire source observation point. The module back-projects the compensated fire source observation point into a compensated fire source observation ray in the world coordinate system, based on the compensated fire source observation point, the intrinsic parameters of the thermal infrared camera, and the extrinsic parameters between the thermal infrared camera and the inertial measurement unit.

[0161] The fusion positioning module selects multiple frames of thermal infrared images within a sliding time window, constructs ray intersection constraints based on the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images, constructs inertial constraints based on inertial measurement data, constructs pose constraints based on UAV pose data, and jointly optimizes the inertial constraints, pose constraints, and ray intersection constraints in the inertial fusion positioning model to output the three-dimensional coordinates of the fire source.

[0162] Example 3

[0163] This embodiment provides an electronic device, including a processor, a memory, and a computer program stored in the memory. When the computer program is executed by the processor, it causes the processor to perform the method steps of Embodiment 1.

[0164] The electronic equipment can be installed on the UAV's onboard computing platform or on a ground station server. When installed on the UAV's onboard computing platform, thermal infrared image sequences, inertial measurement data, and UAV pose data can be processed locally on the UAV in real time, and the three-dimensional coordinates of the fire source can be directly output. When installed on the ground station server, the UAV transmits the thermal infrared image sequences, inertial measurement data, and UAV pose data to the ground station server, which then performs thermal field time-varying distortion compensation and inertial fusion positioning calculations.

[0165] The memory can store intrinsic parameters of the thermal infrared camera, extrinsic parameters between the thermal infrared camera and the inertial measurement unit, synchronous observation data during UAV flight, fire source candidate region data, thermal field time-varying distortion field data, and three-dimensional coordinate results of the fire source. The processor can be a CPU, GPU, embedded AI processor, FPGA, or a combination thereof.

[0166] In the above embodiments, each module can be implemented using software programs, hardware circuits, or a combination of both. Data transmission between modules can be accomplished via airborne bus, Ethernet, serial port, wireless communication link, or shared memory. The specific parameters, thresholds, sensor models, and optimization algorithm types in the above embodiments can be adjusted according to the actual UAV platform, thermal infrared camera resolution, and fire scene environment. As long as they can achieve the thermal field time-varying distortion compensation and inertial fusion positioning process of the present invention, they all fall within the scope of the present invention.

[0167] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A UAV fire source three-dimensional localization method based on thermal field time-varying distortion compensation and inertial fusion, characterized in that, include: Acquire thermal infrared image sequences, inertial measurement data, and drone pose data during drone patrols over fire sites, and synchronize them in time. Fire source candidate regions are extracted from the thermal infrared image sequence, and pixels that meet the positional stability condition within the fire source candidate regions are obtained as stable thermal feature points. Based on the inertial measurement data and the UAV pose data, predict the theoretical image displacement field caused by the UAV motion between adjacent thermal infrared images; Based on the inter-frame changes of the stable thermal feature points in adjacent thermal infrared images, the actual image displacement field is obtained; Based on the difference between the actual image displacement field and the theoretical image displacement field, a time-varying thermal distortion field is constructed using a spatial interpolation method based on distortion confidence weights. The reverse compensation of the fire source observation point in the fire source candidate region based on the thermal field time-varying distortion field includes: obtaining the distortion vector corresponding to the fire source observation point from the thermal field time-varying distortion field, correcting the position of the fire source observation point along the opposite direction of the distortion vector, and obtaining the compensated fire source observation point. A compensation fire source observation ray is generated based on the aforementioned compensation fire source observation point; Multiple frames of thermal infrared images are selected within a sliding time window. Ray intersection constraints are constructed based on the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images. Inertial constraints are constructed based on the inertial measurement data. Pose constraints are constructed based on the UAV pose data. The inertial constraints, pose constraints, and ray intersection constraints are jointly optimized in the inertial fusion positioning model to output the three-dimensional coordinates of the fire source.

2. The UAV fire source three-dimensional positioning method based on thermal field time-varying distortion compensation and inertial fusion according to claim 1, characterized in that, The steps for acquiring and synchronizing the thermal infrared image sequence, inertial measurement data, and drone pose data during drone patrols over fire sites include: The thermal infrared image sequence was acquired using a thermal infrared camera mounted on a drone; The inertial measurement data is collected by an inertial measurement unit carried by the drone; The drone's pose data is obtained through the positioning module mounted on the drone. The inertial measurement data and the UAV pose data are aligned based on the acquisition time of the thermal infrared image sequence.

3. The UAV fire source three-dimensional positioning method based on thermal field time-varying distortion compensation and inertial fusion according to claim 1, characterized in that, The steps of extracting candidate fire source regions from the thermal infrared image sequence and obtaining pixels that meet the positional stability condition within the candidate fire source regions as stable thermal feature points include: Temperature enhancement processing is performed on the thermal infrared image to obtain a temperature-enhanced image; High-temperature connected regions are extracted based on the temperature-enhanced image; The candidate ignition source region is determined based on the temperature distribution characteristics of the high-temperature connected region. Pixels that meet the positional stability condition within consecutive frames are extracted from the fire source candidate region and used as the stable thermal feature points.

4. The UAV fire source three-dimensional positioning method based on thermal field time-varying distortion compensation and inertial fusion according to claim 1, characterized in that, Based on the inertial measurement data and the UAV pose data, the steps for predicting the theoretical image displacement field caused by the UAV motion between adjacent thermal infrared images include: The relative motion of the UAV between adjacent thermal infrared images is calculated based on inertial measurement data between adjacent thermal infrared images; The relative motion is corrected based on the UAV pose data to obtain the pose change of the thermal infrared camera; Theoretical image displacement field is generated by predicting the theoretical pixel migration of pixels in the candidate fire source region based on the pose change of the thermal infrared camera.

5. The UAV fire source three-dimensional positioning method based on thermal field time-varying distortion compensation and inertial fusion according to claim 1, characterized in that, The steps for obtaining the actual image displacement field based on the inter-frame changes of the stable thermal feature points in adjacent thermal infrared images include: The stable thermal feature points in adjacent thermal infrared images are tracked and matched to obtain the inter-frame pixel displacement of the stable thermal feature points. The inter-frame pixel shifts are anomaly removed; The actual image displacement field is generated based on the inter-frame pixel displacements after anomaly removal.

6. The UAV fire source three-dimensional positioning method based on thermal field time-varying distortion compensation and inertial fusion according to claim 1, characterized in that, The steps for constructing a time-varying thermal distortion field using a spatial interpolation method based on distortion confidence weights, based on the difference between the actual image displacement field and the theoretical image displacement field, include: The actual image displacement field is compared with the theoretical image displacement field to obtain the thermal field time-varying distortion residual. The distortion direction, distortion amplitude, and distortion confidence weight within the candidate fire source region are determined based on the time-varying distortion residual of the thermal field. Spatial interpolation is performed based on the distortion direction, the distortion amplitude, and the distortion confidence weight to generate the time-varying distortion field of the thermal field.

7. The UAV fire source three-dimensional positioning method based on thermal field time-varying distortion compensation and inertial fusion according to claim 1, characterized in that, The steps for performing reverse compensation on the fire source observation points in the fire source candidate region based on the time-varying distortion field of the thermal field to obtain the compensated fire source observation points include: The fire source observation point is determined based on the temperature distribution of the fire source candidate area; Obtain the distortion vector corresponding to the fire source observation point from the time-varying distortion field of the thermal field; The position of the fire source observation point is corrected along the opposite direction of the distortion vector to obtain the compensated fire source observation point.

8. The UAV fire source three-dimensional positioning method based on thermal field time-varying distortion compensation and inertial fusion according to claim 1, characterized in that, The steps of selecting multiple frames of thermal infrared images within a sliding time window, constructing ray intersection constraints based on the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images, constructing inertial constraints based on the inertial measurement data, constructing pose constraints based on the UAV pose data, and jointly optimizing the inertial constraints, pose constraints, and ray intersection constraints in the inertial fusion positioning model to output the three-dimensional coordinates of the fire source include: Select multiple frames of thermal infrared images within a sliding time window, and obtain the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images; Inertial constraints are constructed based on the inertial measurement data; Construct pose constraints based on the UAV pose data; A ray intersection constraint is constructed based on the compensated fire source observation ray; The inertial constraints, pose constraints, and ray intersection constraints are jointly optimized in the inertial fusion positioning model to obtain the three-dimensional coordinates of the fire source.

9. A UAV fire source three-dimensional positioning system based on thermal field time-varying distortion compensation and inertial fusion, characterized in that, The UAV fire source three-dimensional localization method based on thermal field time-varying distortion compensation and inertial fusion according to any one of claims 1-8 includes: The data synchronization module is used to acquire thermal infrared image sequences, inertial measurement data, and drone pose data during the drone's patrol of the fire site, and to synchronize them in time. The stable feature extraction module is used to extract fire source candidate regions from the thermal infrared image sequence, and obtain pixels that meet the position stability condition within the fire source candidate regions as stable thermal feature points. The theoretical displacement prediction module is used to predict the theoretical image displacement field caused by the UAV motion between adjacent thermal infrared images based on the inertial measurement data and the UAV pose data. The actual displacement acquisition module is used to acquire the actual image displacement field based on the inter-frame changes of the stable thermal feature points in adjacent thermal infrared images; The distortion field construction module is used to construct a thermal time-varying distortion field based on the difference between the actual image displacement field and the theoretical image displacement field by using a spatial interpolation method based on distortion confidence weight. The observation point compensation module is used to perform reverse compensation on the fire source observation points in the fire source candidate region based on the thermal field time-varying distortion field to obtain compensated fire source observation points. The observation ray generation module is used to generate a compensation fire source observation ray based on the compensation fire source observation point. The fusion positioning module is used to select multiple frames of thermal infrared images within a sliding time window, construct ray intersection constraints based on the compensated fire source observation rays corresponding to the multiple frames of thermal infrared images, construct inertial constraints based on the inertial measurement data, construct pose constraints based on the UAV pose data, and jointly optimize the inertial constraints, pose constraints and ray intersection constraints in the inertial fusion positioning model to output the three-dimensional coordinates of the fire source.

10. An electronic device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory. When the computer program is executed by the processor, it causes the processor to perform the steps of the UAV fire source three-dimensional localization method of thermal field time-varying distortion compensation and inertial fusion as described in any one of claims 1 to 8.

Images