A forest fire temperature error online compensation method and system based on multi-modal data fusion and region division, and a storage medium

Abstract

The application discloses a forest fire temperature error online compensation method and system based on multi-modal data fusion and region division and a storage medium, relates to the technical field of forest fire monitoring, and aims at solving the problems that the existing method cannot accurately distinguish the temperature distribution of different regions, is difficult to overcome the interference of complex environments such as dynamic smoke and complex temperature distribution in forest fires, and has low detection precision of a temperature field. The application fuses a fire field three-dimensional point cloud, a visible light image, an infrared thermal image and pose data, divides the fire field into a flaming area, a burnout area and an unburned area, establishes a flaming area temperature compensation model in view of the influence of radiation heat generated by fierce burning on the flaming area and the influence of smoke absorption and scattering on infrared radiation, establishes temperature compensation models of the burnout area and the unburned area based on the temperature gradient of the burnout area and the unburned area and the difference in fuel combustion characteristics, and realizes online compensation of the fire field temperature by driving the temperature compensation models in real time through the collected multi-modal data.

Description

Technical Field

[0001] This invention relates to the field of forest fire monitoring technology, specifically to an online compensation method, system, and storage medium for forest fire temperature errors based on multimodal data fusion and regional division. Background Technology

[0002] Fire temperature detection is a crucial technical requirement in key areas such as forest fire monitoring, fire assessment, and emergency command. At fire sites, due to the complex distribution of burning areas, a fire typically consists of multiple zones, including open flame zones, burnt-out zones, and unburned zones, each with significantly different temperature and combustion characteristics. This diversity in temperature distribution poses a significant challenge to the accuracy of traditional temperature detection equipment such as infrared thermal imagers. Furthermore, the absorption and scattering of infrared signals by dynamic smoke and high-temperature airflow prevalent in fire environments further exacerbates detection errors, making it difficult for existing single temperature measurement models to meet the accuracy requirements of online temperature field detection in complex fire scenarios.

[0003] Traditional infrared thermography typically relies on uniform parameter settings or model assumptions across the entire fire field to reconstruct the temperature field. However, the physical characteristics of different areas within a fire (such as emissivity, transmittance, temperature distribution patterns, and fuel combustion characteristics) vary significantly. This "one-size-fits-all" approach often fails to accurately describe the temperature characteristics of different areas. For example, the open flame area has high radiation intensity but is easily affected by smoke; the burnout zone has a lower temperature and a relatively uniform temperature distribution.

[0004] Chinese patent application CN 117073847 A discloses an online temperature field detection method and system based on multi-source vision under water mist interference. This method addresses the challenge of water mist interference during temperature measurement by proposing a two-stage water mist transmittance field estimation method and recovering the true temperature field based on multi-source vision technology. Its advantage lies in innovatively solving the interference problem of water mist on temperature detection, making it suitable for high humidity or water mist conditions in environments such as industrial sites, and capable of accurately recovering the temperature field. However, the method's drawback is that it is mainly applicable to industrial scenarios and is not well-suited for application scenarios with complex environments and significantly different fire zone characteristics, such as forest fires. In forest fires, the temperature characteristics of different areas vary greatly, and they may also be subject to complex interference from dynamic smoke. This method lacks regional temperature compensation strategies for these regional characteristics, so its accuracy and robustness are significantly reduced when directly applied to temperature field detection in forest fires. Chinese patent application CN 105723418 A discloses a method for determining fire temperature by capturing images of the fire scene and analyzing fire light information, solving the problem that traditional technologies are difficult to use for fire temperature detection. Its advantage lies in its simplicity; it can quickly determine the overall temperature range of a fire using flame characteristics, making it suitable for relatively simple fire scenarios. However, this method has significant limitations. First, it relies solely on flame information and lacks the ability to identify other areas of the fire (such as high-temperature smoke zones, burnt-out zones, and unburned zones), resulting in an inability to accurately distinguish the temperature distribution of different areas. Second, in actual complex fire scenes, flames may be interfered with by smoke, airflow, and obstructions, making it difficult to maintain accuracy in temperature detection based solely on flame information. Therefore, this method is more suitable for fire temperature detection in simple scenarios, but its application in complex fire environments is limited. Summary of the Invention

[0005] The technical problem to be solved by this invention is:

[0006] Existing methods cannot accurately distinguish the temperature distribution in different areas, and they are also unable to overcome the interference of complex environments such as dynamic smoke and complex temperature distribution in forest fires, resulting in low accuracy in temperature field detection.

[0007] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0008] This invention provides an online compensation method for forest fire temperature error based on multimodal data fusion and regional division, comprising the following steps:

[0009] Step 1: Collect data in the fire area using a drone equipped with a lidar, visible light camera, infrared thermal imager and inertial navigation system. Obtain 3D point cloud, visible light image, infrared thermal image and pose data in geographic coordinate system provided by inertial navigation system. Perform spatiotemporal registration on the 3D point cloud, visible light image, infrared thermal image and pose data in geographic coordinate system.

[0010] Step 2: Based on the spatiotemporally registered fused data, the fire scene is divided into three areas: the open flame area, the burnt-out area, and the unburned area;

[0011] Step 3: Based on the 3D point cloud data of the fire scene from the lidar and the pose data from the inertial navigation system, obtain the slant distance and angle between the infrared thermal imager and the target point, and estimate the smoke transmittance of each area based on the visible light image.

[0012] Step 4: Considering the radiant heat generated by intense combustion in the open flame area, and the absorption and scattering of infrared radiation by smoke, a temperature compensation model for the open flame area is established, combining the slant distance and angle results between the infrared thermal imager and the target point from Step 3, as well as the smoke transmittance results for each region. Based on the temperature gradient and fuel combustion characteristics differences between the burnt-out and unburned zones, temperature compensation models for the burnt-out and unburned zones are established, combining the slant distance and angle results between the infrared thermal imager and the target point from Step 3. The temperature compensation model is driven in real-time by the collected multimodal data to achieve online temperature compensation at the fire site.

[0013] Furthermore, the spatiotemporal registration of the three-dimensional point cloud of the fire scene, visible light image, infrared thermal image, and pose data in the geographic coordinate system described in step one includes the following steps:

[0014] Step 11: Based on 3D-2D constraint extrinsic parameter calibration, according to feature matching of fire scene 3D point cloud, infrared thermal image and visible light image, the spatial reference relationship between each sensor is initially constructed, and a unified coordinate system is established for each sensor;

[0015] Steps 1 and 2: Align the timestamps to address the differences in data collection timestamps from different sensors;

[0016] Step 13: Design adaptive weights based on the performance characteristics of different sensors in different environments;

[0017] Step 14: Apply the LM algorithm to optimize the sensor extrinsic parameter matrix using the adaptively adjusted sensor weights, and then perform spatiotemporal registration of the data from each sensor. The optimization objective function is expressed as shown in formula (1):

[0018] (1)

[0019] in, It is a set of extrinsic parameter matrices; For the first The actual observation values ​​of each sensor; These are the predicted values ​​based on the extrinsic parameter matrix; For the first The weight of each sensor; For constraint functions; is the regularization coefficient.

[0020] Furthermore, in steps one and three, adaptive weights are designed based on the performance characteristics of different sensors under different environments, specifically as follows:

[0021] For densely wooded areas, increase the weight of lidar; for areas with open flames, increase the weight of infrared thermal imagers; for areas with smoke but no visible flames, increase the weight of visible light cameras. The weight calculation formula is as follows:

[0022] (2)

[0023] in, For the first Basic feature scoring of each sensor; The number of sensors; This is the environmental sensitivity factor, used to represent the degree of adaptability of a sensor to a specific environment, with a value range of [value range missing]. .

[0024] Furthermore, step two includes the following process:

[0025] Step 21: Preprocess the visible light image by converting it from RGB to HSV color space and performing threshold segmentation to initially extract the open flame area. Use the same method to mark the burnt-out area that represents ash characteristics.

[0026] Step 22: Identify the local maxima of the highest temperature in the infrared thermal image to determine potential high-temperature hotspots; starting from each maxima, gradually expand outwards and determine the location of significant temperature drops based on the decreasing temperature trend, thereby determining the boundary of the open flame area; using the boundary of the fire area as the potential open flame area, and using the transformation matrix of the visible light camera and the infrared thermal imager, spatially align and overlay the open flame area and the potential open flame area to finally confirm the open flame area in the fire scene;

[0027] Steps 2 and 3: Using a multi-level contour aggregation method based on bright spot expansion, high-temperature bright spots in the infrared thermal image are regarded as independent connected pixels as the initial identifier of the burning area; morphological dilation is used to expand the bright spot area to cover the potential burnout area; combined with the region growing algorithm, the growth direction and range are dynamically determined according to the temperature characteristics of adjacent pixels, and isolated bright spots are gradually connected into a complete burnout area contour. The burnout area contour is extracted as a unified contour covering all burning areas.

[0028] Step Two Four: Calculation The burnout zone of time and Open flame area at all times The intersection of the two is represented as According to the intersection Based on the characteristics of the region, the following area updates will be performed:

[0029] like Then open flame area Set as a new open flame zone, indicated as ;

[0030] like Then the region division needs to be updated: (1) Remove intersections. After The remaining portion is assigned to a new open flame zone, denoted as... (2) Update the burnout zone to and The union of the burnout regions is represented as (3) Update the unburned area to the remaining area after removing the new burnout area and open flame area. , represented as ;

[0031] Step 25: For each subsequent time point, repeat the region update process in Steps 23 and 24. After each update, use... The regional outline of the moment and The contours at any given time are intersected and merged to generate updated open flame areas, burnt-out areas, and unburned areas. These areas are then overlaid onto the global contour map in the geographic coordinate system to gradually build complete fire area coverage.

[0032] Furthermore, the slant distance between the infrared thermal imager and the target point in step three is expressed as shown in equation (3):

[0033] (3)

[0034] in, It is the slant distance between the infrared thermal imager and the target point; It is the camera's intrinsic parameter matrix; It is the scaling factor; It is the rotation matrix from the camera coordinate system to the radar coordinate system; These are pixel coordinates;

[0035] The formula for the angular relationship between the infrared thermal imager and the target point is expressed as equation (4):

[0036] (4)

[0037] in, The angle of a target point at any location in the image observed by the camera; This is the rotation matrix from the combined inertial navigation coordinate system to the northeast-sky coordinate system (ENU); This is the rotation matrix from the camera coordinate system to the combined inertial navigation coordinate system; This is the translation vector from the camera coordinate system to the combined inertial navigation coordinate system; Represents the transformed vector Quantity;

[0038] The formula for the average smoke transmittance of each region in a visible light image is expressed as equation (5):

[0039] (5)

[0040] in, Indicates the region Average smoke transmittance; For the region The total number of pixels in; Indicates the region Any coordinate in the array; Representing coordinates neighborhood Image coordinates within; It represents any one of the three color channels: red, green, and blue. Indicates color channel Foggy images; Indicates color channel The global atmospheric light is the average value of the 0.1% brightest pixels in the foggy image on that channel.

[0041] Furthermore, step four includes the following steps:

[0042] Step 41: For the open flame area, a compensation model based on the principle of radiative heat transfer is adopted. Considering the impact of radiative heat generated by intense combustion on the open flame area, as well as the absorption and scattering of infrared radiation by smoke, the following temperature compensation model is established:

[0043] (6)

[0044] in, The temperature of the target point within the compensated open flame area; The temperature of the target point within the open flame area as measured by an infrared thermal imager; It is the specific heat capacity at constant volume; It is the Stefan-Boltzmann constant proportional term; c is the speed of light; The radiation intensity of the target point measured by the infrared thermal imager is obtained from equation (7):

[0045] (7)

[0046] in, The area of ​​the detector pixel; for Calibration gain of the real-time infrared system; Atmospheric transmittance by slant path. , This is an empirical coefficient; For the magnification of the infrared system, , The effective focal length of the system; The directional emissivity curve of the target. , Let be the emissivity in the direction normal to the target. These are directional parameters related to the target material; The target grayscale value; The background grayscale value; Background radiation; The atmospheric path radiation between the infrared system and the target;

[0047] Step 42: For the burnout zone and the unburnt zone, based on the temperature gradient and differences in fuel combustion characteristics within the zone, the following temperature compensation model is established:

[0048] (8)

[0049] in, The temperature of the target point within the compensated burnout zone or unburned zone; and For polynomial coefficients, ; The ambient temperature.

[0050] This invention provides an online compensation system for forest fire temperature error based on multimodal data fusion and regional division. The system has program modules corresponding to the steps of any of the above-described technical solutions, and executes the steps in the above-described online compensation method for forest fire temperature error based on multimodal data fusion and regional division when running.

[0051] The present invention provides a computer-readable storage medium storing a computer program configured to, when invoked by a processor, implement the steps of online compensation for forest fire temperature error based on multimodal data fusion and regional division as described in any of the above technical solutions.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] This invention provides an online compensation method for forest fire temperature errors based on multimodal data fusion and regional segmentation. Using a drone equipped with lidar, a visible light camera, an infrared thermal imager, and an inertial navigation system, data is collected and multimodal fusion analysis is performed in the fire area, enabling precise segmentation of open flame zones, burnt-out zones, and unburned zones. This invention fully considers the complex and diverse characteristics of the fire environment, optimizing the contributions of various sensors in different fire areas through multi-sensor spatiotemporal registration and adaptive weight allocation, thus solving the problem of insufficient consistency in data fusion in traditional methods. Simultaneously, a multi-level contour aggregation method based on bright spot expansion is proposed, which can dynamically update the fire area contour and accurately identify the characteristics of the fire area. Furthermore, this invention uses visible light camera and inertial navigation system data to estimate smoke transmittance, combined with three-dimensional point cloud information of the fire area, to realize a multimodal driven regional temperature compensation model, effectively overcoming the influence of smoke obstruction and sensor measurement errors.

[0054] This invention constructs an accurate fire zone division and temperature compensation model by efficiently integrating multimodal data. It can not only intuitively present the dynamic changes of the fire, but also improve the accuracy and real-time performance of fire temperature measurement, providing important support for rapid response and scientific decision-making in forest fires. Attached Figure Description

[0055] Figure 1 This is a flowchart of an online forest fire temperature error compensation method based on multimodal data fusion and regional division, as described in an embodiment of the present invention.

[0056] Figure 2 This is a field image of fire scene data collected by a drone in an example of the present invention;

[0057] Figure 3 This is a comparison chart of combustion profile recognition using the Otus+Canny algorithm, the region-based level set algorithm, and the algorithm of this invention in an example of the present invention;

[0058] Figure 4 This is a diagram illustrating the fire zone division in an example of the present invention.

[0059] Figure 5 This is a diagram showing the temperature error compensation results in an example of the present invention. Detailed Implementation

[0060] To enable those skilled in the art to better understand the present invention, exemplary embodiments or examples of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments or examples are merely some, not all, of the embodiments or examples of the present invention. All other embodiments or examples obtained by those skilled in the art based on the embodiments or examples of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0061] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0062] In a typical embodiment of the present invention, a method for online compensation of forest fire temperature error based on multimodal data fusion and regional division is provided, comprising the following steps:

[0063] Step 1: Collect data in the fire area using a drone equipped with a lidar, visible light camera, infrared thermal imager and inertial navigation system. Obtain 3D point cloud of the fire, visible light image, infrared thermal image and pose data in geographic coordinate system provided by inertial navigation system. Perform spatiotemporal registration of multi-source data on the 3D point cloud of the fire, visible light image, infrared thermal image and pose data in geographic coordinate system.

[0064] Step 2: Based on the spatiotemporally registered fused data, the fire scene is divided into three areas: the open flame area, the burnt-out area, and the unburned area;

[0065] Step 3: Based on the 3D point cloud data of the fire scene from the lidar and the pose data from the inertial navigation system, obtain the slant distance and angle between the infrared thermal imager and the target point, and estimate the smoke transmittance of each area based on the visible light image.

[0066] Step 4: Considering the radiant heat generated by intense combustion in the open flame area, and the absorption and scattering of infrared radiation by smoke, a temperature compensation model for the open flame area is established, combining the slant distance and angle results between the infrared thermal imager and the target point from Step 3, as well as the smoke transmittance results for each region. Based on the temperature gradient and fuel combustion characteristics differences between the burnt-out and unburned zones, temperature compensation models for the burnt-out and unburned zones are established, combining the slant distance and angle results between the infrared thermal imager and the target point from Step 3. The temperature compensation model is driven in real-time by the collected multimodal data to achieve online temperature compensation at the fire site.

[0067] This invention addresses the significant differences in physical characteristics across different areas of a fire, necessitating the division of the fire area into zones. By constructing differentiated temperature compensation models for each zone, it overcomes the interference of dynamic smoke and complex temperature distribution on infrared thermography, achieving high-precision online detection of the fire temperature field. This method not only improves the reliability of fire monitoring but also provides accurate temperature data support for fire emergency command. This zone-based fire temperature detection method fully utilizes the physical characteristics of different zones for targeted temperature compensation. In this method, the fire area is divided into real-time zones, identifying open flame, burnt-out, and unburned areas, and corresponding temperature compensation models are constructed for each zone's characteristics. For example, for open flame areas, a correction method based on the principle of radiative heat transfer and enhanced smoke transmittance can be used; for burnt-out areas, dynamic compensation can be achieved using a temperature gradient model; and for unburned areas, precise correction is performed based on fuel characteristics. This zone-based temperature compensation method effectively overcomes the accuracy bottleneck of traditional infrared thermography in complex fire environments.

[0068] In one embodiment of the present invention, to achieve effective fusion and consistency of multi-sensor data, after the above data acquisition is completed, it is necessary to accurately calibrate the spatial reference relationship between each sensor, and simultaneously handle the timestamp differences and environmental adaptability of different sensors. This process is the basis for subsequent optimization and analysis. Step one, which involves spatiotemporal registration of the fire scene's three-dimensional point cloud, visible light image, infrared thermal image, and pose data in the geographic coordinate system, includes the following steps:

[0069] Step 11: Based on 3D-2D constraint extrinsic parameter calibration, according to feature matching of fire scene 3D point cloud, infrared thermal image and visible light image, the spatial reference relationship between each sensor is initially constructed, and a unified coordinate system is established for each sensor;

[0070] Steps 1 and 2: To address the differences in timestamps collected by different sensors, the timestamps are aligned using spline interpolation.

[0071] Step 13: Design adaptive weights based on the performance characteristics of different sensors in different environments;

[0072] Step 14: Apply the LM algorithm to optimize the sensor extrinsic parameter matrix using the adaptively adjusted sensor weights, and then perform spatiotemporal registration of the data from each sensor. The optimization objective function is expressed as shown in formula (1):

[0073] (1)

[0074] in, It is a set of extrinsic parameter matrices; For the first The actual observation values ​​of each sensor; These are the predicted values ​​based on the extrinsic parameter matrix; For the calculation based on steps one and three The weight of each sensor; The objective is to minimize the positional difference between global observations and sensor data after extrinsic parameter transformation. For constraint functions; is the regularization coefficient.

[0075] In one embodiment of the present invention, in step one and three, adaptive weights are designed based on the characteristic performance of different sensors in different environments, specifically as follows:

[0076] For densely wooded areas, increase the weight of lidar; for areas with open flames, increase the weight of infrared thermal imagers; for areas with smoke but no visible flames, increase the weight of visible light cameras. The weight calculation formula is as follows:

[0077] (2)

[0078] in, For the first Basic feature scores for each sensor (such as point cloud density and image sharpness); The number of sensors; This is the environmental sensitivity factor, used to represent the degree of adaptability of a sensor to a specific environment, with a value range of [value range missing]. For example, smoke concentration, combustion intensity, etc.

[0079] In one embodiment of the present invention, after the initial division of the fire area, it is necessary to combine the characteristics and complementary advantages of multi-source data to perform more refined identification and dynamic adjustment of each type of area. Considering the obstruction caused by smoke and trees, as well as differences in fuel combustion characteristics, a multi-level contour aggregation method based on bright spot expansion is proposed, and the fire contour is updated to the map in real time in the form of latitude and longitude coordinates to ensure the accuracy and timeliness of command and firefighting. Step two includes the following process:

[0080] Step 21: Preprocess the visible light image by converting it from RGB to HSV color space and performing threshold segmentation to initially extract the open flame area. Use the same method to mark the burnt-out area that represents ash characteristics.

[0081] Step 22: Identify the local maxima of the highest temperature in the infrared thermal image to determine potential hot spots; starting from each maxima, gradually expand outwards and determine the location of significant temperature drop based on the decreasing temperature trend, thereby determining the boundary of the open flame area; using the boundary of the fire area as a reference for potential open flame areas that were not accurately identified by the visible light image due to factors such as smoke and wind, and using the transformation matrix of the visible light camera and the infrared thermal imager, spatially align and overlay the open flame area and potential open flame area to finally confirm the open flame area in the fire scene;

[0082] Steps 2 and 3: Based on the infrared thermal image, the area burning but without obvious flames is identified by utilizing the temperature distribution within the fire scene. Specifically, a multi-level contour aggregation method based on bright spot expansion is adopted. High-temperature bright spots in the infrared thermal image are treated as independent connected pixels, serving as the initial identifiers of the burning area. Morphological dilation is used to expand the bright spot area to cover the potential burnout area. The degree of dilation is dynamically and adaptively controlled by the surrounding temperature gradient. By analyzing the connectivity between these points and combining a region growing algorithm, the growth direction and range are dynamically determined based on the temperature characteristics of adjacent pixels. Isolated bright spots are gradually connected into a complete burnout area contour, and the burnout area contour is extracted as a unified contour covering all burning areas.

[0083] Step Two Four: In At any given moment, the outline coordinates of each region are projected onto the geographic coordinate system. At any time, it needs to be based on The contour line is updated and the region is divided at each time step. Calculation. The burnout zone of time and Open flame area at all times The intersection of the two is represented as According to the intersection Based on the characteristics of the region, the following area updates will be performed:

[0084] like (Intersection) Completely included ), then open flame area Set as a new open flame zone, indicated as ;

[0085] like (Intersection) Only partially included If so, the region division needs to be updated: (1) Remove intersections. After The remaining portion is assigned to a new open flame zone, denoted as... (2) Update the burnout zone to and The union of the burnout regions is represented as (3) Update the unburned area to the remaining area after removing the new burnout area and open flame area. , represented as ;

[0086] Step 25: For each subsequent time point (e.g.) Repeat steps two and three, and steps two and four, of the region update process. After each update, use... The regional outline of the moment and The contours at any given time are intersected and merged to generate updated open flame areas, burnt-out areas, and unburned areas. These areas are then overlaid onto the global contour map in the geographic coordinate system to gradually build complete fire area coverage.

[0087] In one embodiment of the present invention, for the specific application scenario of UAV carrying multiple sensors to monitor forest fires, the combined influence of slant distance, camera angle, and smoke concentration changes on the temperature measurement results is taken into account. Through theoretical derivation and parameterized analysis, these key influencing factors are accurately determined and used as the main input parameters to ensure the accuracy and reliability of the temperature measurement results. The slant distance between the infrared thermal imager and the target point in step three is expressed as shown in equation (3):

[0088] (3)

[0089] in, It is the slant distance between the infrared thermal imager and the target point; It is the camera's intrinsic parameter matrix; It is the scaling factor; It is the rotation matrix from the camera coordinate system to the radar coordinate system; These are pixel coordinates;

[0090] The formula for the angular relationship between the infrared thermal imager and the target point is expressed as equation (4):

[0091] (4)

[0092] in, The angle of a target point at any location in the image observed by the camera; This is the rotation matrix from the combined inertial navigation coordinate system to the northeast-sky coordinate system (ENU); This is the rotation matrix from the camera coordinate system to the combined inertial navigation coordinate system; This is the translation vector from the camera coordinate system to the combined inertial navigation coordinate system; Represents the transformed vector Quantity;

[0093] Based on the dark channel theory, the formula for estimating the average smoke transmittance of each region in a visible light image is expressed as equation (5):

[0094] (5)

[0095] in, Indicates the region Average smoke transmittance; For the region The total number of pixels in; Indicates the region Any coordinate in the array; Representing coordinates neighborhood Image coordinates within; It represents any one of the three color channels: red, green, and blue. Indicates color channel Foggy images; Indicates color channel The global atmospheric light is the average value of the 0.1% brightest pixels in the foggy image on that channel.

[0096] In one embodiment of the present invention, to improve the accuracy of fire temperature measurement, a differentiated compensation method based on regional characteristics is proposed to address temperature measurement interference factors in different fire areas. Specifically, for open flame areas, a compensation model based on radiation principles is designed, considering the effects of radiative heat transfer and the absorption and scattering of infrared radiation by smoke; while for burnout and unburnt areas, appropriate temperature compensation models are established based on the temperature gradient and fuel combustion characteristics differences within the areas. These compensation models achieve online compensation of fire temperature through real-time driving of multimodal data. Step four includes the following steps:

[0097] Step 41: For the open flame area, a compensation model based on the principle of radiative heat transfer is adopted. Considering the impact of radiative heat generated by intense combustion on the open flame area, as well as the absorption and scattering of infrared radiation by smoke, the following temperature compensation model is established:

[0098] (6)

[0099] in, The temperature of the target point within the compensated open flame area; The temperature of the target point within the open flame area as measured by an infrared thermal imager; It is the specific heat capacity at constant volume; It is the Stefan-Boltzmann constant proportional term; c is the speed of light; The radiation intensity of the target point measured by the infrared thermal imager is obtained from equation (7):

[0100] (7)

[0101] in, The area of ​​the detector pixel; for Calibration gain of the real-time infrared system; Atmospheric transmittance by slant path. , This is an empirical coefficient; For the magnification of the infrared system, , The effective focal length of the system; The directional emissivity curve of the target. , Let be the emissivity in the direction normal to the target. These are directional parameters related to the target material; The target grayscale value; The background grayscale value; Background radiation; The atmospheric path radiation between the infrared system and the target;

[0102] Step 42: For the burnout zone and the unburnt zone, based on the temperature gradient and differences in fuel combustion characteristics within the zone, the following temperature compensation model is established:

[0103] (8)

[0104] in, The temperature of the target point within the compensated burnout zone or unburned zone; and For polynomial coefficients, ; The ambient temperature.

[0105] The online compensation method (algorithm) for forest fire temperature error based on multimodal data fusion and regional division proposed in this invention is the underlying technical core of this invention, and various products can be derived based on the algorithm.

[0106] Based on the method (algorithm) proposed in this invention, an online compensation system for forest fire temperature error based on multimodal data fusion and regional division is developed using a programming language. This system has program modules corresponding to the steps of the above technical solution, and executes the steps in the above-mentioned online compensation method for forest fire temperature error based on multimodal data fusion and regional division when running.

[0107] The developed system (software) computer program is stored on a computer-readable storage medium. This computer program is configured to, when called by a processor, implement the steps of the aforementioned online compensation method for forest fire temperature errors based on multimodal data fusion and regional division. In other words, the invention is materialized on a carrier, becoming a computer program product.

[0108] Various implementations of the system described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] The computational programs (also referred to as programs, software, software applications, or code) of this invention include machine instructions of a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device PLD) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0110] The beneficial effects of this application will be described below with reference to specific embodiments.

[0111] Example 1

[0112] First, initial extrinsic parameter calibration is performed on the multi-sensor system. Initial values ​​of the extrinsic parameter matrix are obtained using a geometric calibration board combined with known prior positional relationships. After calibration, a UAV equipped with a lidar, visible light camera, thermal infrared camera, and inertial navigation system is used. Figure 2 (c) shows that multimodal data was simultaneously collected over the fire area, and the collection scenario is as follows. Figure 2 As shown in (a).

[0113] Based on the initial extrinsic parameter matrix, a transformation is performed to map observations from different sensors to the same global coordinate system. To ensure accurate alignment of multi-sensor observation data, an optimization objective function is defined, including sensor alignment error and geometric consistency constraints in the global coordinate system. The LM nonlinear optimization algorithm is used to iteratively update the extrinsic parameter matrix until the optimization error converges or the number of iterations is reached. After validating the optimization results in multiple scenarios and from multiple angles, it is ensured that high-precision alignment can be achieved. The final optimized extrinsic parameter matrix is ​​used for subsequent real-time data processing and multimodal information fusion.

[0114] The acquired visible light images were preprocessed, converting the RGB images to the HSV color space. Flame regions were extracted using threshold segmentation obtained through statistical analysis, and burnt-out regions exhibiting ash characteristics were marked. Subsequently, temperature gradient information from the infrared thermal images was combined to identify high-temperature regions with local maxima, and the analysis of temperature decrease trends was gradually expanded to determine the boundaries of the open flame region. Using the sensor extrinsic parameter matrix, the infrared thermal and visible light images were spatially aligned and superimposed to correct errors caused by external factors such as smoke and wind, ultimately accurately delineating the open flame region.

[0115] This invention presents a multi-level contour aggregation method based on bright spot expansion. It uses morphological dilation and region growing algorithms to connect high-temperature bright spots in infrared thermal images into coherent contours. To verify the advantages of this method, it addresses issues such as... Figure 3 The infrared thermal images of the forest shown are used for segmentation in two scenarios: one with tree occlusion and the other with less occlusion at the forest edge. Existing algorithms are then used to identify the fire contours. Existing Algorithm 1: The Otsu+Canny method is a hybrid algorithm combining Otsu thresholding and Canny edge detection. First, the Otsu method is used to determine a global threshold by maximizing the inter-class variance for initial fire area segmentation. Then, the Canny algorithm is used to accurately extract the fire edge contours. Existing Algorithm 2: The region-based level set algorithm is a segmentation method based on curve evolution. By introducing the regional energy of the target and background regions, it drives the evolution of the level set function to gradually approach the target boundary. Figure 3 As can be seen, the method of this invention can effectively reduce the impact of fuel thermal inertia and vegetation shading on fire area delineation, showing significant advantages compared to existing algorithms 1 and 2. The Otsu+Canny method is sensitive to interference from vegetation shading and fuel thermal inertia, and is prone to pseudo-contours or insufficient local edge connectivity in complex scenes. The region-based level set algorithm is robust and suitable for scenes with blurred boundaries or complex targets, but it has a large computational load and is highly dependent on initial conditions, which may cause contour loss due to tree shading.

[0116] This invention combines ash features from visible light images to further refine the identification of burnt-out areas. By subtracting the identified open flame and burnt-out areas, unburned areas are determined. The fire zone delineation results at a certain moment are as follows: Figure 4 As shown.

[0117] The intersection of the current and previous time zones is calculated, and the boundaries of the open flame zone, burnt-out zone, and unburned zone are dynamically adjusted based on the fire's development. The real-time updated zone outline is projected onto the geographic coordinate system and overlaid on the global fire map. After data processing is completed at the aerial end, the geographic coordinates are transmitted back to the ground end at regular intervals. Figure 2 (b) shows that it provides dynamic and accurate data support for firefighting command.

[0118] exist Figure 2 Thermocouples were placed at the location shown in (d), which is the midpoint between two trees. The geographic coordinates of this point were pre-calibrated and used for subsequent pixel coordinate mapping. Meteorological data were obtained through a solar radiometer, ground weather stations, and calculations using the general-purpose atmospheric radiation transfer software CART2. Emissivity needs to be determined by consulting the emissivity tables for various materials used in infrared thermometer measurements.

[0119] All collected parameters are input into the temperature compensation model of this invention and the formula. The established global parameter temperature compensation model, in which, For the surface emissivity of the flame, The atmospheric transmittance was used as the reference value, and the temperature measured by thermocouples was used as the true value for verification. The experimental results are as follows: Figure 5 As shown. Experiments show that:

[0120] (1) Without compensation, due to the influence of the external environment, the temperature measured by the infrared thermal imager is lower than the true value, with an average temperature error of 12.35 ℃;

[0121] (2) After temperature compensation using global parameters, the temperature measurement results during the open flame stage are close to the true value, while the temperature during the burnout stage is too high, and the average error is reduced to 8.47 ℃.

[0122] (3) After using regional temperature compensation, the temperature measurement results are more uniform and closer to the true value, and the average error is further reduced to 2.04 ℃.

[0123] By comparing uncompensated temperature, globally parameter compensated temperature, and regionally compensated temperature, it was verified that the present invention can effectively reduce the temperature measurement error of infrared thermal imagers in forest fire scenarios and significantly improve the measurement accuracy of fire thermal data.

[0124] It should be understood that the various processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of this invention, and all such changes and modifications will fall within the protection scope of this invention. Although this invention has been disclosed as above, the protection scope of this invention is not limited thereto.

Claims

1. A method for online compensation of forest fire temperature error based on multimodal data fusion and regional division, characterized in that, Includes the following steps: Step 1: Collect data in the fire area using a drone equipped with a lidar, visible light camera, infrared thermal imager and inertial navigation system. Obtain 3D point cloud, visible light image, infrared thermal image and pose data in geographic coordinate system provided by inertial navigation system. Perform spatiotemporal registration on the 3D point cloud, visible light image, infrared thermal image and pose data in geographic coordinate system. Step 2: Based on the spatiotemporally registered fused data, the fire scene is divided into three areas: the open flame area, the burnt-out area, and the unburned area; Step 3: Based on the 3D point cloud data of the fire scene from the lidar and the pose data from the inertial navigation system, obtain the slant distance and angle between the infrared thermal imager and the target point, and estimate the smoke transmittance of each area based on the visible light image. Step 4: Considering the radiant heat generated by intense combustion in the open flame area, and the absorption and scattering of infrared radiation by smoke, a temperature compensation model for the open flame area is established, combining the slant distance and angle results between the infrared thermal imager and the target point from Step 3, as well as the smoke transmittance results for each region. Based on the temperature gradient and fuel combustion characteristics differences between the burnt-out and unburned zones, temperature compensation models for the burnt-out and unburned zones are established, combining the slant distance and angle results between the infrared thermal imager and the target point from Step 3. The temperature compensation model is driven in real-time by the collected multimodal data to achieve online temperature compensation at the fire scene.

2. The online compensation method for forest fire temperature error based on multimodal data fusion and regional division according to claim 1, characterized in that, Step one involves spatiotemporal registration of the fire scene's 3D point cloud, visible light image, infrared thermal image, and pose data in a geographic coordinate system, including the following steps: Step 11: Based on 3D-2D constraint extrinsic parameter calibration, according to feature matching of fire scene 3D point cloud, infrared thermal image and visible light image, the spatial reference relationship between each sensor is initially constructed, and a unified coordinate system is established for each sensor; Steps 1 and 2: Align the timestamps to address the differences in data collection timestamps from different sensors; Step 13: Design adaptive weights based on the performance characteristics of different sensors in different environments; Step 14: Apply the LM algorithm to optimize the sensor extrinsic parameter matrix using the adaptively adjusted sensor weights, and then perform spatiotemporal registration of the data from each sensor. The optimization objective function is expressed as shown in formula (1): （1） in, It is a set of extrinsic parameter matrices; For the first The actual observation values ​​of each sensor; These are the predicted values ​​based on the extrinsic parameter matrix; For the first The weight of each sensor; For constraint functions; The regularization coefficient is used. This represents the number of sensors.

3. The online compensation method for forest fire temperature error based on multimodal data fusion and regional division according to claim 2, characterized in that, In step one and three, adaptive weights are designed based on the performance characteristics of different sensors in different environments, specifically as follows: For densely wooded areas, increase the weight of lidar; for areas with open flames, increase the weight of infrared thermal imagers; for areas with smoke but no visible flames, increase the weight of visible light cameras. The weight calculation formula is as follows: （2） in, For the first Basic feature scoring of each sensor; This is the environmental sensitivity factor, used to represent the degree of adaptability of a sensor to a specific environment, with a value range of [value range missing]. .

4. The online compensation method for forest fire temperature error based on multimodal data fusion and regional division according to claim 1, characterized in that, Step two includes the following process: Step 21: Preprocess the visible light image by converting it from RGB to HSV color space and performing threshold segmentation to initially extract the open flame area. Use the same method to mark the burnt-out area that represents ash characteristics. Step 22: Identify the regions with the highest local maxima in temperature by analyzing the temperature gradient in the infrared thermal image to determine potential high-temperature hotspots; Starting from each maximum value region, the area is gradually expanded outwards, and based on the decreasing temperature trend, the location of significant temperature drop is determined, thereby defining the boundary of the open flame region. Using the boundary of the fire area as the potential open flame area, and using the transformation matrix of the visible light camera and the infrared thermal imager, the open flame area and the potential open flame area are spatially aligned and superimposed to finally confirm the open flame area in the fire scene. Steps 2 and 3: Using a multi-level contour aggregation method based on bright spot expansion, high-temperature bright spots in the infrared thermal image are regarded as independent connected pixels as the initial identifier of the burning area; morphological dilation is used to expand the bright spot area to cover the potential burnout area; combined with the region growing algorithm, the growth direction and range are dynamically determined according to the temperature characteristics of adjacent pixels, and isolated bright spots are gradually connected into a complete burnout area contour. The burnout area contour is extracted as a unified contour covering all burning areas. Step Two Four: Calculation The burnout zone of time and Open flame area at all times The intersection of the two is represented as According to the intersection Based on the characteristics of the region, the following area updates will be performed: like Then open flame area Set as a new open flame zone, indicated as ; like Then the region division needs to be updated: (1) Remove intersections. After The remaining portion is assigned to a new open flame zone, denoted as... (2) Update the burnout zone to and The union of the burnout regions is represented as (3) Update the unburned area to the remaining area after removing the new burnout area and open flame area. , represented as ; Step 25: For each subsequent time point, repeat the region update process in Steps 23 and 24. After each update, use... The regional outline of the moment and The contours at any given time are intersected and merged to generate updated open flame areas, burnt-out areas, and unburned areas. These areas are then overlaid onto the global contour map in the geographic coordinate system to gradually build complete fire area coverage.

5. The online compensation method for forest fire temperature error based on multimodal data fusion and regional division according to claim 1, characterized in that, In step three, the slant distance between the infrared thermal imager and the target point is expressed as shown in equation (3): （3） in, It is the slant distance between the infrared thermal imager and the target point; It is the camera's intrinsic parameter matrix; It is the scaling factor; It is the rotation matrix from the camera coordinate system to the radar coordinate system; These are pixel coordinates; The formula for the angular relationship between the infrared thermal imager and the target point is expressed as equation (4): （4） in, The angle of a target point at any location in the image observed by the camera; This is the rotation matrix from the combined inertial navigation coordinate system to the northeast-sky coordinate system; This is the rotation matrix from the camera coordinate system to the combined inertial navigation coordinate system; This is the translation vector from the camera coordinate system to the combined inertial navigation coordinate system; Represents the transformed vector Quantity; The formula for the average smoke transmittance of each region in a visible light image is expressed as equation (5): （5） in, Indicates the region Average smoke transmittance; For the region The total number of pixels in; Indicates the region Any coordinate in the array; Representing coordinates neighborhood Image coordinates within; It represents any one of the three color channels: red, green, and blue. Indicates color channel Foggy images; Indicates color channel The global atmospheric light is the average value of the 0.1% brightest pixels in the foggy image on that channel.

6. A forest fire temperature error online compensation system based on multimodal data fusion and regional division, characterized in that, The system has a program module corresponding to the steps of the method described in any one of claims 1 to 5, and executes the steps in the above-described online compensation method for forest fire temperature error based on multimodal data fusion and regional division when it is run.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program configured to, when invoked by a processor, implement the steps of online compensation for forest fire temperature errors based on multimodal data fusion and regional division as described in any one of claims 1-5.

Images