Firefighting and rescue scene SLAM method based on tight coupling of thermal imaging and laser radar

Abstract

The present application relates to a kind of fire fighting and rescue scene SLAM method based on thermal imaging and laser radar tight coupling, it is related to fire emergency rescue field, including space-time synchronization, calibration and preprocessing to laser radar, thermal imager and IMU, the tight coupling feature extraction of thermal imaging and laser radar, carry out tight coupling front end odometry estimation, carry out the loop detection of thermal semantic enhancement, carry out multi-factor tight coupling back-end optimization, obtain global consistent key frame pose and thermal semantic constraint, construct the multi-layer fusion map including geometry layer, thermal radiation intensity layer and thermal semantic label layer, and output fire source label, high temperature dangerous area division, potential life label, passable area and navigation decision information.The present application realizes the stable positioning accuracy in dense smoke environment, accurately identifies and labels fire source, high temperature dangerous area and the like function, and can real-time construct the environment map containing thermal semantic information to support fire emergency rescue scene multi-rescue unit collaborative operation.

Description

Technical Field

[0001] This invention relates to the field of fire rescue, and to a synchronous positioning and mapping method for extreme environments in fire fighting and rescue, particularly a SLAM method for fire fighting and rescue sites based on the tight coupling of thermal imaging and lidar. Background Technology

[0002] In recent years, Simultaneous Localization and Mapping (SLAM) technology has been considered a core technology for achieving autonomous navigation and environmental perception in mobile robots. In structured, well-lit environments such as indoor warehouses, production workshops, and outdoor parks, SLAM systems based on vision and LiDAR have become increasingly mature, achieving centimeter-level positioning accuracy and high-fidelity environmental reconstruction. However, when SLAM technology is applied to extreme scenarios such as fire rescue, it faces challenges in adapting to the physical environment. Fire rescue sites are complex, nonlinear, and unstructured spaces characterized by high temperatures, dense smoke, open flames, water flow disturbances, and structural collapse. Traditional SLAM algorithms / systems commonly fail in fire rescue environments, exhibiting symptoms such as positioning drift, feature loss, map building failure, and system computational collapse. The limitations of existing technologies severely restrict the autonomous operation capabilities of equipment such as fire reconnaissance robots and search and rescue drones in enclosed spaces with low visibility and the risk of secondary disasters.

[0003] The main reasons for the failure of traditional SLAM are as follows: 1) Dense smoke causes physical failure of sensing methods: Visible light cameras suffer from image degradation and feature extraction and matching failures in dense smoke; when lidar passes through dense smoke, the ranging drops sharply and a large number of noisy point clouds are generated, causing point cloud registration failure. Existing technologies cannot solve the signal noise and feature destruction problems caused by smoke. 2) Extreme thermal radiation interference thermal imaging scheme: The high temperature difference at the fire fighting and rescue site causes problems such as overexposure, hot fog, and flame flickering in thermal imaging, resulting in missing texture and unstable geometric constraints in thermal imaging images. This makes it impossible to provide stable and reliable features for SLAM and makes it difficult to achieve stable positioning on its own. 3) Dynamic and semi-structured environments disrupt system stability: Flames, water flow, and heat waves cause the scene to change continuously and dynamically. Traditional SLAM cannot distinguish between static structures and dynamic disturbances, which easily leads to redundant data and positioning oscillations. In particular, when structural collapse or sudden environmental changes occur at the fire fighting and rescue site, it can also cause loop closure detection failure and many other problems such as accumulated errors that cannot be corrected. 4) Existing multi-sensor fusion solutions are insufficient: Existing technologies fail to achieve deep fusion of multi-source data such as laser, thermal imaging, and IMU, and cannot suppress dynamic noise; at the same time, they lack environmental adaptation mechanisms, and are prone to losing tracking under extreme conditions, making it difficult to meet the stable operation requirements of fire and rescue teams in actual fire fighting and rescue operations. Summary of the Invention

[0004] (a) Technical problems to be solved In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a SLAM method for fire fighting and rescue sites based on the tight coupling of thermal imaging and lidar, which can achieve stable positioning accuracy in dense smoke environment, accurately identify and mark fire source and high temperature hazard area, and can build environmental map containing thermal semantic information in real time to support the collaborative operation of multiple rescue units at the fire emergency rescue site.

[0005] (II) Technical Solution To achieve the above objectives, the main technical solution adopted by this invention is a SLAM method for fire fighting and rescue scenes based on the tight coupling of thermal imaging and lidar, comprising the following steps: S1. Perform spatiotemporal synchronization, calibration, and preprocessing of the lidar, thermal imager, and IMU: acquire the original 3D point cloud collected by the lidar, the original thermal radiation image collected by the thermal imager, and the angular velocity and acceleration images of the IMU data collected by the IMU; perform time synchronization on data of different frequencies; perform joint calibration of the lidar and thermal imager using a temperature difference checkerboard calibration board; perform smoke filtering on the original 3D point cloud, perform temperature calibration on the original thermal radiation image, and perform pre-integration processing on the IMU data to obtain IMU pre-integrated data; S2. Tightly Coupled Feature Extraction of Thermal Imaging and LiDAR: The preprocessed LiDAR point cloud is projected onto the thermal radiation image, and thermal radiation intensity and thermal radiation gradient are assigned to the points in the LiDAR point cloud to generate thermal attribute point cloud features. The thermal attribute geometric feature set is obtained through clustering or weighted feature extraction. At the same time, semantic features are extracted based on the thermal radiation image to generate a thermal semantic feature set including fire source, high temperature danger zone, potential life form and thermal edge. S3. Perform tightly coupled front-end odometry estimation: Based on the thermal attribute geometric feature set, thermal semantic feature set and IMU pre-integration data, construct a state vector containing robot motion state and fire-fighting and rescue site environmental parameters, construct geometric residuals, thermal radiation consistency residuals, thermal semantic residuals and IMU pre-integration residuals, and add them together to the front-end optimization objective function. Use the Gauss-Newton method or the Levenberg-Marquardt algorithm for nonlinear optimization to obtain the real-time pose and fire-fighting and rescue site environmental state estimates of continuous key frames; S4. Perform loop closure detection with enhanced thermal semantics: Call the historical keyframe feature database, construct geometric descriptors, thermal distribution descriptors, and semantic histogram descriptors based on the current keyframe and historical keyframes, calculate the geometric similarity, thermal distribution similarity, and semantic similarity between the current keyframe and historical keyframes respectively, dynamically adjust the weights of the three similarities according to the estimated environmental state of the fire-fighting and rescue site, select the candidate historical keyframe with the highest total similarity as the candidate loop closure frame for loop closure verification, the loop closure verification includes dual verification of point cloud registration residual test and thermal semantic consistency test, and generate loop closure constraints; S5. Perform multi-factor tightly coupled back-end optimization: Add the continuous keyframe poses and fire-fighting and rescue site environmental state estimation constraints output from step S3, the loop closure constraints output from S4, and the thermal semantic constraints and thermal radiation consistency constraints output from S2 to the factor graph. Construct odometry edges, geometric loop closure edges, fire source association edges, thermal semantic consistency edges, and thermal radiation consistency edges in the factor graph, and dynamically adjust the weights of semantic constraints and thermal radiation constraints according to information entropy, thermal radiation variance, and time decay factor. Use the Levenberg-Marquardt algorithm to perform global optimization on the factor graph to obtain globally consistent keyframe poses and thermal semantic constraints. S6. Construct a multi-layered fusion map based on global keyframe pose, including a geometry layer, a thermal radiation intensity layer, and a thermal semantic label layer, and output fire source annotation, high-temperature hazard zone delineation, potential life form annotation, passable area and navigation decision information.

[0006] In step S1, data of different frequencies are synchronized in time by a hardware trigger signal; for data that cannot be fully synchronized, linear interpolation is used to align the lidar data, thermal imager data and IMU data to a unified timestamp.

[0007] In step S1, smoke filtering is applied to the original 3D point cloud. Abnormal point clouds formed by smoke scattering are identified based on the reflectivity, distance, and neighborhood point density of the points and are filtered out, while solid structure point clouds are retained. Temperature calibration is performed on the original thermal radiation image, converting the original grayscale values ​​output by the thermal imager into physical temperature values. Thermal halo effect suppression is applied to high-temperature areas in the thermal image to obtain the corrected temperature field. Pre-integration processing is performed on the IMU data to obtain IMU pre-integrated data, thereby obtaining inertial constraints between adjacent time points or adjacent keyframes.

[0008] Step S2 involves semantic feature extraction based on thermal radiation images, including detecting fire sources based on temperature, flicker frequency, and shape irregularities; detecting high-temperature hazard zones based on dangerous temperature ranges and connected area; detecting potential life-generating heat sources based on temperature range, aspect ratio, and height constraints; and extracting thermal contrast edges based on temperature gradients and edge detection algorithms.

[0009] The state vector in step S3 includes at least position, attitude, velocity, gyroscope bias, accelerometer bias, smoke density estimate, and thermal radiation attenuation factor.

[0010] In step S4, the geometric descriptor is constructed based on the improved Scan Context, adding reflection intensity and thermal radiation intensity channels to the height distribution; the thermal distribution descriptor quantifies the temperature distribution into multiple levels and statistically analyzes the proportion of different temperature level points in each spatial sector to form a temperature distribution histogram; the semantic histogram descriptor is formed by statistically analyzing the number and spatial distribution dispersion of semantic categories including fire source, human body, doors and windows, fire protection facilities, high temperature surface, and smoke source.

[0011] In step S4, during loop closure verification, if the geometric error between the current frame and the candidate historical key frame after registration is less than a set threshold, and the spatial error after semantic feature point transformation is less than a set threshold, the loop closure is accepted.

[0012] In step S5, the odometry edge is used to constrain the relative motion relationship between adjacent keyframes, and a degradation factor is set according to the smoke density and thermal radiation attenuation factor to dynamically adjust the confidence of the odometry edge; the geometric closure edge is used to constrain the relative pose relationship between two keyframes that have a closure; the fire source association edge is used to constrain the consistency of the same fire source observed in different keyframes in the global space; the thermal semantic consistency edge is divided into fire source association edge and thermal structure edge. The fire source association edge constrains the relative relationship between different keyframes that observe the same fire source; the thermal structure edge constrains the projection consistency of thermal edge feature points across multiple frames. Thermal radiation consistency edges are used to constrain the thermal radiation values ​​of the same physical point in different keyframes to satisfy the attenuation model.

[0013] Step S6 specifically involves: based on the global keyframe pose optimized in S5, fusing multiple frames of laser point clouds into a unified world coordinate system, and constructing a geometric layer voxel map using the TSDF method to represent 3D structures such as walls, ground, and obstacles; for each voxel, fusing multiple frames of thermal radiation observations, calculating and updating the average thermal radiation value, thermal radiation variance, and confidence level of that voxel to form a thermal radiation intensity layer; for each voxel, fusing semantic observations from different keyframes using Dempster-Shafer evidence theory to form a thermal semantic label layer, and assigning semantic labels such as fire source, high-temperature danger zone, obstacle, and potential life form to the voxel.

[0014] Step S6, which generates application layer information based on the fused map, includes: determining passable areas based on geometric flatness and thermal radiation safety; dividing high-risk, medium-risk, and safe zones based on thermal radiation temperature; identifying heat flow channels based on temperature gradient fields; and outputting the location of fire sources, high-temperature hazardous areas, potential life forms, and robot navigation maps.

[0015] (III) Beneficial Effects The beneficial effects of this invention are: 1. This invention realizes the expression of multi-sensor data under a unified spatiotemporal reference by synchronizing and jointly calibrating the sensors in time and space, laying the hardware foundation for subsequent tight coupling fusion; through preprocessing of smoke filtering and thermal corona suppression, a large part of the systematic noise introduced by the fire fighting and rescue scene environment is removed from the data source before entering the core algorithm, which significantly improves the robustness of subsequent feature extraction. 2. This invention achieves complementary thermal and geometric information through the tight coupling and fusion of lidar and thermal imaging at the feature level, endowing each point in the point cloud with thermal attributes and enriching the dimensions of environmental perception. By extracting thermal semantic features specific to fire scenarios, it upgrades from simple environmental geometric modeling to task-oriented situational awareness, directly serving fire source location and hazard avoidance, and providing a basis for subsequent path planning, risk assessment, personnel location, etc., thereby providing key environmental perception capabilities for fire emergency rescue scenarios. 3. This invention achieves adaptive modeling of the dynamic environment of the fire-fighting and rescue site by explicitly modeling environmental parameters in the state vector. This makes state estimation not only a fitting of the robot's motion trajectory, but also a synchronous perception of the physical processes of the environment. The introduction of thermal radiation consistency residual and thermal semantic residual enables the system to maintain robust positioning even when the geometric features of the lidar are degraded due to dense smoke, relying on the stability of thermal radiation and semantic features. 4. The adaptive fusion multimodal loop closure detection strategy constructed in this invention enables dynamic adjustment of judgment criteria based on the current environmental quality of the fire-fighting and rescue site. When the geometric structure is damaged, it can successfully detect loop closures by relying on semantic clues such as heat distribution or fire source. The constructed geometric and thermal semantic dual verification mechanism minimizes the false detection rate of loop closures and prevents global map misalignment caused by loop closure errors. 5. By introducing degradation factors and adaptive weights, this invention enables backend optimization to proactively respond to changes in the fire and rescue scene environment, automatically weakening unreliable constraints caused by environmental degradation, while giving greater weight to stable, high-information semantic constraints, thus maintaining robustness in the optimization process; by adding high-level semantic information such as fire source and thermal edge as global constraints to the factor graph, it combines the fire and rescue scene situational awareness task with the robot localization and mapping task. 6. This invention constructs a multi-layered map, which fully preserves the geometric, thermal, and semantic information of the original data, resulting in a greater information content than traditional SLAM maps that only contain spatial points. The Dempster-Shafer evidence theory fusion method enables intelligent and probabilistic fusion of multi-source, conflicting, and uncertain information at the fire and rescue site, making the decision-making basis of the final map more reliable. Dynamic object processing and task-oriented navigation information extraction ensure that the final map is not only a snapshot of the past environment but also a strategic situation map that can be used for current decision-making, directly serving the autonomous navigation and on-site command of firefighting robots. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 A structural composition diagram of the system for implementing the present invention; Detailed Implementation

[0017] To better explain and facilitate understanding of this invention, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, is provided. This invention proposes a SLAM method for fire-fighting and rescue sites based on the tight coupling of thermal imaging and lidar. The core idea is to deeply integrate the thermal radiation and thermal semantic information provided by thermal imaging with the precise geometric information provided by lidar at various levels, including feature extraction, data association, state estimation, and map construction, to form a complementary and enhanced perception system. This system achieves stable positioning accuracy in dense smoke environments, accurate identification and labeling of fire sources and high-temperature hazardous areas, and real-time construction of an environmental map containing thermal semantic information to support collaborative operations of multiple rescue units at fire emergency rescue sites. A system implementing this method is as follows: Figure 2 As shown, the overall structure comprises three layers: a hardware support layer, a data processing layer, and an output application layer. The hardware support layer collects geometric, thermal radiation, and motion information of the fire-fighting and rescue scene. The data processing layer performs multi-sensor synchronization, feature fusion, odometry estimation, loop closure detection, and global optimization. The output application layer generates a thermal semantic-geometric fusion map, serving applications such as fire robot navigation, fire source localization, hazardous area delineation, and life-assistance identification.

[0018] The hardware support layer includes solid-state lidar, long-wave infrared thermal imager, and IMU.

[0019] Solid-state lidar is used to collect three-dimensional geometric point cloud data in the fire fighting and rescue scene environment. The point cloud data includes spatial coordinate information and laser reflection intensity information.

[0020] Its function is to provide three-dimensional geometric constraints for solid structures such as walls, door frames, stairs, obstacles, and ground, providing basic data for point cloud registration, geometric mapping, obstacle recognition, and passable area determination.

[0021] Its working principle is to obtain the distance between the target object and the sensor by emitting a laser beam and receiving the echo signal, and to form a three-dimensional point cloud. In this scheme, the point cloud output by the lidar is not used alone for traditional LiDAR SLAM, but is spatially projected and correlated with the thermal radiation image output by the thermal imager, so that the point cloud can further obtain thermal radiation properties.

[0022] Long-wave infrared thermal imagers are used to acquire two-dimensional thermal radiation images of the fire-fighting and rescue scene environment. Thermal images are usually grayscale images, and their grayscale values ​​are related to the temperature or thermal radiation intensity of the target area.

[0023] Its function is to acquire thermal semantic information such as fire source, high temperature area, thermal edge, and potential life source heat source, and to endow the laser point cloud with thermal radiation intensity and thermal radiation gradient, so that the three-dimensional point cloud is expanded from simple geometric data into multimodal feature data with temperature attributes.

[0024] Its working principle is to receive the infrared radiation emitted by the target object and convert the infrared radiation into image grayscale or temperature values. In this scheme, the thermal imager is not only used for fire source identification, but also participates in SLAM front-end registration, loop closure detection, back-end optimization, and map building.

[0025] An IMU is used to collect angular velocity and acceleration information of a robot or mobile platform.

[0026] Its function is to provide high-frequency motion prediction data for the system, and to improve the continuity and stability of pose estimation by using IMU pre-integration constraints when lidar or thermal imaging features degrade.

[0027] Its working principle is to measure angular velocity and linear acceleration by using a gyroscope and an accelerometer respectively, and then form motion constraints between adjacent keyframes after pre-integration processing, which are used for front-end odometry and back-end optimization.

[0028] The data processing layer includes: a spatiotemporal synchronization and calibration module, a tightly coupled feature extraction module, a tightly coupled front-end odometer module, a thermal semantic enhancement loop closure detection module, and a multi-factor tightly coupled back-end optimization module.

[0029] The spatiotemporal synchronization and calibration module is used to unify the heterogeneous data collected by lidar, thermal imager and IMU to the same time reference and spatial coordinate system, and to preprocess the smoke, noise and thermal interference caused by the fire fighting and rescue scene environment.

[0030] In terms of time synchronization, sensor data with different sampling frequencies are aligned to a unified timestamp through hardware trigger signals; for data that cannot be fully synchronized, linear interpolation is used to interpolate data from different times to the same time point.

[0031] For spatial calibration, the system uses a temperature difference checkerboard calibration plate to solve for the rigid body transformation matrix between the lidar coordinate system and the thermal imager coordinate system. This temperature difference checkerboard calibration plate can be made of aluminum plate and heating film to form a thermal difference pattern, enabling the thermal imager to identify the checkerboard corner points, while the lidar can obtain the corresponding three-dimensional points, thereby obtaining the extrinsic parameters by minimizing the reprojection error.

[0032] In terms of thermal image preprocessing, the system first converts the original grayscale values ​​output by the thermal imager into physical temperature values; then it identifies high-temperature areas and performs radiation attenuation compensation on the pixels around the high-temperature areas to suppress the thermal halo effect caused by open flames or high-temperature objects.

[0033] In terms of laser point cloud smoke filtering, the system constructs a smoke point discrimination function based on the characteristics of smoke points, such as low reflectivity, close distance, and spatial sparseness. This function filters out abnormal point clouds formed by smoke scattering from the original point cloud, while retaining point clouds of solid structures such as walls and obstacles.

[0034] Finally, the preprocessed point cloud after smoke removal, the corrected thermal radiation image after eliminating the influence of thermal halo, and the IMU pre-integrated data are obtained. The tightly coupled feature extraction module is used to perform the first deep fusion of the preprocessed laser point cloud and thermal image to extract two types of features: geometric point cloud features with thermal attributes and thermal semantic features such as fire source, high temperature zone, potential life form, and thermal edge.

[0035] For thermal attribute-related point cloud features, the 3D points in the lidar coordinate system are first transformed to the thermal imager coordinate system using the calibrated extrinsic parameter matrix. Then, the 3D points are projected onto the thermal image plane using the thermal imager intrinsic parameters. Subsequently, the thermal radiation intensity corresponding to the projected point is read using bilinear interpolation, and the thermal radiation gradient at that location is calculated.

[0036] In this way, each laser point no longer only includes three-dimensional coordinates and reflection intensity, but also thermal radiation value and thermal radiation gradient, forming a multi-attribute feature point. This multi-attribute feature point can be understood as: three-dimensional position + laser reflection intensity + thermal radiation intensity + thermal radiation gradient.

[0037] For thermal semantic features, based on the corrected thermal image, the following information is extracted: First, fire source characteristics. The system combines temperature threshold, inter-frame flicker frequency, and shape irregularity to calculate the probability that a pixel belongs to a fire source. When the probability is higher than a set threshold, it is marked as a fire source.

[0038] Second, characteristics of high-temperature hazard zones. The system identifies areas where the temperature is within the hazardous temperature range and the area of ​​the connected region is greater than the minimum threshold, and marks these areas as high-temperature hazard zones.

[0039] Third, the characteristics of potential living organisms as heat sources. Based on areas close to human body temperature, and combined with morphological constraints such as aspect ratio and height, the system identifies heat source areas that may match the heat generation characteristics of the human body.

[0040] Fourth, thermal contrast edge features. The system performs edge detection on the thermal image, extracting the contour boundaries of building structures such as walls, doors, windows, and partitions in the thermal image to form thermal edge features.

[0041] Finally, a multimodal feature set is output, including dense thermal attribute geometric features and sparse high-level thermal semantic features. This multimodal feature set serves as the common input for subsequent front-end odometry and back-end optimization.

[0042] The tightly coupled front-end odometry module is used to estimate the robot's current pose in real time based on the multimodal feature matching relationship between consecutive frames and IMU pre-integration information, and at the same time estimate the environmental state parameters of the fire fighting and rescue site.

[0043] Unlike conventional visual-inertial or laser-inertial SLAM, which only estimates position, attitude, velocity, and IMU bias, this scheme further incorporates smoke density estimates and thermal radiation attenuation factors into the state vector. Smoke density reflects the impact of smoke on laser ranging and point cloud quality; the thermal radiation attenuation factor describes the degree of thermal radiation attenuation in smoke or the fire-fighting / rescue scene medium.

[0044] The front-end odometer generates the following types of residuals: First, geometric residuals, including point-to-edge residuals and point-to-plane residuals, are used to maintain the geometric constraints in traditional point cloud registration.

[0045] Second, thermal radiation consistency residual. This residual is based on the assumption that the thermal radiation value of the same physical point is relatively stable over a short period of time, and combined with the thermal radiation attenuation model, it constrains the consistency of thermal radiation intensity of corresponding physical points in adjacent frames.

[0046] Third, thermal semantic feature residuals. These residuals include fire source location consistency residuals and thermal edge-geometric edge alignment residuals. The fire source location consistency residuals are used to constrain the spatial location consistency of the same fire source in different frames within a short period of time; the thermal edge-geometric edge alignment residuals are used to provide supplementary constraints through structural edges in the thermal image when geometric features degrade.

[0047] Fourth, IMU pre-integration residuals. These residuals are used to constrain motion changes between adjacent frames, improving the continuity of pose estimation.

[0048] The aforementioned residuals are uniformly constructed as optimization objectives, and the state increments are solved using the Gauss-Newton method or the Levenberg-Marquardt algorithm, thereby updating the robot pose and environmental state variables.

[0049] Finally, the real-time pose of consecutive frames and the latest estimates of the environmental conditions at the fire and rescue site are obtained, including smoke density and thermal radiation attenuation factor.

[0050] The thermal semantic enhancement loop closure detection module is used to determine whether the robot has reached a historical position, thereby detecting loops and correcting accumulated errors. Since smoke, high temperature, flames, and structural changes in the fire-fighting and rescue environment can cause traditional geometric loop closure detection to fail, this module introduces thermal distribution and thermal semantic information to enhance loop closure judgment.

[0051] Three types of descriptors are constructed for the current keyframe: First, the geometric descriptor. This descriptor is built based on an improved Scan Context, adding a laser reflection intensity channel and a thermal radiation intensity channel to the traditional height distribution channel, forming a geometric-thermal three-channel descriptor.

[0052] Second, the thermal distribution descriptor. The system quantifies the temperature distribution in thermal images or thermal attribute point clouds into multiple temperature levels and calculates the proportion of points at different temperature levels within each spatial sector, thereby forming a temperature distribution histogram.

[0053] Third, semantic histogram descriptors. The system statistically analyzes the frequency and spatial distribution dispersion of semantic features such as fire sources, human bodies, doors and windows, fire-fighting facilities, high-temperature surfaces, and smoke sources in the current keyframe, forming semantic feature vectors.

[0054] During similarity fusion, the weights of the three similarity categories—geometric similarity, thermal distribution similarity, and semantic similarity—are dynamically adjusted based on the smoke density and fire source saliency output by the front-end odometer. For example, when the smoke density is high, the weight of geometric similarity is reduced, while the weights of thermal distribution and semantic similarity are increased; when the fire source saliency is high, the judgment weights of fire source semantics and thermal distribution are increased.

[0055] After the candidate loop closures are determined, dual verification is performed using point cloud registration residual testing and thermal semantic consistency testing: Cloud registration residual test to determine whether the geometric error after ICP registration is less than the threshold; The hot semantic consistency test determines whether the positional error of the transformed semantic feature points is less than a threshold.

[0056] The system will only accept the loopback if both the geometric error and the thermal semantic error meet the requirements.

[0057] Finally, the loop closure constraint is output, which includes the two keyframes that form the loop and their relative pose relationship.

[0058] The multi-factor tightly coupled back-end optimization module is used to integrate front-end odometry constraints, loop closure detection constraints, and global hot semantic constraints into the factor graph for joint optimization, thereby eliminating cumulative errors and obtaining globally consistent robot trajectory and map base data.

[0059] Construct a factor graph containing the states of multiple keyframes. Each keyframe is a vertex, and its state includes not only position and orientation, but also thermal radiation attenuation factor and smoke density estimate.

[0060] Construct the following constraint edges in the factor graph: Odometry Edge: This edge is used to constrain the relative pose changes between adjacent keyframes and introduces a degradation factor based on smoke density and thermal radiation attenuation. The confidence level of this edge is automatically reduced when environmental degradation leads to a decrease in odometry reliability.

[0061] Geometric closure edge: This edge is output by the closure detection module and is used to constrain the relative pose relationship between two keyframes where a closure occurs.

[0062] Fire source association edge: This edge is used to constrain the same fire source observed in different keyframes, so that the position of the same fire source in the global coordinate system remains consistent.

[0063] Fourth, thermal structure edges. These edges are used to constrain the spatial consistency of matched thermal edge features in different keyframes within the global coordinate system.

[0064] Thermal radiation consistency edge: This edge is used to constrain the thermal radiation observations of the same physical point in different keyframes to satisfy the thermal radiation attenuation model.

[0065] Regarding weight adjustment, the weights of semantic edges and thermal radiation edges are dynamically adjusted based on information entropy, thermal radiation variance, and time decay factor. When the determinism of the fire source or semantic features is high, the weight of semantic constraints is increased; when the violent fluctuation of the flames leads to a large thermal radiation variance, the weight of thermal radiation constraints is decreased. Finally, the Levenberg-Marquardt algorithm is used for global optimization.

[0066] Finally, the optimized global keyframe poses and optimized global hot semantic constraints are output. The optimization results are used for subsequent map construction and can also guide the dynamic weight adjustment in loop closure detection.

[0067] The output application layer includes a thermal semantics and geometric fusion map construction module, as well as a business application module based on the map.

[0068] The thermal semantic and geometric fusion map building module is used to fuse optimized global pose, multi-frame point cloud, thermal image and thermal semantic features into a multi-layer 3D map, so that the map not only contains spatial geometric structure, but also fire rescue decision information such as temperature, hazard level, fire source location, potential life location and passable area.

[0069] Geometry layer update: Using the optimized pose, the laser point cloud is fused into the global voxel map through the TSDF algorithm to reconstruct geometric surfaces such as walls, ground, and obstacles.

[0070] Thermal radiation intensity layer update: For each voxel, maintain the thermal radiation weighted average, thermal radiation variance, and confidence level to represent the temperature level and its stability at that spatial location.

[0071] Hot semantic label layer update: Using the Dempster-Shafer evidence theory, the semantic observations of the same voxel from different keyframes are fused as evidence, and finally the voxel is assigned semantic labels such as fire source, high temperature zone, obstacle, potential life form, etc., while retaining uncertainty.

[0072] Dynamic object processing: Identify dynamic targets such as jumping flames and moving people by using motion consistency scores and thermal radiation variance, and mark them as dynamic voxels so that they do not participate in static map updates.

[0073] Map compression: The map is compressed using an octree structure. The decision to merge child nodes is based on the geometric deviation and thermal radiation intensity variance within the nodes, thereby reducing storage while maintaining the detailed representation of complex areas.

[0074] Navigation decision map output: Combines geometric flatness and thermal radiation safety to determine passable areas; divides high-risk areas, medium-risk areas and safe areas according to the average thermal radiation; identifies heat flow channels or high-temperature gas flow directions through temperature gradient fields.

[0075] Finally, a multi-layered 3D map is output, specifically including: Three-dimensional geometric model of building structure; Mean, variance, and confidence level of thermal radiation at each spatial location; Semantic labels for each spatial location; Information such as fire sources, high-temperature zones, obstacles, and potential life forms is marked; Information on the robot's traversable areas, hazard levels, and navigation decision-making.

[0076] like Figure 1 As shown, the present invention provides a fire-fighting and rescue scene SLAM method based on the tight coupling of thermal imaging and lidar, which includes the following steps: S1. Perform spatiotemporal synchronization, calibration, and preprocessing on lidar, thermal imager, and IMU: The system acquires raw 3D point cloud data from lidar, including coordinates and reflection intensity; raw thermal radiation images from thermal imager, which are grayscale images with grayscale values ​​correlated with temperature; and IMU data angular velocity and acceleration images from IMU. Time synchronization is performed on data from different frequencies; hardware trigger signals are used to align sensor data from different frequencies to a unified timestamp. For data that cannot be precisely aligned, a linear interpolation model is used to interpolate sensor data from different times to the same point in time.

[0077] For any time Sensor data is time-aligned using linear interpolation: ; in Indicates time pose or point cloud, and These represent the adjacent lidar scanning times.

[0078] Joint calibration of the lidar and thermal imager was performed using a temperature-difference checkerboard calibration plate. A heating film was adhered to the surface of an aluminum plate to form a temperature-difference checkerboard pattern. The calibration plate's plane appeared as a checkerboard pattern with temperature differences in the thermal imager. The transformation matrix was solved by minimizing the reprojection error of the laser point cloud corner points projected onto the thermal image. ; Let the coordinate system of the lidar be... The thermal imager coordinate system is The world coordinate system is The goal of sensor calibration is to solve the problem from... arrive rigid body transformation matrix ; It is a three-dimensional special Euclidean group (rigid body transformation group).

[0079] Set the first on the calibration board The pixel coordinates of the corner points in the thermal image are: The corresponding three-dimensional coordinates in the lidar point cloud are .

[0080] Projection model: The thermal imager uses a pinhole model, and is set... The intrinsic parameter matrix, focal length and the main point .

[0081] ; The process of projecting laser points onto a thermal image is as follows: ; in For perspective projection functions: , Let be the homogeneous pixel coordinates after projection. The extrinsic parameters are solved by minimizing the reprojection error, using Lie algebras. right Perform parametric rotations and translations, and then iteratively solve using Gauss-Newton or Levenberg-Marquardt algorithms to finally obtain the result. .

[0082] Smoke filtering is applied to the original 3D point cloud. Based on the reflectivity, distance and neighborhood point density of the smoke points, abnormal point clouds formed by smoke scattering are identified and filtered out, while solid structure point clouds are retained. Temperature calibration is performed on the original thermal radiation image, the original grayscale value output by the thermal imager is converted into a physical temperature value, thermal halo effect is suppressed in high-temperature areas of the thermal image, and the corrected temperature field is obtained. Pre-integration processing is performed on the IMU data to obtain IMU pre-integrated data, thereby obtaining inertial constraints between adjacent time points or adjacent keyframes.

[0083] Step S1 outputs preprocessed point cloud data (clean point cloud after removing smoke); corrected thermal radiation image (eliminating the influence of thermal halo, expressed as physical temperature value); and IMU pre-integrated data (used for subsequent state estimation).

[0084] Through spatiotemporal synchronization and joint calibration, the expression of multi-sensor data under a unified spatiotemporal reference was realized, laying the hardware foundation for subsequent tightly coupled fusion.

[0085] By preprocessing with smoke filtering and thermal corona suppression, a large portion of the systematic noise introduced by the fire-fighting and rescue scene environment (dense smoke, high temperature) is removed from the data source before entering the core algorithm, which significantly improves the robustness of subsequent feature extraction.

[0086] S2. Tightly coupled feature extraction of thermal imaging and lidar: Project the preprocessed lidar point cloud onto the thermal radiation image: Transform each laser point to the thermal imager coordinate system using calibration parameters, and then project it onto the image plane through the thermal imager intrinsic parameters to obtain the pixel coordinates.

[0087] Assigning thermal radiation intensity and thermal radiation gradient to points in the lidar point cloud: Thermal radiation values ​​are obtained using bilinear interpolation, and the thermal radiation gradient is calculated in image space: Thermal radiation intensity represents the gradient vector in image space; it points in the direction of the fastest temperature change in the thermal image, and its magnitude indicates the severity of the temperature change. In firefighting and rescue operations, flame boundaries, the interface between high-temperature gases and air, and the contact surfaces of different combustibles typically exhibit large thermal gradients.

[0088] Generate a set of feature points for thermal attribute point cloud. Each feature point is a six-dimensional vector that integrates 3D position, laser reflectivity, thermal radiation value, and thermal gradient. Obtain the set of geometric features for thermal attributes through clustering or weighted feature extraction.

[0089] Simultaneously, semantic features are extracted based on thermal radiation images, including: Fire source detection is based on temperature, flicker frequency, and shape irregularity: For each pixel, three feature values ​​are calculated: the degree to which the temperature exceeds the threshold, the flicker feature value, and the shape irregularity. Then, the three features are weighted and summed, and the probability is obtained by using the Sigmoid function. Pixels with probabilities higher than the set probability threshold (e.g., 0.7) are identified as fire sources.

[0090] High-temperature danger zones are detected based on the dangerous temperature range and the area of ​​the connected region: Temperatures within the danger range [T_min, T_max] are identified. This is the lower limit of the dangerous temperature (e.g., 60°C, which exceeds human tolerance). Areas with a dangerous upper temperature limit (e.g., 300°C, exceeding which may indicate ignition) and whose connected area is greater than the minimum threshold are marked as "high temperature danger zones".

[0091] The detection of potential living body heat sources is based on temperature range, aspect ratio, and height constraints. Threshold segmentation identifies all regions with temperatures close to human body temperature as candidates. Then, for each candidate region, three features—average temperature, aspect ratio, and height—are calculated. The probability that each feature matches human characteristics is then calculated. Assuming the three features are independent, the total probability is the product of the three features. Finally, by comparing the probability with a set probability threshold, regions exceeding the threshold are identified as human bodies. This combination of temperature range and shape constraints enables the detection of humanoid heat sources.

[0092] Thermal contrast edges are extracted based on temperature gradient and edge detection algorithms: Canny edge detection is performed on the thermal image to extract the contours of building structures (walls, doors and windows) and form thermal edge features.

[0093] Using the Canny edge detector, dual threshold detection is performed by calculating temperature gradient, gradient magnitude, and gradient direction. Weak edge pixels are connected to strong edge pixels to form continuous edge recognition of the outline of building structures such as walls, doors, and windows.

[0094] Finally, a thermal semantic feature set is generated, including fire source, high-temperature danger zone, potential life form, and thermal edge; this set contains a variety of semantic labels, such as <pixel location, category: fire source>, <pixel location, category: high-temperature zone>, <connected region, category: potential life form>, <pixel location, category: thermal edge>, etc.

[0095] By tightly coupling and fusing lidar and thermal imaging at the feature level, complementary thermal information (temperature, gradient) and geometric information (position, shape) are achieved, endowing each point in the point cloud with thermal attributes and enriching the dimensions of environmental perception. By extracting thermal semantic features specific to fire scenarios (fire source, high-temperature zone), the system upgrades from simple environmental geometric modeling to task-oriented situational awareness, directly serving fire source location and hazard avoidance. This provides a basis for subsequent path planning, risk assessment, and personnel positioning, thus providing crucial environmental perception capabilities for fire emergency rescue scenarios.

[0096] S3. Perform tightly coupled front-end odometry estimation: Utilize the multimodal features output from S2, combined with IMU data, to perform tightly coupled registration and state estimation between continuous frames / new frames and the local map, and output the robot's real-time pose at every instant.

[0097] Based on thermal property geometric feature sets, thermal semantic feature sets, and IMU pre-integrated data, a state vector is constructed that includes the robot's motion state and environmental parameters of the fire-fighting and rescue site. Compared to traditional VIO / SLAM, which only estimates pose and IMU bias, this invention considers environmental parameters, which are directly related to and coupled with sensor observations. The motion state vector is defined as follows: : ; in: Position vector ; Rotation quaternions; It is the velocity vector; The gyroscope and accelerometer are respectively zero biased; For smoke density estimation, this will affect the ranging accuracy of lidar; This is the thermal radiation attenuation factor, used to describe the attenuation of thermal radiation in smoke.

[0098] Construct geometric residuals: Establish point-to-edge distance residuals to align edge points of the current frame to edge lines in the map; establish point-to-plane distance residuals to align plane points of the current frame to planes in the map. Constructing thermal radiation consistency residuals: Based on the physical assumption that the thermal radiation value of the same object is stable over a short period of time, and considering the attenuation of thermal radiation in smoke, constructing the radiation intensity residual of the same physical point between adjacent frames. This is equivalent to adding a thermal radiation dimension constraint to laser point cloud registration; Constructing thermal semantic residuals: Using the fire source location in the thermal semantic feature set, construct a fire source location consistency residual; using thermal edge features, construct an alignment residual between the thermal edge and the geometric edge, providing additional constraints when geometric features degrade; assuming the fire source location is basically fixed in a short time (except for moving fire sources), establish a fire source location consistency residual; based on the consistency between the thermal edge (temperature change boundary) and the geometric edge (object boundary) in physical space, align the thermal edge point with the nearest geometric edge point, providing supplementary constraints when lidar fails (smoke scattering) or visible light fails (darkness), and establish a thermal edge alignment residual.

[0099] Constructing IMU pre-integration residuals: Constructing the residuals between IMU measurements and state predictions.

[0100] Since IMU integration depends on the initial pose, it is necessary to re-integrate each time the initial pose changes are optimized. Therefore, a pre-integration scheme is introduced to integrate the IMU measurement in the body coordinate system to obtain the relative motion and establish the residual between the IMU pre-integration residual predicted measurement value and the state prediction value. They are then incorporated into the front-end optimization objective function, and nonlinear optimization is performed using the Gauss-Newton method or the Levenberg-Marquardt algorithm. By iteratively solving the state increment, the state vector is continuously updated to obtain the real-time pose of continuous key frames and the estimated environmental state of the fire-fighting and rescue site. By explicitly modeling environmental parameters (smoke density, thermal radiation attenuation) at the fire-fighting and rescue site in the state vector, adaptive modeling of the dynamic environment at the fire-fighting and rescue site is achieved. This makes state estimation not only a fitting of the robot's motion trajectory, but also a synchronous perception of the physical processes of the environment. The introduction of thermal radiation consistency residuals and thermal semantic residuals enables the system to maintain robust positioning even when the geometric features of the lidar are degraded due to dense smoke. This is achieved by relying on the stability of thermal radiation and semantic features (such as the fixed location of the fire source), which greatly improves robustness.

[0101] S4. Perform loop closure detection with enhanced thermal semantics: Smoke, high temperature and flame are often present at fire rescue sites. Traditional loop closure detection based on single geometric features is prone to failure. Therefore, this invention will integrate geometric, thermal distribution and semantic information to achieve highly robust and accurate loop closure detection, so as to reduce the unique interference factors of fire emergency rescue scenarios such as smoke, thermal disturbance and dynamic flame.

[0102] The historical keyframe feature database is invoked, and geometric descriptors, thermal distribution descriptors, and semantic histogram descriptors are constructed based on the current keyframe and historical keyframes. Among them, the geometric descriptor is constructed based on the improved Scan Context, and reflection intensity and thermal radiation intensity channels are added to the height distribution to form a 3-channel geometric-thermal descriptor.

[0103] The thermal distribution descriptor quantifies the temperature distribution of the thermal image into multiple levels and statistically analyzes the proportion of points at different temperature levels within each spatial sector to form a temperature distribution histogram. It also statistically analyzes the proportion of point clouds in different temperature ranges within each sector to form a temperature distribution histogram, and then forms a circular partition with the robot-centered Scan Context.

[0104] Semantic histograms are used to statistically analyze the frequency of occurrence and spatial distribution dispersion of various semantic features (fire source, human body, doors and windows, etc.) in the current frame, forming a semantic histogram. Quantitative normalization and spatial weighting methods are applied to perform double normalization on fire sources, human bodies, doors and windows, fire-fighting facilities, high-temperature surfaces, smoke sources, etc. at the fire rescue scene, establishing spatial distribution sensitivity weights for each category of semantics.

[0105] Geometric similarity, thermal distribution similarity, and semantic similarity are calculated between the current keyframe and historical keyframes. Geometric similarity is calculated using the cosine similarity of column vectors. Thermal distribution similarity is constructed based on temperature level weights. Semantic similarity is calculated by combining histogram intersection and spatial distribution consistency.

[0106] The weights of the three similarities are dynamically adjusted based on the estimated environmental state of the fire and rescue site. The candidate historical keyframe with the highest total similarity is selected as the candidate loop closure frame for loop closure verification. The loop closure verification includes dual verification of point cloud registration residual test and thermal semantic consistency test, and loop closure constraints are generated. Point cloud registration residual verification calculates the root mean square error after ICP registration. The relative pose transformation is calculated using the ICP algorithm, and the registration error is calculated to measure the average alignment deviation of two point clouds under a given transformation. The smaller the registration error, the better the registration effect.

[0107] The hot semantic consistency test calculates the positional error of the transformed semantic feature points. For a matched semantic feature pair, the transformed positional error is calculated. If the geometric error between the current frame and the candidate historical key frame after registration is less than a set threshold, and the spatial error of the transformed semantic feature points is less than a set threshold, the loopback is accepted.

[0108] This step employs an adaptive fusion multimodal loop closure detection strategy, enabling the system to dynamically adjust its judgment criteria based on the current environmental quality (smoke, fire intensity) at the fire-fighting and rescue site. When the geometric structure is damaged, it can successfully detect loop closures by relying on semantic clues such as heat distribution or fire source. The dual verification mechanism of geometry and thermal semantics minimizes the false detection rate of loop closures and prevents global map misalignment caused by loop closure errors.

[0109] S5. Perform multi-factor tightly coupled back-end optimization: Add the continuous keyframe pose and fire-fighting and rescue site environmental state estimation constraints output in step S3, the loop closure constraints output in S4, and the thermal semantic constraints and thermal radiation consistency constraints output in S2 to the factor graph for joint optimization, and finally output a globally consistent robot trajectory and map.

[0110] Construct odometry edges, geometric loop edges, fire source-related edges, thermal structure edges, and thermal radiation consistency edges in the factor graph: Odometry edges are used to constrain the relative motion relationship between adjacent keyframes, and degradation factors are set according to smoke density and thermal radiation attenuation factor to dynamically adjust the confidence of odometry edges. Geometric closure edges are used to constrain the relative pose relationship between two keyframes where a closure occurs; Fire source association edges are used to constrain the consistency of the same fire source observed in different keyframes in the global space; Thermal semantic consistency edges are divided into fire source-related edge residuals and thermal structure edge residuals. The former constrains the relative relationships between different keyframes observing the same fire source; the latter constrains the projection consistency of thermal edge feature points across multiple frames. Thermal radiation consistency edges are used to constrain the thermal radiation values ​​of the same physical point in different keyframes to satisfy the attenuation model.

[0111] The weights of semantic and thermal radiation constraints are dynamically adjusted based on information entropy, thermal radiation variance, and time decay factor. For example, when the fire source location is explicitly defined as a stationary point (low information entropy), its constraint weight is increased; when the flame is violently fluctuating (large thermal radiation variance), the weight of the thermal radiation edge is automatically reduced. The Levenberg-Marquardt algorithm is used to globally optimize the factor graph to obtain globally consistent keyframe poses and thermal semantic constraints.

[0112] This step, through the introduction of degradation factors and adaptive weights, enables the backend optimization to proactively respond to changes in the fire-fighting and rescue site environment. It automatically weakens unreliable constraints caused by environmental degradation, while giving greater weight to stable, high-information semantic constraints, thus maintaining the robustness of the optimization process. By adding high-level semantic information such as fire source and thermal edge as global constraints to the factor graph, the fire-fighting and rescue site situational awareness task and robot localization and mapping task are intrinsically and tightly coupled together.

[0113] S6. Construct a multi-layered fusion map based on global keyframe pose, including a geometry layer, a thermal radiation intensity layer, and a thermal semantic label layer, and output fire source annotation, high-temperature hazard zone delineation, potential life form annotation, passable area and navigation decision information.

[0114] Based on the S5-optimized global keyframe pose, multiple frames of laser point clouds are fused into a unified world coordinate system, and a geometric layer voxel map is constructed using TSDF to represent 3D structures such as walls, ground, and obstacles. For each voxel, multiple frames of thermal radiation observations are fused, and the average thermal radiation value, variance, and confidence level of that voxel are calculated and updated to form a thermal radiation intensity layer. For each voxel, semantic observations from different keyframes are fused using Dempster-Shafer evidence theory to form a thermal semantic label layer. Dempster-Shafer theory can effectively handle conflicting information (such as identifying a fire source in one time period and the ground in another). Finally, each voxel is assigned the most reasonable semantic label while preserving the unknowns. Semantic labels for fire source, high-temperature danger zone, obstacle, and potential life form are assigned to the voxels.

[0115] By calculating motion consistency scores, time-varying objects (such as flickering flames or moving people) are separated from the map and marked as dynamic voxels, which do not participate in static map updates.

[0116] The motion one-time detection rule is as follows: For a candidate dynamic point, calculate its motion consistency across different frames. If the motion one-time score is greater than the preset motion one-time threshold and the thermal radiation variance is greater than the preset thermal radiation variance threshold, then the dynamic point is filtered out.

[0117] The map is compressed and stored using an octree data structure. The decision to merge child nodes is based on the geometric deviation of voxels within a node and the variance of thermal radiation intensity.

[0118] The system extracts task-oriented information from the fused map and outputs a navigation decision map. Combining geometric flatness and thermal radiation safety, it marks areas where the robot can safely pass. Based on the average temperature of the thermal radiation of voxels, it divides high-risk, medium-risk, and safe zones. It identifies heat flow channels (direction of high-temperature gas flow) based on the temperature gradient field. Finally, it outputs a multi-layered, business-value-added 3D map, including a 3D geometric model of the building structure, the average thermal radiation and confidence level of each spatial location, semantic labels for each spatial location (fire source, high-temperature zone, obstacles, etc.), and a navigation map for the robot (passable areas, hazard levels).

[0119] This step constructs a multi-layered map, which fully preserves the geometric, thermal, and semantic information of the original data, resulting in a much larger information content than traditional SLAM maps that only contain spatial points. The Dempster-Shafer evidence theory fusion method enables intelligent and probabilistic fusion of multi-source, conflicting, and uncertain information at the fire and rescue site, making the decision-making basis of the final map more reliable. Dynamic object processing and task-oriented navigation information extraction ensure that the final map is not only a snapshot of the past environment but also a strategic situation map that can be used for current decision-making, directly serving the autonomous navigation and on-site command of firefighting robots.

[0120] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0121] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A SLAM method for fire fighting and rescue scenes based on tight coupling of thermal imaging and lidar, characterized in that, Includes the following steps: S1. Perform spatiotemporal synchronization, calibration, and preprocessing of the lidar, thermal imager, and IMU: acquire the original 3D point cloud collected by the lidar, the original thermal radiation image collected by the thermal imager, and the IMU data angular velocity and acceleration images collected by the IMU; perform time synchronization of data at different frequencies; and perform joint calibration of the lidar and thermal imager using a temperature difference checkerboard calibration board. Smoke filtering is applied to the original 3D point cloud, temperature calibration is performed on the original thermal radiation image, and pre-integration processing is performed on the IMU data to obtain IMU pre-integrated data. S2. Tightly Coupled Feature Extraction of Thermal Imaging and LiDAR: The preprocessed LiDAR point cloud is projected onto the thermal radiation image, and thermal radiation intensity and thermal radiation gradient are assigned to the points in the LiDAR point cloud to generate thermal attribute point cloud features. The thermal attribute geometric feature set is obtained through clustering or weighted feature extraction. At the same time, semantic features are extracted based on the thermal radiation image to generate a thermal semantic feature set including fire source, high temperature danger zone, potential life form and thermal edge. S3. Perform tightly coupled front-end odometry estimation: Based on the thermal attribute geometric feature set, thermal semantic feature set and IMU pre-integration data, construct a state vector containing robot motion state and fire-fighting and rescue site environmental parameters, construct geometric residuals, thermal radiation consistency residuals, thermal semantic residuals and IMU pre-integration residuals, and add them together to the front-end optimization objective function. Use the Gauss-Newton method or the Levenberg-Marquardt algorithm for nonlinear optimization to obtain the real-time pose and fire-fighting and rescue site environmental state estimates of continuous key frames; S4. Perform loop closure detection with enhanced thermal semantics: Call the historical keyframe feature database, construct geometric descriptors, thermal distribution descriptors, and semantic histogram descriptors based on the current keyframe and historical keyframes, calculate the geometric similarity, thermal distribution similarity, and semantic similarity between the current keyframe and historical keyframes respectively, dynamically adjust the weights of the three similarities according to the estimated environmental state of the fire-fighting and rescue site, select the candidate historical keyframe with the highest total similarity as the candidate loop closure frame for loop closure verification, the loop closure verification includes dual verification of point cloud registration residual test and thermal semantic consistency test, and generate loop closure constraints; S5. Perform multi-factor tightly coupled back-end optimization: Add the continuous keyframe poses and fire-fighting and rescue site environmental state estimation constraints output from step S3, the loop closure constraints output from S4, and the thermal semantic constraints and thermal radiation consistency constraints output from S2 to the factor graph. Construct odometry edges, geometric loop closure edges, fire source association edges, thermal semantic consistency edges, and thermal radiation consistency edges in the factor graph, and dynamically adjust the weights of semantic constraints and thermal radiation constraints according to information entropy, thermal radiation variance, and time decay factor. Use the Levenberg-Marquardt algorithm to perform global optimization on the factor graph to obtain globally consistent keyframe poses and thermal semantic constraints. S6. Construct a multi-layered fusion map based on global keyframe pose, including a geometry layer, a thermal radiation intensity layer, and a thermal semantic label layer, and output fire source annotation, high-temperature hazard zone delineation, potential life form annotation, passable area and navigation decision information.

2. The SLAM method for fire fighting and rescue sites based on tight coupling of thermal imaging and lidar as described in claim 1, characterized in that: In step S1, data of different frequencies are synchronized in time by a hardware trigger signal; for data that cannot be fully synchronized, linear interpolation is used to align the lidar data, thermal imager data and IMU data to a unified timestamp.

3. The SLAM method for fire fighting and rescue sites based on tight coupling of thermal imaging and lidar as described in claim 1, characterized in that: In step S1, smoke filtering is performed on the original 3D point cloud. Abnormal point clouds formed by smoke scattering are identified based on the reflectivity, distance, and neighborhood point density of the points and are filtered out, while solid structure point clouds are retained. Temperature calibration is performed on the original thermal radiation image, converting the original grayscale values ​​output by the thermal imager into physical temperature values. Thermal halo effect is suppressed in high-temperature areas of thermal images to obtain the corrected temperature field; IMU data is pre-integrated to obtain IMU pre-integrated data, and inertial constraints between adjacent time points or adjacent keyframes are obtained.

4. The SLAM method for fire fighting and rescue sites based on tight coupling of thermal imaging and lidar as described in claim 1, characterized in that: Step S2 involves semantic feature extraction based on thermal radiation images, including detecting fire sources based on temperature, flicker frequency, and shape irregularities; detecting high-temperature hazard zones based on dangerous temperature ranges and connected area; detecting potential life-generating heat sources based on temperature range, aspect ratio, and height constraints; and extracting thermal contrast edges based on temperature gradients and edge detection algorithms.

5. The SLAM method for fire fighting and rescue sites based on tight coupling of thermal imaging and lidar as described in claim 1, characterized in that: The state vector in step S3 includes at least position, attitude, velocity, gyroscope bias, accelerometer bias, smoke density estimate, and thermal radiation attenuation factor.

6. The SLAM method for fire fighting and rescue sites based on tight coupling of thermal imaging and lidar as described in claim 1, characterized in that: In step S4, the geometric descriptor is constructed based on the improved Scan Context, adding reflection intensity and thermal radiation intensity channels to the height distribution; the thermal distribution descriptor quantifies the temperature distribution into multiple levels and statistically analyzes the proportion of different temperature level points in each spatial sector to form a temperature distribution histogram; the semantic histogram descriptor is formed by statistically analyzing the number and spatial distribution dispersion of semantic categories including fire source, human body, doors and windows, fire protection facilities, high temperature surface, and smoke source.

7. The SLAM method for fire fighting and rescue sites based on tight coupling of thermal imaging and lidar as described in claim 1, characterized in that: In step S4, during loop closure verification, if the geometric error between the current frame and the candidate historical key frame after registration is less than a set threshold, and the spatial error after semantic feature point transformation is less than a set threshold, the loop closure is accepted.

8. The SLAM method for fire fighting and rescue sites based on tight coupling of thermal imaging and lidar as described in claim 1, characterized in that, In step S5, the odometry edge is used to constrain the relative motion relationship between adjacent keyframes, and a degradation factor is set according to the smoke density and thermal radiation attenuation factor to dynamically adjust the confidence of the odometry edge; the geometric closure edge is used to constrain the relative pose relationship between two keyframes that have a closure; the fire source association edge is used to constrain the consistency of the same fire source observed in different keyframes in the global space; the thermal semantic consistency edge is divided into fire source association edge and thermal structure edge. The fire source association edge constrains the relative relationship between different keyframes that observe the same fire source; the thermal structure edge constrains the projection consistency of thermal edge feature points across multiple frames. Thermal radiation consistency edges are used to constrain the thermal radiation values ​​of the same physical point in different keyframes to satisfy the attenuation model.

9. The SLAM method for fire fighting and rescue sites based on tight coupling of thermal imaging and lidar as described in claim 1, characterized in that, Step S6 specifically involves: based on the global keyframe pose optimized in S5, fusing multiple frames of laser point clouds into a unified world coordinate system, and constructing a geometric layer voxel map using the TSDF method to represent the three-dimensional structure of walls, ground, and obstacles; for each voxel, fusing multiple frames of thermal radiation observations, calculating and updating the average thermal radiation value, thermal radiation variance, and confidence level of that voxel to form a thermal radiation intensity layer; for each voxel, fusing semantic observations from different keyframes using Dempster-Shafer evidence theory to form a thermal semantic label layer, and assigning semantic labels such as fire source, high-temperature danger zone, obstacle, and potential life form to the voxel.

10. The SLAM method for fire fighting and rescue sites based on tight coupling of thermal imaging and lidar as described in claim 1, characterized in that, Step S6, which generates application layer information based on the fused map, includes: determining passable areas based on geometric flatness and thermal radiation safety; dividing high-risk, medium-risk, and safe zones based on thermal radiation temperature; identifying heat flow channels based on temperature gradient fields; and outputting the location of fire sources, high-temperature hazardous areas, potential life forms, and robot navigation maps.

Images