Multi-sensor fusion positioning and mapping equipment and thermal imaging equipment and computer-readable media

Abstract

This invention discloses a multi-sensor fusion positioning, mapping, and thermal imaging equipment and a computer-readable medium. The equipment integrates positioning and mapping sensors such as a thermal imaging camera, a visible light camera, a lidar system, and an inertial navigation system, specifically designed for high-risk and complex fire emergency environments. Through an edge-cloud integrated cluster collaborative system and multi-link stable transmission technology, combined with intelligent decision support, this invention can achieve high-precision real-time positioning and dynamic 3D temperature measurement and mapping in environments with weak or unavailable satellite navigation signals. The system's multi-mode portability and environmental adaptability design ensure the efficiency and safety of the operator / unmanned system under extreme conditions, while its advanced data processing and analysis technology significantly improves the accuracy and response speed of rescue operations. These innovative features give this invention significant practical application value and market prospects in the field of fire emergency rescue.

Description

Technical Field

[0001] This invention relates to the field of mapping and positioning, and in particular to a multi-sensor fusion positioning and mapping and thermal imaging equipment and a computer-readable medium. Background Technology

[0002] In emergency scenarios, such as fire rescue, mine disasters, and underground facility search and rescue, complex environmental conditions pose significant challenges to positioning, mapping, and thermal imaging capabilities. These scenarios typically involve low-light or no-light conditions, dense smoke, extreme temperature variations, narrow passages, and multiple obstacles, and cannot rely on satellite navigation signals. Existing single-sensor systems often fail to provide sufficient accuracy, robustness, and real-time performance in such extreme environments. Therefore, there is an urgent need for a multi-sensor fusion system that combines the advantages of various sensors to provide comprehensive environmental perception, high-precision positioning, and thermal imaging information support.

[0003] Thermal imaging cameras offer significant advantages in low-light or no-light environments, generating clear thermal images by capturing the thermal radiation information of objects. In fire emergency rescue scenarios, thermal imaging cameras can penetrate smoke, detect high-temperature targets, and identify fire sources and living beings (such as trapped personnel), providing crucial support for rescue efforts. Even in environments with insufficient or no visible light, the equipment can still effectively perceive the environment and generate temperature distribution maps by combining data from thermal imaging cameras, providing more comprehensive information for decision-making. LiDAR, with its high-precision 3D point cloud data, is an important tool for 3D environment modeling. Its data can be used for accurate environmental reconstruction, performing particularly well in complex terrain and confined spaces. The fusion of LiDAR data and infrared thermal images further enhances the system's adaptability in dynamic environments, providing rescue equipment with precise positioning and real-time mapping capabilities.

[0004] Inertial navigation systems (INS) can achieve precise short-term position and attitude estimation by measuring acceleration and angular velocity in the absence of external signals or references, making them particularly suitable for underground environments or enclosed spaces. In a short time, INS can provide the system with high-precision pose information, ensuring continuous positioning and navigation in extreme environments. Furthermore, Global Navigation Satellite Systems (GNSS, such as GPS and BeiDou) provide global positioning capabilities, offering reliable long-term positioning support, especially in open environments or before entering indoor / underground facilities. By fusing GNSS global positioning data with other sensors (inertial navigation, magnetometer, barometer, etc.), the system can achieve continuous navigation and mapping over a wide area, while also efficiently fusing absolute and relative positions using multi-sensor information. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-sensor fusion positioning, mapping, and thermal imaging equipment and a computer-readable medium. By fusing data from multiple sensors, including thermal imaging cameras, lidar, visible light cameras, inertial navigation systems, and satellite systems, a system is formed that combines real-time positioning, accurate mapping, and temperature detection capabilities, making it particularly suitable for complex fire emergency scenarios. The complementary advantages of each sensor provide robust sensing capabilities under different environmental conditions. Simultaneously, the thermal imaging function generates intuitive information on fire sources, living beings, and temperature distribution, providing comprehensive support for emergency rescue and thus improving the efficiency and safety of emergency response.

[0006] The technical solution of this invention is:

[0007] Multi-sensor fusion positioning and mapping and thermal imaging equipment, including:

[0008] The sensor module includes at least a thermal imaging camera and an inertial navigation system;

[0009] The data fusion and mapping module is responsible for processing data from multiple sensors to fuse and map data, including building a local two-dimensional temperature occupancy map.

[0010] Preferably, the data fusion mapping module also includes the construction of a multi-machine global three-dimensional environmental temperature map on the server side.

[0011] Preferred options also include:

[0012] The scene perception module senses dynamic and static environmental information in emergency scenarios in real time, including sensing the movement of equipment itself, real-time tracking of dynamic targets, and detection of high and low temperature targets.

[0013] Preferred options also include:

[0014] The multi-sensor fusion positioning module integrates data from multiple sensors to generate device pose estimation.

[0015] Preferably, the scene perception module perceives the device's own motion through an inertial navigation system combined with its own motion perception model: the inertial navigation system monitors the device's pose changes using data from accelerometers and gyroscopes; based on a thermal imaging camera and the inertial navigation system, the self-motion perception model optimizes data processing to ensure accurate identification of movement, falls, going up and down stairs / escalators / elevators, walking, and running; the perception of the device's own motion helps the system dynamically adjust sensor strategies, providing motion parameters and decision support for path planning.

[0016] Preferably, the inertial navigation system combines its own inertial measurement unit data and thermal imaging data, utilizes sensor fusion technology, and optimizes motion estimation through Kalman filtering or particle filtering algorithms to reduce the impact of sensor noise and improve positioning accuracy. Specifically, the inertial navigation system collects acceleration information of the device through accelerometers, obtains angular velocity data through gyroscopes, and estimates the instantaneous velocity and displacement of the device by combining sensor data fusion models. To ensure that the system can accurately distinguish between vertical and horizontal motion states, its own motion perception model introduces a motion state classification algorithm in data processing, using a combination of threshold-based detection methods, machine learning algorithms, and deep learning algorithms. The Z-axis data of the accelerometer is used to monitor whether there are obvious changes in vertical acceleration, thereby determining whether it is a vertical motion state such as going up or down stairs, escalators, or elevators. The data based on the gyroscope is used to determine whether it is a horizontal walking or running dynamic state.

[0017] Preferably, the scene perception module's perception of the device's own motion enables the inertial navigation system to dynamically adjust the sensor's operating mode and strategy. Based on the device's motion state, the inertial navigation system selects an adaptive sampling rate, adjusts the sensor's operating frequency, or switches between a high-precision mode and a low-power mode to ensure efficient and accurate data acquisition. Through collaborative work with the path planning module, the inertial navigation system provides real-time motion parameters and decision support for path planning, thereby achieving better positioning and navigation performance in complex environments.

[0018] Preferably, for different motion states, the self-motion perception model further optimizes path planning and system parameter adjustment through joint analysis of inertial measurement unit data and thermal imaging data, ensuring that it can maintain high positioning accuracy and robustness in dynamic and complex environments.

[0019] Preferably, the self-motion perception model employs a deep neural network that fuses information from multiple sensors;

[0020] The self-motion perception model uses a convolutional neural network to extract heat source features from infrared images and combines them with a recurrent neural network to process time-series sensor data provided by the inertial navigation system, track the device's motion trajectory and predict its position changes; the convolutional neural network infers the device's motion state by capturing the time dependence of sensor data, and can maintain stability even in the presence of noise or signal loss.

[0021] Preferably, the self-motion perception model is further designed with an adaptive weight allocation fusion judgment module; the fusion judgment module dynamically adjusts the weights based on the density of heat source distribution in the scene, thereby avoiding incorrect judgments in areas with scarce infrared features or large interference, and ensuring the accuracy of motion state estimation and thermal imaging results.

[0022] Preferably, the scene perception module tracks dynamic targets in real time by processing thermal imaging camera data through a lightweight dynamic target recognition model, thereby achieving real-time perception of dynamic targets. The lightweight dynamic target recognition model analyzes multi-frame information, tracks moving personnel or equipment, removes noise points, and extracts key dynamic features, thereby improving the system's perception accuracy of dynamic targets and providing real-time dynamic information support.

[0023] Preferably, the lightweight dynamic target recognition model is based on multi-layer convolutional neural networks for temporal feature extraction. The lightweight dynamic target recognition model includes a shallow feature extraction part and a deep feature extraction part. Targets are divided into small targets and large targets. The shallow feature extraction part is used to detect small targets, while the deep feature extraction part is used to identify large targets. An interactive branch differential network is introduced into the lightweight dynamic target recognition model to perform dynamic target indexing in time, further filtering out misidentified targets and ensuring the accuracy of target recognition and tracking.

[0024] Preferably, the scene perception module detects high and low temperature targets by using thermal imaging camera data and a temperature analysis model to detect heat radiation sources in the environment and identify areas with abnormal temperatures; it also performs in-depth analysis of infrared imaging data to generate a heat distribution map, thereby locating fire sources or potential rescue targets.

[0025] Preferably, the scene perception module also automatically identifies and marks entrances and floors; the scene perception module automatically identifies and marks entrances, floors and attack points in the environment through a depth model;

[0026] The depth model automatically generates a spatial model of a building by analyzing data from thermal imaging cameras, and identifies the locations of entrances, floors, and attack points; the floor identification and entrance / exit detection algorithms, combined with inertial navigation system data, determine the floor location in real time.

[0027] Preferably, the scene perception module automatically identifies and marks entrances and floors, and the corresponding sensor module further includes a lidar module and / or a barometer and / or a visible light camera module.

[0028] Preferably, the scene perception module employs a deep neural network representation. By analyzing 3D point cloud data provided by the lidar module and / or barometer data and / or image data captured by the visible light camera module, it automatically generates a spatial model of the building and identifies entrances, floors, and attack point locations. The scene perception module uses a spatial convolutional neural network to extract local features from the lidar module and / or visible light camera module data and further identifies the geometric shapes of these structures. Furthermore, a graph neural network is used to construct a spatial relationship graph to optimize the topological structure between spatial elements, thereby accurately identifying the relative positions of entrances and stairs. The floor recognition and entrance detection algorithm combines barometer and inertial navigation data to calculate the absolute height of the current floor of the building through depth feature representation and determine the floor position in real time.

[0029] Preferably, the scene perception module automatically identifies and marks entrances, floors, and attack points, and can also manually mark or modify them.

[0030] Preferably, the scene perception module also performs smoke and dust identification, and the corresponding sensor module further includes a lidar module and / or a visible light camera module; the scene perception module automatically identifies the smoke and dust environment by analyzing the data from the lidar module and / or the visible light camera module; the lidar module generates high-precision point cloud data of the surrounding environment by emitting lasers and receiving return signals; the scene perception module uses data preprocessing technology to determine the presence of smoke or dust by analyzing the sparsity, reflection intensity changes, and distance changes of the point cloud.

[0031] Preferably, in the smoke and dust recognition of the scene perception module, the corresponding sensor module also includes a visible light camera module; the image data provided by the visible light camera module assists in the recognition of environmental features of smoke and dust; through threshold-based detection methods, machine learning algorithms or deep learning algorithms, the scene perception module extracts the visual features of smoke or dust from the image, and by combining the image's illumination changes, edge detection and color difference features, it determines the presence of smoke and dust in the dynamic environment.

[0032] Preferably, in the smoke and dust recognition of the scene perception module, based on historical data and environmental models, the scene perception module gradually optimizes its recognition accuracy, and provides accurate environmental perception information in real time in complex and dynamically changing environments.

[0033] Preferably, the multi-sensor fusion positioning module further includes a lidar module, and the multi-sensor fusion positioning module includes specific sub-modules:

[0034] Data cleaning module: First, the data from the thermal imaging camera, inertial navigation module and lidar module are preprocessed. Noise is filtered out by a noise suppression model. Then, the data from multiple sensors are synchronized by a sensor synchronization signal to ensure that the observation data from different sensors are synchronized in time.

[0035] Odometry positioning module: By fusing multi-source data from thermal imaging camera, inertial navigation module and lidar module through filtering, optimization or end-to-end state estimation methods, the pose of the device is recursively estimated to build a multi-sensor fusion odometer; through state estimation model, nonlinear motion in emergency scenarios is handled, thereby ensuring that the device can still provide high-precision real-time positioning when there is no satellite signal or the signal is interfered with.

[0036] Preferably, the odometer positioning module filters the thermal image data acquired by the thermal imaging camera to smooth stripe noise, and further extracts key radiation features based on local radiation gradients; after feature extraction, feature association is performed based on local radiation feature descriptors to ensure robust matching of heat sources in the infrared image; the local radiation feature descriptors are obtained by encoding local heat source regions in the infrared image.

[0037] Preferably, the sensor module corresponding to the multi-sensor fusion positioning module further includes a lidar module. To balance the efficiency and accuracy of lidar matching, a classification feature matching method is adopted.

[0038] When surface features are identified in the LiDAR data, a point-to-surface distance matching method is used; while for non-surface features, a point-to-distribution matching method is used. The low dimensionality of surface features reduces the amount of computation, and the point-to-distribution constraint can be used to enhance the robustness of the algorithm on non-surface features.

[0039] Preferably, to address potential degradation in the environment, an adaptive point cloud downsampling and matching parameter adjustment strategy is introduced to ensure the accuracy and stability of matching when the point cloud density is low or the feature distribution is uneven. In the event of eventual degradation, a fast normal distribution transformation is used for point cloud registration to ensure that the algorithm can still provide accurate matching and localization results even under extreme conditions.

[0040] Preferably, the sensor module further includes a satellite navigation module, and the corresponding multi-sensor fusion positioning module includes sub-modules:

[0041] Relocation and Global State Optimization Module: Utilizes global position information and relocation constraints from the satellite navigation module to correct long-term pose errors, thereby improving the overall system's positioning robustness and accuracy.

[0042] Preferably, the multi-sensor fusion positioning module further includes a sub-module:

[0043] Positioning feedback and navigation guidance module: It integrates positioning data and environmental perception information to provide real-time navigation guidance for rescue equipment, and also supports relative positioning between multiple devices.

[0044] Preferably, the relocation and global state optimization module corrects long-term pose errors, specifically including:

[0045] Absolute position observation fusion of satellite positioning and navigation modules: In outdoor scenarios, the global position information provided by the satellite positioning and navigation module serves as an absolute pose reference, reducing cumulative drift. A multi-modal data fusion strategy is employed, using the satellite's absolute position data as a low-frequency update soft constraint, which is then fused with high-frequency data from the lidar module, thermal imaging camera, and inertial navigation to gradually correct the pose. Furthermore, an adaptive data reliability adjustment mechanism is introduced, automatically reducing the weight of satellite data when the satellite signal is weak or obstructed, and using the fusion results from lidar, thermal imaging camera, and inertial navigation.

[0046] Multi-sensor location re-identification: The corresponding sensor module also includes a visible light camera module; a depth re-localization algorithm is used to fuse multi-sensor data to overcome the limitations of traditional re-localization algorithms in complex environments; firstly, a re-localization feature description algorithm is used to extract global feature descriptors from LiDAR point clouds and construct a historical point cloud database; candidate matching frames of the current point cloud in the historical database are retrieved and determined; then, multimodal matching of infrared and / or visible light images is introduced, and a similarity model is used to calculate the matching similarity between the current image and candidate images to optimize the candidate frame ranking; in smoke or low-light environments, the scene perception module automatically judges the scene state and prioritizes the use of infrared image data to ensure the robustness of the re-localization algorithm; after determining the re-localization constraint markers, point cloud registration is performed to generate the re-localization relative pose, which serves as the input for global pose graph optimization;

[0047] Global pose map optimization: By combining the state estimation method of multi-sensor fusion, the generated repositioning relative pose, and the absolute position observation provided by the satellite navigation system, a global pose map is constructed and optimized; by solving the optimization problem, the optimal estimate of the pose at all historical moments is obtained, the historical frame sub-map and global map are updated, long-term accumulated errors are eliminated, and a globally consistent high-precision map is achieved.

[0048] Preferably, the positioning feedback and navigation guidance module integrates positioning data and environmental perception information to provide real-time navigation guidance for rescue equipment, while also supporting relative positioning between multiple devices, specifically including:

[0049] Through collaborative computing and communication between devices, combined with decision-making algorithms, accurate relative pose determination and path optimization are achieved, building an efficient multi-device collaborative network. The decision-making algorithm analyzes multi-sensor data in real time, assesses environmental complexity, dynamic obstacle distribution, and device operating status, providing intelligent support for navigation guidance. For different scenarios, the decision-making algorithm dynamically adjusts navigation strategies, prioritizing obstacle avoidance paths in scenarios with dense obstacles or dense smoke. In life detection missions, it optimizes collaborative paths between devices to quickly reach the target location.

[0050] Preferably, the equipment also includes a communication module to enable real-time communication between the equipment and the control center; the positioning information is fed back to the control center in real time through the communication module, and the global information is comprehensively analyzed to generate real-time scheduling suggestions and optimize the task allocation and collaboration path between multiple devices.

[0051] Preferably, multiple devices communicate with each other via LoRa, 4G, or WiFi; in the case of poor 4G and WiFi signals, they communicate with each other via LoRa and calculate the floor difference and 2D horizontal distance between them.

[0052] Preferably, the real-time dispatch suggestions generated by the control center also include dispatching personnel from other devices to rescue people who are trapped or in distress on a certain device.

[0053] Preferably, the equipment consists of one or more units, each of which communicates in real time with the control center and server via a relay base station.

[0054] Preferably, the data fusion mapping module specifically includes the following tasks:

[0055] Single-device data fusion and local 2D temperature occupancy map construction: On a single device, thermal imaging data captured by a thermal imaging camera and local point clouds generated by an inertial navigation system are fused together, and an algorithm model is used to construct a 2D temperature occupancy map. The 2D temperature occupancy map represents the spatial structure and temperature distribution information around the device in a rasterized form, displaying fire sources, living organisms, and high-temperature hazard areas. The algorithm model intelligently filters noise and optimizes the raster update rate by analyzing infrared thermal images and point cloud data in real time, ensuring the accuracy and real-time performance of the map construction. In dynamic environments, the algorithm model automatically adjusts the perception weights according to environmental changes.

[0056] Preferably, the sensor module corresponding to the data fusion mapping module also includes a lidar module. Under smoke-free and dust-free conditions, the local point cloud generated by fusing thermal imaging data captured by the thermal imaging camera and data from the inertial navigation system and lidar module is used to construct a two-dimensional temperature occupancy map using an algorithm model.

[0057] Preferably, the construction of the multi-machine global three-dimensional environmental temperature map on the server side specifically includes:

[0058] Server-side multi-machine global 3D environmental temperature mapping and dynamic updating: The server integrates sensor data from multiple devices and utilizes scene re-identification and inter-device relocalization technologies to construct a globally consistent 3D fused environment map. It dynamically updates static structures and dynamic targets within the 3D map, achieving refined modeling of the global environment. The scene re-identification algorithm analyzes infrared thermal image data from multiple devices to identify and match similar scene areas, aligning observation data between different devices. Inter-device relocalization detection technology, combined with global feature descriptors generated by deep learning, retrieves similar scenes from a historical point cloud database. Further verification using infrared images determines the relocalization matching frame. The server utilizes the fused relocalization constraint information to globally optimize the local maps of each device, eliminating drift accumulation errors through pose adjustment to generate a global 3D map.

[0059] Preferably, the sensor modules corresponding to the server-side multi-machine global 3D environmental temperature mapping and dynamic update also include a lidar module and a visible light camera module. In specific steps, the scene re-identification algorithm analyzes multi-sensor data from lidar point clouds, infrared thermal images, and visible light images from multiple devices to identify and match similar scene areas, thereby achieving alignment of observation data between different devices. The multi-machine relocation detection technology, combined with global feature descriptors generated by deep learning, searches for similar scenes in the historical point cloud database and determines the relocation matching frame through further verification using visible light and infrared images.

[0060] Preferably, the construction of the multi-machine global three-dimensional environmental temperature map on the server side further includes:

[0061] Accurate segmentation of remote 3D map: The server-side global 3D environment map is accurately segmented using a segmentation algorithm, precisely classifying objects in the scene into at least one of the following elements: entrances / exits, walls, ground, obstacles, and living organisms; based on the segmentation model, it utilizes fused multi-source data and combines global scene features to achieve semantic segmentation and object recognition.

[0062] Preferably, the construction of the multi-machine global three-dimensional environmental temperature map on the server side further includes:

[0063] 3D rendering of temperature map: The remote 3D environment map is further combined with thermal imaging data from thermal imaging cameras to achieve 3D rendering of temperature map through temperature mapping algorithm; the real-time temperature information is superimposed on the 3D point cloud to intuitively display the temperature distribution in different areas of the scene, including the specific location and temperature change trend of at least one of the attack point, fire source, heat source and living organism.

[0064] Preferably, the equipment adopts end-to-cloud integrated cluster collaboration and multi-link stable transmission technology: it adopts end-to-cloud data processing and cluster collaboration technology, and supports multiple signal transmission methods and relay communication modes between end and cloud.

[0065] Preferably, the equipment is wearable equipment.

[0066] Preferably, the equipment is a handheld device.

[0067] A computer-readable medium storing at least one computer program, which is loaded and executed by a processor to enable an electronic device to perform a method of operating the device.

[0068] A fire emergency robot is equipped with the aforementioned multi-sensor fusion positioning and mapping and thermal imaging equipment.

[0069] A fire emergency robot dog is equipped with the aforementioned multi-sensor fusion positioning and mapping and thermal imaging equipment.

[0070] A fire-fighting emergency drone is equipped with the aforementioned multi-sensor fusion positioning and mapping and thermal imaging equipment.

[0071] The advantages of this invention are:

[0072] This invention proposes a multi-sensor fusion positioning, mapping, and thermal imaging equipment for fire emergency scenarios, along with its related computer-readable medium. The system integrates positioning and mapping sensors such as a thermal imaging camera, a visible light camera, lidar, and an inertial navigation system, specifically designed for high-risk and complex fire emergency environments. Through an edge-cloud integrated cluster collaborative system and multi-link stable transmission technology, combined with intelligent decision support, this invention can achieve high-precision real-time positioning and dynamic 3D temperature measurement and mapping in environments with weak or unavailable satellite navigation signals. The system's multi-mode portability and environmental adaptability design ensure the efficiency and safety of operators under extreme conditions, while its advanced data processing and analysis technologies significantly improve the accuracy and response speed of rescue operations. These innovative features give this invention significant practical application value and market prospects in the field of fire rescue. The specific advantages of this invention can be summarized as follows:

[0073] 1. End-to-cloud integrated cluster collaboration and multi-link stable transmission technology: Adopting end-to-cloud data processing and cluster collaboration technology, it supports multiple signal transmission methods such as 4G, WiFi, and LoRa, ensuring real-time transmission and efficient processing of information in extreme fire rescue environments, especially in situations with severe smoke and dust.

[0074] 2. Devices communicate with each other via LoRa and calculate the floor difference and 2D horizontal distance between them, which is especially suitable for situations where 4G / WiFi signals are poor.

[0075] 3. Multi-sensor fusion positioning and mapping technology: By using state estimation algorithms (filtering, optimization, depth end-to-end, etc.), sensors such as thermal imaging, visible light cameras, lidar and inertial navigation systems are deeply fused to achieve high-precision real-time positioning and mapping in environments with limited vision and weak or missing navigation satellite signals.

[0076] 4. AI-driven intelligent decision support: Introducing AI algorithms to optimize data processing and decision support, improving the system's response speed and decision accuracy in rapidly changing rescue environments.

[0077] 5. Multi-mode portability and environmental adaptability design: The equipment supports handheld and wearable modes, adapting to harsh environments such as high temperature, high humidity, and smoke, improving the flexibility and efficiency of operators.

[0078] 6. Efficient spatial information processing and real-time key point marking: The system can accurately detect and mark key locations such as entrances and exits and safety passages, and supports automatic floor judgment and identification, providing real-time spatial information and navigation support for rescue personnel.

[0079] 7. Temperature map construction through thermal imaging and multi-source data fusion: By combining 3D point cloud data obtained from thermal imaging and other sensors, 3D temperature contour mapping and comprehensive mapping technology are realized to provide detailed information on the distribution of fire sources and hotspots for rescue decision-making. Attached Figure Description

[0080] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0081] Figure 1 This is a schematic diagram of the device structure and algorithm of the present invention;

[0082] Figure 2 This is a comprehensive diagram of the device and cloud integration of the present invention. Detailed Implementation

[0083] This invention proposes a multi-sensor fusion positioning, mapping, and thermal imaging equipment for fire emergency scenarios, along with its computer-readable medium. The system integrates multiple sensing and mapping sensors, including thermal imaging, visible light cameras, lidar, and inertial navigation systems. Through edge-cloud integrated cluster collaboration and stable multi-link transmission technology, it achieves high-precision real-time positioning and 3D mapping in extreme environments, making it particularly suitable for complex fire emergency scenarios such as smoke, dust, and high temperatures. Combined with intelligent decision support, the system possesses rapid data processing and response capabilities, providing visualization of dynamic 3D scenes and temperature distribution information. Its multi-mode portability design supports handheld and wearable operation, ensuring operator flexibility and safety. The equipment features efficient spatial information processing and real-time key point marking capabilities, automatically identifying and marking entrances, floors, and other locations. Through multi-sensor fusion, it generates a 3D temperature map, clearly displaying the distribution of fire sources and hotspots, providing precise support for fire and rescue decision-making.

[0084] like Figure 1 As shown, one embodiment of the equipment of the present invention is as follows.

[0085] Thermal imaging cameras: acquire image information through thermal imaging in no-light or low-light environments, ensuring effective visual perception capabilities even under complex lighting conditions, and are especially suitable for scenarios such as fires or mine disasters.

[0086] Visible light camera module: Provides standard visual information, supplements the imaging needs of thermal imaging cameras under different lighting conditions, further enhances the environmental recognition capabilities of the device, and provides clear visual information support for 3D mapping.

[0087] LiDAR module: Provides high-precision 3D point cloud data to generate detailed 3D environment models. Its superior 3D reconstruction capabilities are suitable for narrow, complex underground passages and building interiors, providing comprehensive spatial information in various terrains and environments.

[0088] Inertial Navigation Systems (INS): Provide short-term, high-precision motion status information, including acceleration and angular velocity, enabling devices to maintain their position and orientation awareness even in the absence of external signals. For example, in environments without navigation satellite signals, such as mines or basements, INS can accurately identify the device's movement (up / down stairs, stationary, and horizontal motion), providing reliable data for path planning and sensor strategy optimization.

[0089] Satellite positioning and navigation module: Provides global navigation and positioning information in open-air scenarios, ensuring long-term global positioning capability, especially suitable for wide-area fire emergency scenarios. For indoor rescue, it supports algorithm absolute position initialization outdoors, realizing the correlation between indoor algorithm positioning results and absolute geographical location.

[0090] Power module: Provides long-term battery life, ensuring that the equipment can maintain continuous and stable operation during long-term high or low temperature emergency tasks, and adapt to extreme high and low temperature environments.

[0091] Communication module: Supports multiple data transmission methods such as 4G, LoRa, and WiFi, and supports relay base stations to enable real-time communication between the device and the command center or other devices, facilitating data sharing and command decision support in fire emergency scenarios.

[0092] The above modules specifically support the operation of the following modules.

[0093] I. Scene Awareness Module.

[0094] The scene perception module is responsible for real-time perception of dynamic and static environmental information in emergency scenarios, ensuring accurate acquisition of global and local information about the surrounding environment even under complex environmental conditions (such as low light, smoke, dust, and lack of navigation satellite signals). This module integrates the inertial navigation system's perception of the device's own motion, real-time tracking of dynamic targets, and the high and low temperature target detection capabilities of the thermal imaging camera, providing comprehensive perception of static and dynamic obstacles, living beings, fire sources, etc., and providing precise support for emergency response.

[0095] ① Self-Motion Perception: Utilizing an inertial navigation system combined with a self-motion perception model (preferably an AI model), this module accurately determines the device's own motion state, such as going up and down stairs, remaining stationary, walking on flat ground, or falling. The INS uses data from accelerometers and gyroscopes to accurately monitor the device's pose changes even in environments without navigation satellite signals. The self-motion perception model optimizes data processing, ensuring accurate identification of movement, falling, going up and down stairs / escalators / elevators, walking, and running in complex environments (such as mines or basements). This function helps the system dynamically adjust sensor strategies, providing reliable motion parameters and efficient decision support for path planning.

[0096] The scene perception module senses the device's own motion through an inertial navigation system combined with its own motion perception model. The inertial navigation system monitors the device's pose changes in real time using data from accelerometers and gyroscopes. This system combines inertial measurement unit (IMU) data and thermal imaging data, utilizing sensor fusion technology and Kalman filtering or particle filtering algorithms to optimize motion estimation, reduce the impact of sensor noise, and improve positioning accuracy. Specifically, the inertial navigation system collects acceleration information from the accelerometer and angular velocity data from the gyroscope, combining this with the sensor data fusion model to estimate the device's instantaneous velocity and displacement. To ensure the system can accurately distinguish between vertical and horizontal motion states, the motion perception model incorporates a motion state classification algorithm in data processing, employing a combination of threshold-based detection methods, machine learning algorithms, and deep learning algorithms. For example, accelerometer Z-axis data is used to monitor for significant vertical acceleration changes, thus determining whether it is a vertical motion state such as going up or down stairs, escalators, or elevators; while gyroscope data can determine whether it is a dynamic state such as walking or running horizontally. Furthermore, the device's own motion perception capability allows the system to dynamically adjust the sensor's operating mode and strategy. Depending on the device's motion state, the system can select an adaptive sampling rate, adjust the sensor's operating frequency, or switch between different operating modes (such as high-precision mode or low-power mode) to ensure efficient and accurate data acquisition. Simultaneously, through collaboration with the path planning module, the system can provide real-time motion parameters and decision support for path planning, thereby achieving superior positioning and navigation performance in complex environments. For different motion states, such as walking, running, climbing stairs, and falling, the model further optimizes path planning and system parameter adjustments through joint analysis of IMU data and thermal imaging data, ensuring high positioning accuracy and robustness even in dynamic and complex environments.

[0097] The self-motion perception model employs a deep neural network that fuses multi-sensor information. Specifically, the model uses a convolutional neural network (CNN) to extract heat source features from infrared images and combines this with a recurrent neural network (RNN) to process temporal sensor data provided by the inertial navigation module, tracking the device's trajectory and predicting its position changes. The RNN effectively infers the device's motion state by capturing the temporal dependence of sensor data, maintaining high stability even in the presence of noise or signal loss. To further enhance the model's robustness, we designed an adaptive weight allocation fusion judgment module. This module dynamically adjusts the weights based on the density of heat source distribution in the scene, thereby avoiding incorrect judgments in areas with scarce infrared features or significant interference, ensuring the accuracy of motion state estimation and thermal imaging results.

[0098] ② Dynamic Target Perception: This module processes multi-source data using a lightweight dynamic target recognition model, which preferably employs an AI model to achieve real-time perception of dynamic targets. By analyzing multi-frame information, the dynamic target recognition model accurately identifies and tracks moving personnel or equipment. It can remove noise points and extract key dynamic features in complex environments such as smoke and dust, thereby improving the system's accuracy in perceiving dynamic targets and providing reliable real-time dynamic information support for rescue operations.

[0099] Specifically, the lightweight dynamic target recognition model is based on a multi-layer convolutional neural network (CNN) for temporal feature extraction. In this model, the shallow feature extraction part has strong spatial information representation ability and a small receptive field, making it suitable for detecting small targets; while the deep feature extraction part has strong semantic information capture ability and a large receptive field, mainly used for recognizing larger targets. To improve the accuracy and robustness of dynamic target detection, an inter-branch differential network is introduced into the model. This network can perform dynamic target indexing in time, further filtering out misidentified targets and ensuring the accuracy of target recognition and tracking.

[0100] ③ High and Low Temperature Target Detection: Based on thermal imaging camera data, a temperature analysis model enables high-precision detection of thermal radiation sources in the environment, particularly suitable for low-light or no-light conditions. The preferred temperature analysis model is an AI model. In emergency scenarios such as fires or mine disasters, this module can identify areas of abnormal temperature, including fire sources and living beings (such as trapped personnel or animals). Deep analysis of infrared imaging data generates thermal distribution maps, helping rescuers quickly locate fire sources or potential rescue targets. This significantly improves the system's scene perception capabilities under invisible light conditions, providing crucial information support for emergency response.

[0101] ④ Automatic Identification and Marking of Entrances, Exits, and Floors: The corresponding sensor modules also include a lidar module and / or a barometer and / or a visible light camera module. This module combines data from multiple sensors to automatically identify and mark entrances, exits, and floor information in the environment through a depth model. The system analyzes 3D data from sensors such as lidar and visible light cameras to automatically generate a spatial model of the building and identify key locations such as entrances, exits, floors, and attack points. The floor identification and entrance / exit detection algorithm, combined with sensor data, can determine floor locations in real time, providing rescue personnel with accurate floor positioning and entrance / exit information. This helps in quickly planning escape routes and determining safe passages, providing crucial support for command decisions and path planning. The floor identification and entrance / exit detection algorithm preferably uses an AI algorithm. This module significantly improves the system's navigation and identification capabilities in complex environments, providing efficient spatial information support for emergency response and rescue missions.

[0102] The depth model automatically generates a spatial model of the building by analyzing data from thermal imaging cameras, and identifies the locations of entrances, floors, and attack points; the floor identification and entrance / exit detection algorithms, combined with inertial navigation system data, determine the floor location in real time.

[0103] The scene perception module uses a spatial convolutional neural network to extract local features from the data of the lidar module and / or visible light camera module, and further identifies the geometric shape of these structures; furthermore, a graph neural network is used to construct a spatial relationship graph to optimize the topological structure between spatial elements, thereby accurately identifying the relative positions of entrances and exits and stairs; the floor recognition and entrance detection algorithm combines barometer and inertial navigation data to calculate the absolute height of the current floor of the building through depth feature representation, and determines the floor position in real time.

[0104] The scene perception module employs a deep neural network representation. By analyzing 3D point cloud data provided by the LiDAR module, barometer data, and image data captured by the visible light camera module, it automatically generates a spatial model of the building and identifies the locations of key structures such as entrances and exits, and staircases. Specifically, a spatial convolutional neural network (CNN) is used to extract local features, such as doors, windows, and stairwells, from the LiDAR and visible light camera data, and further identifies the geometric shapes of these structures. Furthermore, a graph neural network (GNN) is used to construct a spatial relationship graph, optimizing the topological structure between spatial elements, thereby accurately identifying the relative positions of entrances and exits and staircases. The floor recognition and entrance / exit detection algorithm combines barometer and inertial navigation data, calculates the absolute height of the current floor of the building through deep feature representation, and determines the floor position in real time, ensuring accurate location of floor changes in a multi-story environment.

[0105] The scene perception module automatically identifies and marks entrances, floors, and attack points, and can also manually mark or modify them.

[0106] ⑤ Smoke and Dust Recognition: The corresponding sensor modules also include a LiDAR module and / or a visible light camera module; the scene perception module automatically identifies smoke and dust environments by analyzing data from the LiDAR and / or visible light camera modules. Specifically, the LiDAR module generates high-precision point cloud data of the surrounding environment by emitting lasers and receiving return signals. In smoke or dust environments, laser signals are scattered and attenuated, leading to a decrease in the density of the point cloud data or obvious abnormal distribution. To identify smoke and dust, the scene perception module uses data preprocessing techniques to determine the presence of smoke or dust by analyzing features such as the sparsity of the point cloud, changes in reflection intensity, and distance changes. Image data provided by the visible light camera module can also assist in identifying environmental features of smoke and dust. Through threshold-based detection methods, machine learning algorithms, or deep learning algorithms, the system can extract visual features of smoke or dust from images, such as blurred vision, floating particles, and halos in the image. By combining features such as image illumination changes, edge detection, and color differences, the system can determine the presence of smoke and dust in dynamic environments. Furthermore, based on historical data and environmental models, the scene perception module can gradually optimize its recognition accuracy, enabling the system to respond in real time and provide accurate environmental perception information in complex and dynamically changing environments.

[0107] II. Multi-sensor fusion positioning module.

[0108] The multi-sensor fusion localization module, based on state estimation methods such as filtering, optimization, and end-to-end integration, fuses data from multiple sensors (including thermal imaging cameras, visible light cameras, LiDAR, INS, and satellite positioning and navigation systems) to generate accurate device pose estimates. The core task of this module is to comprehensively process the sensing data acquired from different sensors to ensure that the system maintains accurate positioning and navigation capabilities even in extreme environments. Specific sub-modules are as follows:

[0109] ① Data cleaning module: First, the data from multiple sensors such as thermal imaging camera, LiDAR, INS and GNSS are preprocessed. Noise is filtered out by noise suppression model, and the data from multiple sensors are synchronized by sensor synchronization signals (PPS, etc.) to ensure that the observation data from different sensors are synchronized in time and to guarantee data quality.

[0110] ② Odometry positioning module: By fusing multi-source data from the thermal imaging camera module and the inertial navigation module through filtering, optimization or end-to-end state estimation methods, the device pose is recursively estimated to construct a multi-sensor fusion odometer; through the state estimation model, nonlinear motion in emergency scenarios is handled, thereby ensuring that the device can still provide high-precision real-time positioning when there is no satellite signal or the signal is interfered with.

[0111] In the presence of smoke or dust, the odometer positioning module filters the thermal image data acquired by the thermal imaging camera to smooth out stripe noise and further extracts key radiation features based on local radiation gradients. After feature extraction, feature association is performed based on local radiation feature descriptors to ensure robust matching of heat sources in the infrared image. Specifically, the local radiation feature descriptor encodes local heat source regions in the infrared image, similar to key point descriptors in ORB feature extraction, maintaining strong matching stability and robustness even under noise interference and heat source changes. This method improves the matching accuracy and reliability of thermal imaging data in motion, providing precise infrared positioning support for fusing data from the inertial navigation module.

[0112] In smoke-free and dust-free environments, the sensor module corresponding to the multi-sensor fusion positioning module also includes a lidar module. To balance the efficiency and accuracy of lidar matching, a classification feature matching method is adopted. When lidar data identifies surface features, a point-to-surface distance matching method is used; for non-surface features, a point-to-distribution matching method is used. This approach fully utilizes the low dimensionality of surface features, reducing computational load, while also enhancing the robustness of the algorithm by leveraging point-to-distribution constraints on non-surface features. Furthermore, to address potential degradation in the environment, an adaptive point cloud downsampling and matching parameter adjustment strategy is introduced to ensure matching accuracy and stability when point cloud density is low or feature distribution is uneven. In the event of final degradation, Fast-NDT (Fast Normal Distribution Transform) is used for point cloud registration to ensure that even under extreme conditions, the algorithm can still provide accurate matching and positioning results, guaranteeing the robustness and efficiency of the system in complex environments.

[0113] ③ Relocation and Global State Optimization: The corresponding sensor module also includes a satellite navigation module. The pose obtained by the INS and state estimation algorithms such as filtering, optimization, and end-to-end estimation is continuous, but positioning drift exists during long-term operation. To further improve positioning accuracy, the global position information and relocation constraints of the satellite navigation system are used to correct long-term pose errors, improving the overall system's positioning robustness and accuracy, as detailed below:

[0114] Absolute position observation fusion of satellite positioning and navigation systems (taking a single satellite as an example): In outdoor scenarios, the global position information provided by the satellite navigation system serves as an absolute pose reference, effectively reducing cumulative drift. Through a multimodal data fusion strategy based on Bayesian filtering, the satellite's absolute position data is used as a soft constraint for low-frequency updates, fused with high-frequency data such as lidar and inertial navigation data to gradually correct the pose. Furthermore, the system introduces an adaptive data reliability adjustment mechanism. When satellite signals are weak or obstructed, the weight of satellite data is automatically reduced, prioritizing the use of the fusion results from lidar and inertial navigation, ensuring the system's stability and robustness in complex environments such as indoors and underground.

[0115] Multi-sensor location re-identification: The system employs a deep re-localization algorithm to fuse multi-sensor data, overcoming the limitations of traditional re-localization algorithms in complex environments. The deep re-localization algorithm prioritizes AI algorithms. First, an AI-based re-localization feature description algorithm extracts global feature descriptors from LiDAR point clouds and constructs a historical point cloud database. Through rapid retrieval, candidate matching frames for the current point cloud in the historical database are identified. Then, multimodal matching between infrared and visible light images is introduced, and an AI similarity model is used to calculate the matching similarity between the current image and candidate images, optimizing the candidate frame ranking. In smoke or low-light environments, the system automatically determines the scene state through a scene perception module, prioritizing the use of infrared image data to ensure the robustness of the re-localization algorithm. After determining the re-localization constraint markers, a deep algorithm is used for point cloud registration, generating a high-precision re-localization relative pose, which serves as input for global pose graph optimization.

[0116] Global pose map optimization: By combining multi-sensor fusion state estimation methods, repositioning relative poses generated by self-developed AI algorithms, and absolute position observations provided by satellite navigation systems, a global pose map is constructed and optimized. By solving the optimization problem, the system obtains the optimal estimate of the pose at all historical moments, updates the historical frame sub-maps and the global map, eliminates long-term accumulated errors, and achieves globally consistent high-precision mapping. This optimization method ensures that the system possesses stable long-term positioning and mapping capabilities in complex environments.

[0117] ④ Positioning Feedback and Navigation Guidance: Integrating positioning data and environmental perception information, this module provides real-time navigation guidance for rescue equipment. It also supports relative positioning between multiple devices, serving battlefield collaboration and missing persons rescue scenarios. Through collaborative computing and communication between devices, combined with decision-making algorithms, the system achieves accurate relative pose determination and path optimization, building an efficient multi-device collaborative network. The decision-making algorithm prioritizes AI algorithms. The AI ​​decision-making algorithm can analyze multi-sensor data in real time, assessing environmental complexity, dynamic obstacle distribution, and equipment operating status, providing intelligent support for navigation guidance. For different scenarios, the AI ​​algorithm dynamically adjusts navigation strategies. For example, in scenarios with dense obstacles or dense smoke, it prioritizes obstacle avoidance paths; in life detection missions, it optimizes equipment collaboration paths to quickly reach the target location. Furthermore, positioning information is fed back to the control center in real time via the communication module. The AI ​​algorithm comprehensively analyzes global information, generating real-time scheduling suggestions and optimizing task allocation and collaborative paths between multiple devices. This function not only improves rescue efficiency but also enhances the system's responsiveness in complex and dynamic environments, providing accurate and reliable navigation and scheduling support for emergency missions.

[0118] III. Multi-machine, multi-source data fusion mapping module.

[0119] The multi-machine, multi-source data fusion mapping module is responsible for processing and fusing various sensor data from single and multiple machines to create maps. This primarily includes two parts: building a local 2D temperature occupancy map and constructing a server-side, multi-machine, global, 3D detailed environmental map. Through deep fusion and real-time dynamic updates of multi-source data, this module provides emergency response personnel with accurate and comprehensive environmental awareness, helping them to grasp the on-site layout and dynamic changes in real time, and providing strong support for decision-making, route planning, and task scheduling.

[0120] ① Standalone Data Fusion and Local 2D Temperature Occupancy Map Construction: On the standalone device, local point clouds generated by fusing thermal imaging data captured by thermal imaging cameras and sensor data from inertial navigation systems, etc., are efficiently constructed using lightweight algorithms, with AI algorithms being the preferred choice. This map represents the spatial structure and temperature distribution information around the equipment in a rasterized form, intuitively displaying fire sources, living beings, and high-temperature hazard areas. The lightweight AI algorithm intelligently filters noise and optimizes the raster update rate by analyzing infrared thermal images and point cloud data in real time, ensuring the accuracy and real-time performance of map construction. In dynamic environments, the AI ​​model can automatically adjust perception weights according to environmental changes, such as prioritizing thermal image data processing under smoke or low-light conditions, making the generated occupation map more consistent with the current scenario requirements. This module not only improves the efficiency of occupation map construction but also significantly reduces computational resource consumption, ensuring that the equipment can continuously generate efficient and reliable local environmental information in complex emergency scenarios, providing strong support for local navigation, obstacle avoidance, and path planning, and effectively improving the operational efficiency and safety of frontline personnel.

[0121] In the absence of smoke and dust, the sensor module corresponding to the data fusion mapping module also includes a lidar module. It integrates thermal imaging data captured by the thermal imaging camera with local point clouds generated by the inertial navigation system and lidar module data, and uses an algorithm model to construct a two-dimensional temperature occupancy map.

[0122] ② The server-side multi-machine global 3D environmental temperature map construction: The server integrates sensor data from multiple devices and utilizes scene re-identification and inter-machine re-localization technologies to construct a globally consistent 3D fused environmental map. This module can dynamically update static structures and dynamic targets in the 3D map, achieving fine modeling of the global environment. The scene re-identification algorithm preferentially uses AI algorithms, analyzing multi-sensor data such as infrared thermal images from multiple devices to identify and match similar scene areas, achieving alignment of observation data between different devices. The inter-machine re-localization detection technology combines global feature descriptors generated by deep learning to quickly retrieve similar scenes in the historical point cloud database. Through further verification with infrared images, the re-localization matching frame is determined. For complex scenes (such as smoke or dark environments), the system can dynamically adjust the perception strategy according to the scene state to ensure algorithm robustness. The server uses the fused re-localization constraint information to globally optimize the local maps of each device, eliminating drift accumulation errors through pose adjustment, and generating a highly consistent global 3D map with wide coverage. Real-time dynamic update technology can accurately reflect changes in the scene, including the location of new obstacles, dynamic targets, and fire source spread. This module provides the command center with a globally consistent 3D environmental perception capability, provides complete environmental model support for multi-device collaborative rescue missions, optimizes task allocation and path planning, realizes global multi-machine collaboration and improves emergency response efficiency, and provides strong rear support for combat personnel.

[0123] The sensor modules corresponding to the server-side multi-machine global 3D environmental temperature mapping and dynamic update also include a LiDAR module and a visible light camera module. In the specific steps, the scene re-identification algorithm analyzes multi-sensor data from multiple devices, including LiDAR point clouds, infrared thermal images, and visible light images, to identify and match similar scene areas, thereby achieving alignment of observation data between different devices. The multi-machine relocation detection technology, combined with global feature descriptors generated by deep learning, searches for similar scenes in the historical point cloud database and determines the relocation matching frame through further verification using visible light and infrared images.

[0124] ③ Precise Segmentation of Remote 3D Map: The server-side global 3D environment map is precisely segmented using a segmentation algorithm. The preferred segmentation algorithm is an AI segmentation algorithm, accurately dividing objects in the scene into key elements such as entrances / exits, walls, ground, obstacles, and living organisms. The deep learning-based segmentation model utilizes fused multi-source data (including LiDAR point clouds, infrared thermal imaging data, and visible light images) combined with global scene features to achieve high-precision semantic segmentation and object recognition. The precise segmentation results help commanders quickly grasp the location and status of key areas in the scene, identify safe paths, passable areas, and potential rescue targets, providing refined environmental perception support for rescue missions. Through this algorithm, the system can dynamically update the segmentation results, promptly reflecting scene changes and providing more intelligent guidance for equipment navigation, path planning, and mutual rescue.

[0125] ④ 3D Temperature Map Rendering: The remote 3D environment map is further combined with thermal imaging data from thermal imaging cameras to achieve 3D rendering of the temperature map through a temperature mapping algorithm. The preferred temperature mapping algorithm is AI-driven. This function overlays real-time temperature information onto a 3D point cloud, intuitively displaying the temperature distribution in different areas of the scene, including the specific locations and temperature change trends of attack points, fire sources, heat sources, and living beings. 3D temperature rendering helps commanders quickly identify high-temperature areas and potential directions of fire spread, providing real-time reference for dynamic adjustments to rescue missions and the delineation of safe zones. Simultaneously, this function provides combat personnel with three-dimensional visualization support for heat sources, improving emergency response efficiency and the accuracy of rescue decisions in complex environments.

[0126] like Figure 2 As shown, the equipment of this invention consists of one or more units, with multiple units communicating in real time with the control center and server via relay base stations. The communication employs end-to-cloud integrated cluster collaboration and multi-link stable transmission technology: end-to-cloud data processing and cluster collaboration technology are used. Location information is fed back to the control center in real time via the communication module, enabling comprehensive analysis of global information, generating real-time scheduling suggestions, and optimizing task allocation and collaboration paths among multiple devices.

[0127] The devices in this invention support multiple signal transmission methods such as 4G, WiFi, and LoRa, ensuring real-time transmission and efficient processing of information in extreme fire rescue environments. Especially in situations with severe smoke and dust, and when 4G and WiFi signals are weak, multiple devices can communicate with each other via LoRa and calculate the floor differences and 2D horizontal distances between them.

[0128] The real-time dispatch suggestions generated by the control center also include dispatching personnel from other equipment to rescue people who are trapped or in distress on a certain device.

[0129] The multi-sensor fusion positioning and mapping and thermal imaging equipment described in this invention is a wearable device.

[0130] The multi-sensor fusion positioning and mapping and thermal imaging equipment described in this invention is a handheld device.

[0131] The present invention also proposes a computer-readable medium storing at least one computer program, which is loaded and executed by a processor to enable an electronic device to perform the operation method of the above-described equipment.

[0132] The present invention also proposes a fire emergency robot equipped with the aforementioned multi-sensor fusion positioning and mapping and thermal imaging equipment.

[0133] The present invention also proposes a fire emergency robot dog equipped with the aforementioned multi-sensor fusion positioning and mapping and thermal imaging equipment.

[0134] The present invention also proposes a fire emergency drone equipped with the aforementioned multi-sensor fusion positioning and mapping and thermal imaging equipment.

[0135] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All modifications made according to the spirit and essence of the main technical solution of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A multi-sensor fusion positioning and mapping and thermal imaging equipment, characterized in that, include: The sensor module includes at least a thermal imaging camera and an inertial navigation system; The data fusion and mapping module is responsible for processing data from multiple sensors to fuse and map data, including building a local two-dimensional temperature occupancy map. The data fusion and mapping module specifically includes the following tasks: Single-device data fusion and local 2D temperature occupancy map construction: On a single device, thermal imaging data captured by a thermal imaging camera and data from an inertial navigation system are fused to generate a local point cloud. An algorithm model is then used to construct a 2D temperature occupancy map. The 2D temperature occupancy map represents the spatial structure and temperature distribution information around the device in a rasterized form, displaying fire sources, living organisms, and high-temperature hazard areas. The algorithm model intelligently filters noise and optimizes the raster update rate by analyzing infrared thermal images and point cloud data in real time, ensuring the accuracy and real-time performance of the map construction. In dynamic environments, the algorithm model automatically adjusts the perception weights according to environmental changes.

2. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 1, characterized in that, The data fusion mapping module also includes the construction of a multi-machine global 3D environmental temperature map on the server side.

3. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 1, characterized in that, Also includes: The scene perception module senses dynamic and static environmental information in emergency scenarios in real time, including sensing the movement of equipment itself, real-time tracking of dynamic targets, and detection of high and low temperature targets.

4. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 1, characterized in that, Also includes: The multi-sensor fusion positioning module integrates data from multiple sensors to generate device pose estimation.

5. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 1, characterized in that, The equipment also includes a communication module to enable real-time communication between the equipment and the control center; the positioning information is fed back to the control center in real time through the communication module, and the global information is comprehensively analyzed to generate real-time scheduling suggestions and optimize the task allocation and collaboration path between multiple devices.

6. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 1, characterized in that, The equipment adopts end-to-cloud integrated cluster collaboration and multi-link stable transmission technology: it uses end-to-cloud data processing and cluster collaboration technology, and supports multiple signal transmission methods and relay communication modes between end and cloud.

7. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 3, characterized in that, The scene perception module perceives the device's own motion through an inertial navigation system combined with its own motion perception model: the inertial navigation system monitors the device's pose changes using data from accelerometers and gyroscopes; based on a thermal imaging camera and the inertial navigation system, the self-motion perception model optimizes data processing to ensure accurate identification of movement, falls, going up and down stairs / escalators / elevators, walking, and running; the perception of the device's own motion helps the system dynamically adjust sensor strategies, providing motion parameters and decision support for path planning.

8. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 3, characterized in that, The inertial navigation system combines its own inertial measurement unit (IMU) data and thermal imaging data, utilizing sensor fusion technology and Kalman filtering or particle filtering algorithms to optimize motion estimation, reduce the impact of sensor noise, and improve positioning accuracy. Specifically, the IMU collects acceleration information from the device via accelerometers and angular velocity data via gyroscopes, combining this with a sensor data fusion model to estimate the device's instantaneous velocity and displacement. To ensure the system can accurately distinguish between vertical and horizontal motion states, its motion perception model incorporates a motion state classification algorithm in data processing, employing a combination of threshold-based detection methods, machine learning algorithms, and deep learning algorithms. Accelerometer Z-axis data is used to monitor for significant changes in vertical acceleration, thereby determining whether the motion is vertical, such as going up or down stairs, escalators, or elevators. Gyroscope data is used to determine whether the motion is horizontal, such as walking or running.

9. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 8, characterized in that, The scene perception module's perception of the device's own motion enables the inertial navigation system to dynamically adjust the sensor's working mode and strategy. Based on the device's motion state, the inertial navigation system selects an adaptive sampling rate, adjusts the sensor's working frequency, or switches between high-precision mode and low-power mode to ensure the efficiency and accuracy of data acquisition. By working in conjunction with the path planning module, the inertial navigation system provides real-time motion parameters and decision support for path planning, thereby achieving better positioning and navigation performance in complex environments.

10. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 9, characterized in that, For different motion states, the self-motion perception model further optimizes path planning and system parameter adjustment through joint analysis of inertial measurement unit data and thermal imaging data, ensuring that it can maintain high positioning accuracy and robustness in dynamic and complex environments.

11. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 7, characterized in that, The self-motion perception model employs a deep neural network that fuses information from multiple sensors; The self-motion perception model uses a convolutional neural network to extract heat source features from infrared images and combines them with a recurrent neural network to process time-series sensor data provided by the inertial navigation system, track the device's motion trajectory and predict its position changes; the convolutional neural network infers the device's motion state by capturing the time dependence of sensor data, and can maintain stability even in the presence of noise or signal loss.

12. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 11, characterized in that, The self-motion perception model also incorporates an adaptive weight allocation fusion judgment module. This module dynamically adjusts the weights based on the density of heat source distribution in the scene, thereby avoiding incorrect judgments in areas with scarce infrared features or significant interference, and ensuring the accuracy of motion state estimation and thermal imaging results.

13. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 3, characterized in that, The scene perception module tracks dynamic targets in real time and processes thermal imaging camera data through a lightweight dynamic target recognition model to achieve real-time perception of dynamic targets. The lightweight dynamic target recognition model analyzes multi-frame information, tracks moving people or equipment, removes noise points, and extracts key dynamic features, thereby improving the system's perception accuracy of dynamic targets and providing real-time dynamic information support.

14. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 13, characterized in that, The lightweight dynamic target recognition model is based on multi-layer convolutional neural networks for temporal feature extraction; The lightweight dynamic target recognition model includes a shallow feature extraction part and a deep feature extraction part; The targets are divided into small targets and large targets. The shallow feature extraction part is used to detect small targets, while the deep feature extraction part is used to identify large targets. An interactive branch differential network is introduced into the lightweight dynamic target recognition model to perform dynamic target indexing in time series, further filtering out misidentified targets and ensuring the accuracy of target recognition and tracking.

15. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 3, characterized in that, The scene perception module detects targets at high and low temperatures. Based on thermal imaging camera data, it uses a temperature analysis model to detect heat radiation sources in the environment and identify areas with abnormal temperatures. It also performs in-depth analysis of infrared imaging data to generate a thermal distribution map, thereby locating fire sources or potential rescue targets.

16. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 3, characterized in that, The scene perception module also automatically identifies and marks entrances and floors; the scene perception module automatically identifies and marks entrances, floors and attack points in the environment through a depth model; The depth model automatically generates a spatial model of a building by analyzing data from thermal imaging cameras, and identifies the locations of entrances, floors, and attack points; the floor identification and entrance / exit detection algorithms, combined with inertial navigation system data, determine the floor location in real time.

17. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 16, characterized in that, The scene perception module automatically identifies and marks entrances and floors, and the corresponding sensor module also includes a lidar module and / or a barometer and / or a visible light camera module.

18. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 17, characterized in that, The scene perception module employs a deep neural network representation. By analyzing 3D point cloud data provided by the LiDAR module and / or barometer data and / or image data captured by the visible light camera module, it automatically generates a spatial model of the building and identifies entrances, floors, and attack points. The scene perception module uses a spatial convolutional neural network to extract local features from the LiDAR and / or visible light camera module data and further identifies the geometric shapes of these structures. Furthermore, a graph neural network is used to construct a spatial relationship graph, optimizing the topological structure between spatial elements to accurately identify the relative positions of entrances and stairs. The floor recognition and entrance detection algorithm combines barometer and inertial navigation data to calculate the absolute height of the current floor of the building through depth feature representation and determine the floor position in real time.

19. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 16, characterized in that, The scene perception module automatically identifies and marks entrances, floors, and attack points, and can also manually mark or modify them.

20. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 3, characterized in that, The scene perception module also identifies smoke and dust. The corresponding sensor modules include a lidar module and / or a visible light camera module. The scene perception module automatically identifies smoke and dust environments by analyzing data from the lidar module and / or the visible light camera module. The lidar module generates high-precision point cloud data of the surrounding environment by emitting lasers and receiving return signals. The scene perception module uses data preprocessing technology to determine the presence of smoke or dust by analyzing the sparsity, reflection intensity changes, and distance changes of the point cloud.

21. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 20, characterized in that, In the smoke and dust recognition of the scene perception module, the corresponding sensor module also includes a visible light camera module; the image data provided by the visible light camera module assists in the recognition of environmental features of smoke and dust; through threshold-based detection methods, machine learning algorithms or deep learning algorithms, the scene perception module extracts the visual features of smoke or dust from the image, and by combining the image's illumination changes, edge detection and color difference features, it determines the presence of smoke and dust in the dynamic environment.

22. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 20, characterized in that, In the smoke and dust recognition of the scene perception module, based on historical data and environmental models, the scene perception module gradually optimizes its recognition accuracy, and provides accurate environmental perception information in real time in complex and dynamically changing environments.

23. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 4, characterized in that, The multi-sensor fusion positioning module further includes a lidar module, and the multi-sensor fusion positioning module includes specific sub-modules: Data cleaning module: First, the data from the thermal imaging camera, inertial navigation module and lidar module are preprocessed. Noise is filtered out by a noise suppression model. Then, the data from multiple sensors are synchronized by a sensor synchronization signal to ensure that the observation data from different sensors are synchronized in time. Odometry positioning module: By fusing multi-source data from thermal imaging camera, inertial navigation module and lidar module through filtering, optimization or end-to-end state estimation methods, the pose of the device is recursively estimated to construct a multi-sensor fusion odometry; By using a state estimation model, nonlinear motion in emergency scenarios is handled, thereby ensuring that the equipment can still provide high-precision real-time positioning even when there is no satellite signal or the signal is interfered with.

24. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 23, characterized in that, The odometer positioning module filters the thermal image data acquired by the thermal imaging camera to smooth stripe noise and further extracts key radiation features based on local radiation gradients. After feature extraction, feature association is performed based on local radiation feature descriptors to ensure robust matching of heat sources in the infrared image. The local radiation feature descriptors are obtained by encoding local heat source regions in the infrared image.

25. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 23, characterized in that, The sensor module corresponding to the multi-sensor fusion positioning module also includes a lidar module. To balance the efficiency and accuracy of lidar matching, a classification feature matching method is adopted. When surface features are identified in the LiDAR data, a point-to-surface distance matching method is used; while for non-surface features, a point-to-distribution matching method is used. The low dimensionality of surface features reduces the amount of computation, and the point-to-distribution constraint can be used to enhance the robustness of the algorithm on non-surface features.

26. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 23, characterized in that, To address potential degradation in the environment, an adaptive point cloud downsampling and matching parameter adjustment strategy is introduced to ensure the accuracy and stability of matching when the point cloud density is low or the feature distribution is uneven. In the event of eventual degradation, a fast normal distribution transformation is used for point cloud registration to ensure that the algorithm can still provide accurate matching and localization results even under extreme conditions.

27. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 23, characterized in that, The sensor module also includes a satellite navigation module, and the corresponding multi-sensor fusion positioning module includes sub-modules: Relocation and Global State Optimization Module: Utilizes global position information and relocation constraints from the satellite navigation module to correct long-term pose errors, thereby improving the overall system's positioning robustness and accuracy.

28. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 23, characterized in that, The multi-sensor fusion positioning module also includes a sub-module: Positioning feedback and navigation guidance module: It integrates positioning data and environmental perception information to provide real-time navigation guidance for rescue equipment, and also supports relative positioning between multiple devices.

29. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 27, characterized in that, The relocation and global state optimization module corrects long-term pose errors, specifically including: Absolute position observation fusion of satellite positioning and navigation modules: In outdoor scenarios, the global position information provided by the satellite positioning and navigation module serves as an absolute pose reference, reducing cumulative drift. A multi-modal data fusion strategy is employed, using the satellite's absolute position data as a low-frequency update soft constraint, which is then fused with high-frequency data from the lidar module, thermal imaging camera, and inertial navigation to gradually correct the pose. Furthermore, an adaptive data reliability adjustment mechanism is introduced, automatically reducing the weight of satellite data when the satellite signal is weak or obstructed, and using the fusion results from lidar, thermal imaging camera, and inertial navigation. Multi-sensor location re-identification: The corresponding sensor module also includes a visible light camera module; a depth re-localization algorithm is used to fuse multi-sensor data to overcome the limitations of traditional re-localization algorithms in complex environments; firstly, a re-localization feature description algorithm is used to extract global feature descriptors from LiDAR point clouds and construct a historical point cloud database; candidate matching frames of the current point cloud in the historical database are retrieved and determined; then, multimodal matching of infrared and / or visible light images is introduced, and a similarity model is used to calculate the matching similarity between the current image and candidate images to optimize the candidate frame ranking; in smoke or low-light environments, the scene perception module automatically judges the scene state and prioritizes the use of infrared image data to ensure the robustness of the re-localization algorithm; after determining the re-localization constraint markers, point cloud registration is performed to generate the re-localization relative pose, which serves as the input for global pose graph optimization; Global pose map optimization: By combining the state estimation method of multi-sensor fusion, the generated repositioning relative pose, and the absolute position observation provided by the satellite navigation system, a global pose map is constructed and optimized; by solving the optimization problem, the optimal estimate of the pose at all historical moments is obtained, the historical frame sub-map and global map are updated, long-term accumulated errors are eliminated, and a globally consistent high-precision map is achieved.

30. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 28, characterized in that, The positioning feedback and navigation guidance module integrates positioning data and environmental perception information to provide real-time navigation guidance for rescue equipment, while also supporting relative positioning between multiple devices, specifically including: Through collaborative computing and communication between devices, combined with decision-making algorithms, accurate relative pose determination and path optimization are achieved, building an efficient multi-device collaborative network. The decision-making algorithm analyzes multi-sensor data in real time, assesses environmental complexity, dynamic obstacle distribution, and device operating status, providing intelligent support for navigation guidance. For different scenarios, the decision-making algorithm dynamically adjusts navigation strategies, prioritizing obstacle avoidance paths in scenarios with dense obstacles or dense smoke. In life detection missions, it optimizes collaborative paths between devices to quickly reach the target location.

31. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 5, characterized in that, Multiple devices can communicate with each other via LoRa, 4G, or WiFi; in cases where 4G and WiFi signals are weak, they can communicate with each other via LoRa and calculate the floor difference and 2D horizontal distance between them.

32. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 5, characterized in that, The real-time dispatch suggestions generated by the control center also include dispatching personnel from other equipment to rescue people who are trapped or in distress on a certain device.

33. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 5, characterized in that, The equipment consists of one or more units, and each unit communicates in real time with the control center and server via a relay base station.

34. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 1, characterized in that, The sensor module corresponding to the data fusion mapping module also includes a lidar module. Under smoke-free and dust-free conditions, it fuses thermal imaging data captured by a thermal imaging camera with local point clouds generated by inertial navigation system and lidar module data, and uses an algorithm model to construct a two-dimensional temperature occupancy map.

35. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 2, characterized in that, The construction of the multi-machine global three-dimensional environmental temperature map on the server side specifically includes: Server-side multi-machine global 3D environmental temperature mapping and dynamic updating: The server integrates sensor data from multiple devices and utilizes scene re-identification and inter-device relocalization technologies to construct a globally consistent 3D fused environment map. It dynamically updates static structures and dynamic targets within the 3D map, achieving refined modeling of the global environment. The scene re-identification algorithm analyzes infrared thermal image data from multiple devices to identify and match similar scene areas, aligning observation data between different devices. Inter-device relocalization detection technology, combined with global feature descriptors generated by deep learning, retrieves similar scenes from a historical point cloud database. Further verification using infrared images determines the relocalization matching frame. The server utilizes the fused relocalization constraint information to globally optimize the local maps of each device, eliminating drift accumulation errors through pose adjustment to generate a global 3D map.

36. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 35, characterized in that, The sensor modules corresponding to the server-side multi-machine global 3D environmental temperature mapping and dynamic update also include a LiDAR module and a visible light camera module. In the specific steps, the scene re-identification algorithm analyzes multi-sensor data from multiple devices, including LiDAR point clouds, infrared thermal images, and visible light images, to identify and match similar scene areas, thereby achieving alignment of observation data between different devices. The multi-machine relocation detection technology, combined with global feature descriptors generated by deep learning, searches for similar scenes in the historical point cloud database and determines the relocation matching frame through further verification using visible light and infrared images.

37. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 35, characterized in that, The construction of the multi-machine global three-dimensional environmental temperature map on the server side also includes the following specific tasks: Accurate segmentation of remote 3D map: The server-side global 3D environment map is accurately segmented using a segmentation algorithm, precisely classifying objects in the scene into at least one of the following elements: entrances / exits, walls, ground, obstacles, and living organisms; based on the segmentation model, it utilizes fused multi-source data and combines global scene features to achieve semantic segmentation and object recognition.

38. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 35, characterized in that, The construction of the multi-machine global three-dimensional environmental temperature map on the server side also includes the following specific tasks: 3D rendering of temperature map: The remote 3D environment map is further combined with thermal imaging data from thermal imaging cameras to achieve 3D rendering of temperature map through temperature mapping algorithm; the real-time temperature information is superimposed on the 3D point cloud to intuitively display the temperature distribution in different areas of the scene, including the specific location and temperature change trend of at least one of the attack point, fire source, heat source and living organism.

39. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to any one of claims 1-38, characterized in that, The equipment in question is wearable equipment.

40. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to any one of claims 1-38, characterized in that, The equipment in question is a handheld device.

41. A multi-sensor fusion positioning and mapping and thermal imaging equipment, characterized in that, include: The sensor module includes at least a thermal imaging camera and an inertial navigation system; The data fusion and mapping module is responsible for processing data from multiple sensors to fuse and map data, including the construction of a multi-machine global 3D environmental temperature map on the server side. The construction of the multi-machine global three-dimensional environmental temperature map on the server side specifically includes: Server-side multi-machine global 3D environmental temperature mapping and dynamic updating: The server integrates sensor data from multiple devices and utilizes scene re-identification and inter-device relocalization technologies to construct a globally consistent 3D fused environment map. It dynamically updates static structures and dynamic targets within the 3D map, achieving refined modeling of the global environment. The scene re-identification algorithm analyzes infrared thermal image data from multiple devices to identify and match similar scene areas, aligning observation data between different devices. Inter-device relocalization detection technology, combined with global feature descriptors generated by deep learning, retrieves similar scenes from a historical point cloud database. Further verification using infrared images determines the relocalization matching frame. The server utilizes the fused relocalization constraint information to globally optimize the local maps of each device, eliminating drift accumulation errors through pose adjustment to generate a global 3D map.

42. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 41, characterized in that, The equipment in question is wearable equipment.

43. The multi-sensor fusion positioning and mapping and thermal imaging equipment according to claim 41, characterized in that, The equipment in question is a handheld device.

44. A computer-readable medium, characterized in that, The computer-readable medium stores at least one computer program, which is loaded and executed by a processor to enable the electronic device to perform the method of operation of the apparatus as described in any one of claims 1-38 and 41.

45. A fire emergency robot, characterized in that, The device is equipped with the multi-sensor fusion positioning and mapping and thermal imaging equipment as described in any one of claims 1-38 and 41.

46. ​​A fire emergency robot dog, characterized in that, The device is equipped with the multi-sensor fusion positioning and mapping and thermal imaging equipment as described in any one of claims 1-38 and 41.

47. A fire-fighting emergency drone, characterized in that, The device is equipped with the multi-sensor fusion positioning and mapping and thermal imaging equipment as described in any one of claims 1-38 and 41.

Images