Multi-sensor fusion positioning mapping and thermal imaging equipment and computer readable medium

Abstract

The invention discloses a multi-sensor fusion positioning mapping and thermal imaging device and a computer readable medium. The equipment integrates positioning mapping sensors such as a thermal imaging camera, a visible light camera, a laser radar and an inertial navigation system, and is specially designed for high-risk and complex fire emergency environments. Through an end-cloud integrated cluster cooperation system and a multi-link stable transmission technology and in combination with intelligent decision support, high-precision real-time positioning and dynamic 3D temperature measurement mapping can be realized in an environment in which satellite navigation signals are weak or unavailable. The multi-mode portability and environmental adaptability design of the system ensures the high efficiency and safety of an operator / unmanned system under extreme conditions, and the advanced data processing and analysis technology of the system significantly improves the accuracy and response speed of rescue operation. Due to the innovative characteristics, the method has remarkable practical application value and market prospect in the field of fire-fighting emergency rescue.

Description

Technical Field

[0001] The present invention relates to the field of mapping and positioning, and in particular to a multi-sensor fusion positioning mapping and thermal imaging equipment and a computer-readable medium. Background Art

[0002] In emergency scenarios, such as fire rescue, mining disasters, underground facility search and rescue, etc., complex environmental conditions pose great challenges to positioning, mapping and thermal imaging functions. These scenarios usually include low-light or no-light conditions, smoke, extreme temperature changes, narrow passages and multiple obstacles, and cannot rely on satellite navigation signals. Existing single sensor systems usually cannot provide sufficient accuracy, robustness and real-time performance in such extreme environments. Therefore, there is an urgent need for a multi-sensor fusion system that combines the advantages of various sensors to provide comprehensive environmental perception, high-precision positioning and thermal imaging information support.

[0003] Thermal imaging cameras have significant advantages in lightless or low-light environments, and can generate clear thermal images by capturing the thermal radiation information of objects. In fire emergency rescue scenarios, thermal imaging cameras can penetrate smoke, detect high-temperature targets, identify fire sources and living things (such as trapped people), and provide key support for rescue. In environments with insufficient visible light or no light at all, combined with the data from thermal imaging cameras, the equipment can still perform effective environmental perception and generate temperature distribution maps, providing more dimensional information support for decision-making. Laser radar (LiDAR) is an important tool for three-dimensional environmental modeling due to its high-precision three-dimensional point cloud data. Its data can be used for accurate environmental reconstruction, especially in complex terrain and narrow spaces. The fusion of LiDAR data and infrared thermal images further enhances the system's adaptability in dynamic environments, providing accurate positioning and real-time mapping capabilities for rescue equipment.

[0004] The inertial navigation system can achieve short-term accurate position and attitude estimation by measuring acceleration and angular velocity without external signals or references, which is particularly suitable for underground scenes or closed spaces. In a short period of time, inertial navigation can provide the system with high-precision position and attitude information to ensure that the equipment can continue to locate and navigate in extreme environments. In addition, the global satellite navigation system (GNSS, such as GPS, single Beidou) provides the system with global positioning functions, especially in open-air environments or before entering indoor / underground facilities, it can provide reliable long-term positioning support for equipment. By integrating the global positioning data of the GNSS system with other sensors (inertial navigation, magnetometer, barometer, etc.), the system can achieve continuous navigation and mapping in a wide area, and use multi-sensor perception information to efficiently integrate absolute and relative positions. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-sensor fusion positioning mapping and thermal imaging equipment and a computer-readable medium, which forms a system with real-time positioning, accurate mapping and temperature detection functions by integrating multi-sensor data such as thermal imaging cameras, laser radars, visible light cameras, inertial navigation and satellite systems, and is particularly suitable for complex fire emergency scenarios. The advantages of each sensor complement each other, providing robust perception capabilities under different environmental conditions, while generating intuitive fire source, life form and temperature distribution information through thermal imaging functions, providing comprehensive support for emergency rescue, thereby improving the efficiency and safety of emergency response.

[0006] The technical solution of the present invention is: Multi-sensor fusion positioning, mapping and thermal imaging equipment, including: A sensor module, including at least a thermal imaging camera and an inertial navigation system; The data fusion mapping module is responsible for processing multiple sensor data for fusion mapping, including the construction of local two-dimensional temperature occupancy maps.

[0007] Preferably, the fusion mapping of the data fusion mapping module also includes the construction of a multi-machine global three-dimensional environmental temperature map on the server side.

[0008] Preferably, it also includes: The scene perception module perceives the dynamic and static environmental information in the emergency scene in real time, including the perception of the device's own movement, real-time tracking of dynamic targets, and detection of high and low temperature targets.

[0009] Preferably, it also includes: The multi-sensor fusion positioning module fuses data from multiple sensors to generate device pose estimation.

[0010] Preferably, the scene perception module perceives the device's own motion through an inertial navigation system combined with a self-motion perception model: the inertial navigation system monitors the device's posture changes through data from an accelerometer and a gyroscope; based on a thermal imaging camera and an inertial navigation system, the self-motion perception model optimizes data processing to ensure accurate identification of movement, stillness, falls, going up and down stairs / escalators / elevators, walking, and running; the perception of the device's own motion helps the system dynamically adjust the sensor strategy to provide motion parameters and decision support for path planning.

[0011] Preferably, the inertial navigation system combines its own inertial measurement unit data and thermal imaging data, utilizes sensor fusion technology, optimizes motion estimation through Kalman filtering or particle filtering algorithm, reduces the impact of sensor noise, and improves positioning accuracy; specifically, the inertial navigation system collects the acceleration information of the device through an accelerometer, obtains angular velocity data through a gyroscope, and estimates the instantaneous speed and displacement of the device in combination with the sensor data fusion model; in order to ensure that the system can accurately distinguish between vertical and horizontal motion states, the self-motion perception model introduces a motion state classification algorithm in data processing, and adopts a combination of a threshold-based detection method, a machine learning algorithm, and a deep learning algorithm; the Z-axis data of the accelerometer is used to monitor whether there is an obvious change in vertical acceleration, so as to determine whether it is a vertical motion state of going up and down stairs, escalators or elevators; and the data based on the gyroscope is used to determine whether it is a horizontal walking or running dynamic state.

[0012] Preferably, the scene perception module's perception of the device's own movement enables the inertial navigation system to dynamically adjust the sensor's operating mode and strategy; according to the device's motion state, the inertial navigation system selects an adaptive sampling rate, adjusts the sensor's operating frequency, or switches to a high-precision mode or a low-power mode to ensure the efficiency and accuracy of data collection; by working in collaboration with the path planning module, the inertial navigation system provides real-time motion parameters and decision support for path planning, thereby achieving better positioning and navigation performance in complex environments.

[0013] Preferably, for different motion states, the self-motion perception model further optimizes path planning and system parameter adjustment through joint analysis of inertial measurement unit data and thermal imaging data, ensuring that high positioning accuracy and robustness can be maintained in dynamic and complex environments.

[0014] Preferably, the self-motion perception model adopts a deep neural network of multi-sensor information fusion; The self-motion perception model uses a convolutional neural network to extract heat source features in the environment from infrared images, and combines it with a recurrent neural network to process the time series sensor data provided by the inertial navigation system to track the motion trajectory of the device and predict its position changes. The convolutional neural network infers the motion state of the device by capturing the time dependency of sensor data, and can maintain stability even in the presence of noise or signal loss.

[0015] Preferably, the self-motion perception model also designs a fusion judgment module with adaptive weight allocation; the fusion judgment module dynamically adjusts the weight based on the density of heat source distribution in the scene, thereby avoiding wrong judgment in areas with scarce infrared features or large interference, ensuring the accuracy of motion state estimation and thermal imaging results.

[0016] Preferably, the scene perception module tracks dynamic targets in real time and processes thermal imaging camera data through a lightweight dynamic target recognition model to achieve real-time perception of dynamic targets; the lightweight dynamic target recognition model analyzes multi-frame information, tracks moving personnel or equipment, removes noise points, and extracts key dynamic features, thereby improving the system's perception accuracy of dynamic targets and providing real-time dynamic information support.

[0017] Preferably, the lightweight dynamic target recognition model performs time series feature extraction based on a multi-layer convolutional neural network; the lightweight dynamic target recognition model includes a shallow feature extraction part and a deep feature extraction part; the targets are divided into small targets and large targets, the shallow feature extraction part is used to detect small targets, and the deep feature extraction part is used to identify large targets; an interactive branch differential network is introduced into the lightweight dynamic target recognition model, dynamic target indexing is performed in time series, and misidentified targets are further screened out to ensure the accuracy of target recognition and tracking.

[0018] Preferably, the scene perception module detects high and low temperature targets and, based on the thermal imaging camera data, detects thermal radiation sources in the environment through a temperature analysis model and identifies areas with abnormal temperatures; it performs in-depth analysis on infrared imaging data to generate a thermal distribution map, thereby locating the source of the fire or potential rescue targets.

[0019] Preferably, the scene perception module also automatically identifies and marks entrances and exits and floors; the scene perception module automatically identifies and marks entrances and exits, floors and attack points in the environment through a deep model; The deep model automatically generates a spatial model of the building by analyzing the data from the thermal imaging camera, and identifies the locations of entrances, exits, floors, and attack points. The floor recognition and entrance and exit detection algorithms are combined with the inertial navigation system data to determine the floor location in real time.

[0020] Preferably, the scene perception module automatically identifies and marks entrances, exits and floors, and the corresponding sensor module also includes a lidar module and / or a barometer and / or a visible light camera module.

[0021] Preferably, the scene perception module is represented by a deep neural network, and automatically generates a spatial model of the building by analyzing the 3D point cloud data provided by the lidar module and / or the barometer data and / or the image data captured by the visible light camera module, and identifies the locations of entrances and exits, floors and attack points; the scene perception module extracts local features from the lidar module and / or visible light camera module data using a spatial convolutional neural network, and further identifies the geometric forms of these structures; further, a graph neural network is used to construct a spatial relationship graph, and the topological structure between spatial elements is optimized, so as to accurately identify the relative positions of entrances and exits and stairs; the floor recognition and entrance and exit detection algorithm is combined with the barometer and inertial navigation data, and the absolute height of the current floor of the building is calculated through deep feature representation, and the floor position is determined in real time.

[0022] Preferably, the scene perception module automatically identifies and marks entrances, exits, floors and attack points, and also manually marks or modifies them.

[0023] Preferably, the scene perception module also performs smoke and dust identification, and the corresponding sensor module also includes a lidar module and / or a visible light camera module; the scene perception module automatically identifies smoke and dust environments by analyzing the data of the lidar module and / or the visible light camera module; the lidar module generates high-precision point cloud data of the surrounding environment by emitting lasers and receiving return signals; the scene perception module uses data preprocessing technology to determine whether there is smoke or dust through the sparsity of the point cloud, changes in reflection intensity, and distance change characteristics.

[0024] Preferably, in the smoke and dust identification of the scene perception module, the corresponding sensor module also includes a visible light camera module; the image data provided by the visible light camera module assists in identifying the environmental characteristics of smoke and dust; through a threshold-based detection method, a machine learning algorithm or a deep learning algorithm, the scene perception module extracts the visual characteristics of smoke or dust from the image, and judges the presence of smoke and dust in a dynamic environment by combining the image's lighting changes, edge detection and color difference characteristics.

[0025] Preferably, in the smoke and dust recognition of the scene perception module, based on historical data and environmental models, the scene perception module gradually optimizes its recognition accuracy, responds in real time and provides accurate environmental perception information in a complex and dynamically changing environment.

[0026] Preferably, the multi-sensor fusion positioning module, the corresponding sensor module also includes a laser radar module, and the multi-sensor fusion positioning module includes specific sub-modules: Data cleaning module: First, pre-process the data of thermal imaging camera, inertial navigation module and lidar module, filter out noise observations through noise suppression model, and synchronize multi-sensor data through sensor synchronization signal to ensure the time synchronization of observation data of different sensors; Odometer positioning module: Through filtering, optimization or end-to-end state estimation methods, multi-source data from thermal imaging cameras, inertial navigation modules and lidar modules are integrated to recursively estimate the device's posture and build a multi-sensor fusion odometer; through the state estimation model, nonlinear motion in emergency scenarios is handled to ensure that the device can still provide high-precision real-time positioning when there is no satellite signal or the signal is interfered.

[0027] Preferably, the odometer positioning module filters the thermal imaging data collected by the thermal imaging camera, smoothes the stripe noise, and further extracts key radiation features based on the local radiation gradient; after the feature extraction, feature association is performed based on the local radiation feature descriptor to ensure robust matching of the heat source in the infrared image; the local radiation feature descriptor is encoded by the local heat source area in the infrared image.

[0028] Preferably, the sensor module corresponding to the multi-sensor fusion positioning module also includes a laser radar module. In order to balance the efficiency and accuracy of laser radar matching, a classification feature matching method is adopted: When the LiDAR data is identified as a surface feature, the point-to-surface distance matching method is used; for non-surface features, the point-to-distribution matching method is used, which uses the low dimensionality of surface features to reduce the amount of calculation. At the same time, the point-to-distribution constraints can also be used on non-surface features to enhance the robustness of the algorithm.

[0029] Preferably, in order to cope with possible degradation phenomena in the environment, an adaptive point cloud downsampling and matching parameter adjustment strategy is introduced to ensure the accuracy and stability of matching when the point cloud density is low or the feature distribution is uneven; in the case of final degradation, a fast normal distribution transformation is used for point cloud registration to ensure that the algorithm can still provide accurate matching and positioning results even under extreme conditions.

[0030] Preferably, the sensor module further includes a satellite navigation module, and the corresponding multi-sensor fusion positioning module includes submodules: Repositioning and global state optimization module: Use the global position information and repositioning constraints of the satellite navigation module to correct long-term posture errors and improve the positioning robustness and accuracy of the overall system.

[0031] Preferably, the multi-sensor fusion positioning module further includes a submodule: Positioning feedback and navigation guidance module: integrates positioning data and environmental perception information to provide real-time navigation guidance for rescue equipment, while supporting relative positioning between multiple machines.

[0032] Preferably, the relocalization and global state optimization module corrects the long-term posture error, specifically including: Absolute position observation fusion of satellite positioning and navigation module: In outdoor scenes, the global position information provided by the satellite positioning and navigation module is used as an absolute position reference to reduce cumulative drift. Based on a multimodal data fusion strategy, the absolute position data of the satellite is used as a soft constraint for low-frequency updates, and is fused with the high-frequency data of the lidar module, thermal imaging camera, and inertial navigation to gradually correct the position. In addition, an adaptive data credibility adjustment mechanism is introduced. When the satellite signal is weak or blocked, the satellite data weight is automatically reduced, and the fusion result of the lidar, thermal imaging camera, and inertial navigation is used. Multi-sensor position re-identification: The corresponding sensor module also includes a visible light camera module; a deep re-localization algorithm is used to fuse multi-sensor data to solve the limitations of traditional re-localization algorithms in complex environments; first, a re-localization feature description algorithm is used to extract global feature descriptors from the LiDAR point cloud and build a historical point cloud database; candidate matching frames of the current point cloud in the historical database are retrieved and determined; then multi-modal matching of infrared images and / or visible light images is introduced, and a similarity model is used to calculate the matching similarity between the current image and the candidate image, and the candidate frame sorting is optimized; in smoke or low-light environments, the scene state is automatically judged through the scene perception module, and infrared image data is used first to ensure the robustness of the re-localization algorithm; after determining the re-localization constraint identifier, point cloud registration is performed to generate the re-localization relative pose as the input for global pose graph optimization; Global pose graph optimization: Combine the state estimation method of multi-sensor fusion, the generated repositioning relative pose, and the absolute position observation provided by the satellite navigation system to construct and optimize the global pose graph; by solving the optimization problem, obtain the optimal estimate of the pose at all historical moments, update the historical frame sub-map and the global map, eliminate long-term accumulated errors, and achieve globally consistent high-precision mapping.

[0033] Preferably, the positioning feedback and navigation guidance module integrates positioning data and environmental perception information to provide real-time navigation guidance for rescue equipment and supports relative positioning between multiple machines, specifically including: Through collaborative computing and communication between devices, combined with decision-making algorithms, accurate relative posture determination and path optimization are achieved to build an efficient multi-device collaborative network; the decision-making algorithm analyzes multi-sensor data in real time, evaluates environmental complexity, dynamic obstacle distribution, and equipment operating status, and provides intelligent support for navigation guidance; for different scenarios, the decision algorithm dynamically adjusts the navigation strategy, and prioritizes obstacle avoidance paths in scenes with dense obstacles or smoke; in life detection tasks, the device collaborative path is optimized to quickly reach the target location.

[0034] Preferably, the equipment also includes a communication module to realize real-time communication between the device and the control center; the positioning information is fed back to the control center in real time through the communication module, and the global information is comprehensively analyzed to generate real-time scheduling suggestions to optimize the task allocation and collaboration paths between multiple devices.

[0035] Preferably, multiple devices communicate with each other via Lora or 4G or WiFi; when 4G and WiFi signals are poor, they communicate with each other via Lora and calculate the floor difference and 2D horizontal distance between each other.

[0036] Preferably, the real-time dispatch suggestion generated by the control center also includes dispatching personnel from other equipment to rescue when a problem occurs to personnel of a certain equipment and they are trapped.

[0037] Preferably, the equipment is one or more, and each equipment communicates with the control center and the server in real time through a relay base station.

[0038] Preferably, the data fusion mapping module specifically includes: Single-machine data fusion and local two-dimensional temperature occupancy map construction: On the single-machine side, the thermal imaging data captured by the thermal imaging camera and the local point cloud generated by the inertial navigation system are integrated to realize the construction of a two-dimensional temperature occupancy map using an algorithm model. The two-dimensional temperature occupancy map represents the spatial structure and temperature distribution information around the equipment in a rasterized form, showing fire sources, living things and high-temperature hazardous areas. The algorithm model analyzes infrared thermal images and point cloud data in real time, intelligently filters out noise and optimizes the grid update rate to ensure the accuracy and real-time nature of map construction. In a dynamic environment, the algorithm model automatically adjusts the perception weight according to environmental changes.

[0039] Preferably, the sensor module corresponding to the data fusion mapping module also includes a lidar module. Under smoke and dust-free conditions, the thermal imaging data captured by the thermal imaging camera and the local point cloud generated by the inertial navigation system and the lidar module data are integrated to realize the construction of a two-dimensional temperature occupancy map using an algorithm model.

[0040] Preferably, the server-side multi-machine global three-dimensional environmental temperature map construction specifically includes: Server-side multi-machine global three-dimensional environmental temperature mapping and dynamic update: The server-side integrates the sensor data of multiple devices and uses scene re-identification and multi-machine relocalization technology to build a globally consistent three-dimensional fusion environment map, dynamically update the static structure and dynamic targets in the three-dimensional map, and realize fine modeling of the global environment; the scene re-identification algorithm analyzes infrared thermal imaging data from multiple devices to identify and match similar scene areas, thereby aligning the observation data between different devices; the multi-machine relocalization detection technology combines the global feature descriptors generated by deep learning to retrieve similar scenes in the historical point cloud database, and determines the relocalization matching frames through further verification of infrared images; the server uses the fused relocalization constraint information to globally optimize the local maps of each device, eliminate the drift accumulation error through posture adjustment, and generate a global three-dimensional map.

[0041] Preferably, the sensor modules corresponding to the global three-dimensional ambient temperature mapping and dynamic updating of multiple machines on the server side also include a lidar module and a visible light camera module: in the specific steps, the scene re-identification algorithm identifies and matches similar scene areas by analyzing multi-sensor data of lidar point clouds, infrared thermal images and visible light images from multiple devices, thereby realizing alignment of observation data between different devices; the multi-machine relocation detection technology is combined with the global feature descriptor generated by deep learning, similar scenes are retrieved in the historical point cloud database, and the relocation matching frames are determined through further verification of visible light images and infrared images.

[0042] Preferably, the server-side multi-machine global three-dimensional environmental temperature map construction specifically includes: Precise segmentation of remote 3D maps: The server-side global 3D environment map is accurately segmented through a segmentation algorithm, and the objects in the scene are accurately divided into at least one of the following elements: entrances and exits, walls, ground, obstacles, and living things. The segmentation model uses fused multi-source data and combines global scene features to achieve semantic segmentation and object recognition.

[0043] Preferably, the server-side multi-machine global three-dimensional environmental temperature map construction specifically includes: 3D rendering of temperature map: The remote 3D environmental map is further combined with the thermal imaging data of the thermal imaging camera to achieve 3D rendering of the temperature map through the temperature mapping algorithm. The real-time temperature information is superimposed on the 3D point cloud to intuitively display the temperature distribution of different areas in the scene, including the specific location and temperature change trend of at least one of the attack point, fire source, heat source and life form.

[0044] Preferably, the equipment adopts end-to-cloud integrated cluster collaboration and multi-link stable transmission technology: it adopts end-to-cloud data processing and cluster collaboration technology, supports multiple signal transmission methods and relay communication modes between end and cloud.

[0045] Preferably, the equipment is wearable equipment.

[0046] Preferably, the equipment is a handheld equipment.

[0047] A computer-readable medium stores at least one computer program, which is loaded and executed by a processor to enable an electronic device to implement the working method of the equipment.

[0048] A fire emergency robot is equipped with the multi-sensor fusion positioning mapping and thermal imaging equipment.

[0049] A fire emergency robot dog is equipped with the multi-sensor fusion positioning mapping and thermal imaging equipment.

[0050] A fire emergency drone is equipped with the multi-sensor fusion positioning mapping and thermal imaging equipment.

[0051] The advantages of the present invention are: The present invention proposes a multi-sensor fusion positioning, mapping and thermal imaging equipment for fire emergency scenarios and related computer-readable media. The system integrates positioning and mapping sensors such as thermal imaging cameras, visible light cameras, lidars and inertial navigation systems, and is designed for high-risk and complex fire emergency environments. Through the end-cloud integrated cluster collaboration system and multi-link stable transmission technology, combined with intelligent decision support, the present invention can achieve high-precision real-time positioning and dynamic 3D temperature measurement and mapping in environments where satellite navigation signals are weak or unavailable. The system's multi-mode portability and environmental adaptability design ensure the efficiency and safety of operators under extreme conditions, and its advanced data processing and analysis technology significantly improves the accuracy and response speed of rescue operations. These innovative features make the present invention have significant practical application value and market prospects in the field of fire rescue. The specific advantages of the present invention are summarized as follows: 1. End-to-cloud integrated cluster collaboration and multi-link stable transmission technology: It adopts end-to-cloud data processing and cluster collaboration technology, supports multiple signal transmission methods such as 4G, WiFi, LoRa, etc., and ensures real-time transmission and efficient processing of information in extreme fire rescue environments, especially in severe smoke and dust conditions.

[0052] 2. Devices communicate with each other through Lora and calculate the floor difference and 2D horizontal distance between each other, which is especially suitable for situations where 4G / WiFi signals are poor.

[0053] 3. Multi-sensor fusion positioning and mapping technology: Through state estimation algorithms (filtering, optimization, deep end-to-end, etc.), sensors such as thermal imaging, visible light cameras, lidar and inertial navigation systems are deeply integrated to achieve high-precision real-time positioning and mapping in environments with limited vision and weak or missing navigation satellite signals.

[0054] 4. AI-driven intelligent decision support: Introduce AI algorithms to optimize data processing and decision support, and improve the system's response speed and decision accuracy in rapidly changing rescue environments.

[0055] 5. Multi-mode portability and environmental adaptability design: The equipment supports handheld and wearable modes, adapting to harsh environments such as high temperature, high humidity, and smoke, improving the flexibility and efficiency of operators.

[0056] 6. Efficient spatial information processing and real-time key point marking: The system can accurately detect and mark key locations such as entrances and exits, safety passages, and support automatic judgment and identification of floors, providing real-time spatial information and navigation support for rescue personnel.

[0057] 7. Temperature map construction by integrating thermal imaging and multi-source data: Combining thermal imaging with 3D point cloud data obtained by other sensors, 3D temperature contour mapping and comprehensive mapping technology are realized to provide detailed fire source and hot spot distribution information for rescue decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 It is a schematic diagram of the device structure and algorithm of the present invention; Figure 2 It is a comprehensive diagram of the device and cloud of the present invention. DETAILED DESCRIPTION

[0059] The present invention proposes a multi-sensor fusion positioning, mapping and thermal imaging equipment for fire emergency scenarios and its computer-readable medium. The system integrates multiple perception and mapping sensors such as thermal imaging, visible light cameras, lidar and inertial navigation systems. Through end-cloud integrated cluster collaboration and multi-link stable transmission technology, it can achieve high-precision real-time positioning and 3D mapping in extreme environments, which is particularly suitable for complex fire emergency scenarios such as smoke, dust, and high temperature. Combined with intelligent decision support, the system has fast data processing and response capabilities, and provides visualization of dynamic 3D scenes and temperature distribution information. Its multi-mode portability design supports handheld and wearable use, ensuring the flexibility and safety of operators. The equipment has efficient spatial information processing and real-time key point marking functions, which can automatically identify and mark locations such as entrances and exits, floors, and generate 3D temperature maps through multi-sensor fusion, clearly displaying the distribution of fire sources and hot spots, and providing accurate support for firefighting and rescue decisions.

[0060] like Figure 1 As shown, an implementation structure of the specific equipment of the present invention is as follows.

[0061] Thermal imaging camera: Acquires image information through thermal imaging in dark or low-light environments, ensuring effective visual perception capabilities under complex lighting conditions, especially suitable for scenes such as fires or mining disasters.

[0062] Visible light camera module: provides standard visual information, supplements the imaging requirements of thermal imaging cameras under different lighting conditions, further enhances the equipment's environmental recognition capabilities, and provides clear visual information support for 3D mapping.

[0063] Laser Radar (LiDAR): Provides high-precision 3D point cloud data and generates detailed 3D environmental models. Its excellent 3D reconstruction capability is suitable for narrow and complex underground passages and building interiors, and can provide comprehensive spatial information in different terrains and environments.

[0064] Inertial Navigation System (INS): Provides short-term high-precision motion status information, including acceleration and angular velocity, so that the device can maintain its own posture perception capability even in the absence of external signals. For example, in environments without navigation satellite signals such as mines and basements, INS can accurately identify the device's upstairs and downstairs, stationary and horizontal motion states, providing reliable data for path planning and sensor strategy optimization.

[0065] Satellite positioning and navigation module: Provides global navigation and positioning information in open-air scenarios, ensuring long-term global positioning capabilities, especially suitable for wide-area fire emergency scenarios. For indoor rescue, it can support the algorithm to initialize the absolute position outdoors, and realize the association between the indoor algorithm positioning results and the absolute geographic location.

[0066] Power module: Provides long-term endurance to ensure that the equipment maintains continuous and stable operation during long-term high-temperature or low-temperature emergency tasks and adapts to extreme high-temperature and low-temperature environments.

[0067] Communication module: supports multiple data transmission methods such as 4G, LoRa, WiFi, etc., and supports relay base stations to achieve real-time communication between the device and the command center or other equipment, facilitating data sharing and command decision support in fire emergency scenarios.

[0068] The above modules specifically support the work of the following modules.

[0069] 1. Scene perception module.

[0070] The scene perception module is responsible for real-time perception of dynamic and static environmental information in emergency scenarios, ensuring that global and local information about the surrounding environment can be accurately obtained under complex environmental conditions (such as low light, smoke, dust, no navigation satellite signals, etc.). The module integrates the inertial navigation system's perception of the device's own motion, the real-time tracking of dynamic targets, and the thermal imaging camera's high and low temperature target detection capabilities, providing comprehensive perception of static and dynamic obstacles, life forms, fire sources, etc., and providing precise support for emergency response.

[0071] ① Self-motion perception: Through the inertial navigation system combined with the self-motion perception model, the self-motion perception model is preferably an AI model. This module can accurately judge the device's own motion state, such as going up and down stairs, standing still, walking on flat ground, falling, etc. INS uses data from the accelerometer and gyroscope to accurately monitor the posture changes of the device even in an environment without navigation satellite signals. The self-motion perception model optimizes data processing to ensure accurate identification of movement, falling, going up and down stairs / escalators / elevators, walking, and running in complex environments (such as mines or basements). This function helps the system dynamically adjust sensor strategies and provide reliable motion parameters and efficient decision support for path planning.

[0072] The scene perception module perceives the device's own motion through the inertial navigation system combined with the self-motion perception model: the inertial navigation system monitors the device's posture changes in real time through the data of the accelerometer and gyroscope. The system combines the inertial measurement unit (IMU) data and thermal imaging data, uses sensor fusion technology, and optimizes motion estimation through Kalman filtering or particle filtering algorithms to reduce the impact of sensor noise and improve positioning accuracy. Specifically, the inertial navigation system collects the acceleration information of the device through the accelerometer, obtains the angular velocity data through the gyroscope, and estimates the instantaneous speed and displacement of the device in combination with the sensor data fusion model. In order to ensure that the system can accurately distinguish between vertical and horizontal motion states, the self-motion perception model introduces a motion state classification algorithm in data processing, using a combination of threshold-based detection methods, machine learning algorithms, and deep learning algorithms. For example, the Z-axis data of the accelerometer is used to monitor whether there is an obvious vertical acceleration change, so as to determine whether it is a vertical motion state such as going up and down stairs, escalators or elevators; based on the data of the gyroscope, it can be determined whether it is a dynamic state such as horizontal walking or running. In addition, the device's ability to perceive its own motion enables the system to dynamically adjust the working mode and strategy of the sensor. According to the motion state of the device, the system can select an adaptive sampling rate, adjust the operating frequency of the sensor, or switch to different working modes (such as high-precision mode or low-power mode) to ensure the efficiency and accuracy of data collection. At the same time, by working in collaboration with the path planning module, the system can provide real-time motion parameters and decision support for path planning, thereby achieving better positioning and navigation performance in complex environments. For different motion states, such as walking, running, going up and down stairs, and falling, the model further optimizes path planning and system parameter adjustments through joint analysis of IMU data and thermal imaging data, ensuring that high positioning accuracy and robustness can be maintained in dynamic and complex environments.

[0073] The self-motion perception model adopts a deep neural network with multi-sensor information fusion. Specifically, the model uses a convolutional neural network (CNN) to extract heat source features in the environment from infrared images, and combines a recurrent neural network (RNN) to process the time series sensor data provided by the inertial navigation module to track the motion trajectory of the device and predict its position changes. RNN effectively infers the motion state of the device by capturing the temporal dependency of sensor data, and can maintain high stability even in the presence of noise or signal loss. In order to further improve the robustness of the model, we designed a fusion judgment module with adaptive weight allocation. This module dynamically adjusts the weights based on the density of heat source distribution in the scene, thereby avoiding misjudgments in areas with scarce infrared features or large interference, and ensuring the accuracy of motion state estimation and thermal imaging results.

[0074] ② Dynamic target perception: This module processes multi-source data through a lightweight dynamic target recognition model, which preferably uses an AI model to achieve real-time perception of dynamic targets. The dynamic target recognition model analyzes multi-frame information to accurately identify and track moving personnel or equipment. It can remove noise points and extract key dynamic features in complex environments such as smoke and dust, thereby improving the system's perception accuracy of dynamic targets and providing reliable real-time dynamic information support for rescue operations.

[0075] Specifically, the lightweight dynamic target recognition model extracts temporal features based on a multi-layer convolutional neural network (CNN). In this model, the shallow feature extraction part has a strong spatial information expression capability and a small receptive field, which is suitable for detecting small targets; while the deep feature extraction part has a strong semantic information capture capability and a large receptive field, which is mainly used to identify larger targets. In order to improve the detection accuracy and robustness of dynamic targets, an Inter-branch Differential Network is introduced into the model, which can perform dynamic target indexing in time series, further filter out misidentified targets, and ensure the accuracy of target recognition and tracking.

[0076] ③ High and low temperature target detection: Based on the thermal imaging camera data, the temperature analysis model is used to achieve high-precision detection of thermal radiation sources in the environment, which is especially suitable for low-light or no-light conditions. The temperature analysis model is preferably an AI model. In emergency scenarios such as fires or mine disasters, this module can identify areas with abnormal temperatures, including fire sources and living things (such as trapped people or animals). In-depth analysis of infrared imaging data is performed to generate a thermal distribution map, which helps rescuers quickly locate fire sources or potential rescue targets, significantly improves the system's scene perception capabilities under invisible light conditions, and provides vital information support for emergency response.

[0077] ④ Automatic identification and marking of entrances and exits and floors: The corresponding sensor module also includes a lidar module and / or a barometer and / or a visible light camera module; this module combines data from multiple sensors to automatically identify and mark the entrances and exits and floor information in the environment through a deep model. The system automatically generates a spatial model of the building by analyzing 3D data from sensors such as lidar and visible light cameras, and identifies key locations such as entrances and exits, floors, and attack points. The floor identification and entrance and exit detection algorithm combined with sensor data can determine the floor position in real time, provide accurate floor positioning and entrance and exit information for rescue personnel, help quickly plan escape routes and determine safe passages, and provide key support for command decisions and path planning. The floor identification and entrance and exit detection algorithm preferably uses an AI algorithm. This module significantly improves the system's navigation and recognition capabilities in complex environments, and provides efficient spatial information support for emergency response and rescue missions.

[0078] The deep model automatically generates a spatial model of the building by analyzing the data from the thermal imaging camera, and identifies the locations of entrances, exits, floors, and attack points; the floor recognition and entrance and exit detection algorithm is combined with the inertial navigation system data to determine the floor location in real time.

[0079] The scene perception module uses a spatial convolutional neural network to extract local features from the lidar module and / or visible light camera module data, and further identifies the geometric forms of these structures; further, a graph neural network is used to construct a spatial relationship graph and optimize the topological structure between spatial elements, so as to accurately identify the relative positions of entrances and exits and stairs; the floor recognition and entrance and exit detection algorithm is combined with the barometer and inertial navigation data, and the absolute height of the current floor of the building is calculated through deep feature representation, and the floor position is determined in real time.

[0080] The scene perception module uses a deep neural network representation to automatically generate a spatial model of the building by analyzing the 3D point cloud data provided by the lidar module, the barometer data, and the image data captured by the visible light camera module, and identify the key structural locations such as entrances and exits, stairs, etc. Specifically, a spatial convolutional neural network (CNN) is used to extract local features such as doors, windows, and stairways from the lidar and visible light camera data, and further identify the geometric forms of these structures. Furthermore, a graph neural network (GNN) is used to construct a spatial relationship graph and optimize the topological structure between spatial elements, so as to accurately identify the relative positions of entrances and exits and stairs. The floor recognition and entrance and exit detection algorithm combines barometer and inertial navigation data, calculates the absolute height of the current floor of the building through deep feature representation, and determines the floor position in real time to ensure accurate positioning of floor changes in a multi-layer environment.

[0081] The scene perception module automatically identifies and marks entrances, exits, floors and attack points, and also manually marks or modifies them.

[0082] ⑤ Smoke and dust identification: The corresponding sensor module also includes a laser radar module and / or a visible light camera module; the scene perception module automatically identifies the smoke and dust environment by analyzing the data of the laser radar module and / or the visible light camera module. Specifically, the laser radar module generates high-precision point cloud data of the surrounding environment by emitting lasers and receiving return signals. In a smoke or dust environment, the laser signal will be scattered and attenuated, resulting in a decrease in the density of the point cloud data or an obvious abnormal distribution. In order to identify smoke and dust, the scene perception module uses data preprocessing technology to determine whether there is smoke or dust through features such as the sparsity of the point cloud, changes in reflection intensity, and changes in distance. The image data provided by the visible light camera module can also assist in identifying the environmental characteristics of smoke and dust. Through threshold-based detection methods, machine learning algorithms, or deep learning algorithms, the system can extract visual features of smoke or dust from images, such as blurred vision, floating particles, and halos in images. By combining features such as image illumination changes, edge detection, and color differences, the system can determine the presence of smoke and dust in a dynamic environment. In addition, based on historical data and environmental models, the scene perception module can gradually optimize its recognition accuracy, enabling the system to respond in real time and provide accurate environmental perception information in complex and dynamically changing environments.

[0083] 2. Multi-sensor fusion positioning module.

[0084] The multi-sensor fusion positioning module is based on state estimation methods such as filtering, optimization, and end-to-end, fusing data from multiple sensors (including thermal imaging cameras, visible light cameras, LiDAR, INS, and satellite positioning and navigation systems) to generate accurate device pose estimates. The core task of this module is to perform comprehensive processing of perception data obtained by different sensors to ensure that the system can still maintain accurate positioning and navigation capabilities in extreme environments. The specific submodules are as follows: ① Data cleaning module: First, pre-process the multi-sensor data such as thermal imaging cameras, LiDAR, INS and GNSS, filter out noise observations through the noise suppression model, and synchronize the multi-sensor data through sensor synchronization signals (PPS, etc.) to ensure the time synchronization of observation data from different sensors and ensure data quality.

[0085] ② Odometer positioning module: Through filtering, optimization or end-to-end state estimation methods, multi-source data from the thermal imaging camera module and inertial navigation module are integrated, the posture of the device is recursively estimated, and a multi-sensor fusion odometer is constructed; through the state estimation model, nonlinear motion situations in emergency scenarios are handled to ensure that the device can still provide high-precision real-time positioning when there is no satellite signal or the signal is interfered.

[0086] In the presence of smoke and dust, the odometer positioning module filters the thermal imaging data collected by the thermal imaging camera, smoothes the stripe noise, and further extracts key radiation features based on the local radiation gradient; after feature extraction, feature association is performed based on the local radiation feature descriptor to ensure robust matching of heat sources in infrared images. Specifically, the local radiation feature descriptor encodes the local heat source area in the infrared image, similar to the key point descriptor in ORB feature extraction, and can maintain strong matching stability and robustness under conditions such as noise interference and heat source changes. This method can improve the matching accuracy and reliability of thermal imaging data in motion, and provide accurate infrared positioning support for the data of the fused inertial navigation module.

[0087] In the absence of smoke and dust, the sensor module corresponding to the multi-sensor fusion positioning module also includes a laser radar module. In order to balance the efficiency and accuracy of laser radar matching, a classification feature matching method is adopted. When the laser radar data is identified as a surface feature, the point-to-surface distance matching method is adopted; for non-surface features, the point-to-distribution matching method is adopted. This method can make full use of the low dimensionality of surface features and reduce the amount of calculation. At the same time, it can also use the point-to-distribution constraint on non-surface features to enhance the robustness of the algorithm. Furthermore, in order to cope with possible degradation phenomena in the environment, we introduced adaptive point cloud downsampling and matching parameter adjustment strategies to ensure the accuracy and stability of matching when the point cloud density is low or the feature distribution is uneven. In the case of final degradation, Fast-NDT (fast normal distribution transform) is used for point cloud registration to ensure that even under extreme conditions, the algorithm can still provide accurate matching and positioning results, ensuring the robustness and efficiency of the system in complex environments.

[0088] ③ Repositioning and global state optimization: The corresponding sensor module also includes a satellite navigation module. The position and posture obtained by the INS and filtering, optimization and end-to-end state estimation algorithms are continuous, but there is positioning drift in long-term operation. In order to further improve the positioning accuracy, the global position information and repositioning constraints of the satellite navigation system are used to correct the long-term position and posture errors, and improve the positioning robustness and accuracy of the overall system, as follows: Absolute position observation fusion of satellite positioning and navigation system (taking a single satellite as an example): In outdoor scenarios, the global position information provided by the satellite navigation system is used as an absolute position reference to effectively reduce cumulative drift. Through a multimodal data fusion strategy based on Bayesian filtering, the absolute position data of the satellite is used as a soft constraint for low-frequency updates, and is fused with high-frequency data such as lidar and inertial navigation to gradually correct the position. In addition, the system introduces an adaptive data credibility adjustment mechanism. When the satellite signal is weak or blocked, the satellite data weight is automatically reduced, and the fusion results of lidar and inertial navigation are used first to ensure the stability and robustness of the system in complex environments such as indoors and underground.

[0089] Multi-sensor position re-identification: The system uses a deep relocation algorithm to fuse multi-sensor data to solve the limitations of traditional relocation algorithms in complex environments. The deep relocation algorithm preferably uses an AI algorithm. First, the AI ​​relocation feature description algorithm is used to extract global feature descriptors from the LiDAR point cloud and build a historical point cloud database. Through rapid retrieval, the candidate matching frames of the current point cloud in the historical database are determined. Then, multimodal matching of infrared images and visible light images is introduced, and the matching similarity between the current image and the candidate image is calculated using the AI ​​similarity model to optimize the candidate frame sorting. In smoke or low-light environments, the system automatically determines the scene state through the scene perception module, and gives priority to infrared image data to ensure the robustness of the relocation algorithm. After determining the relocation constraint identifier, a deep algorithm is used for point cloud registration to generate a high-precision relocation relative pose as the input for global pose graph optimization.

[0090] Global pose graph optimization: Combine the state estimation method of multi-sensor fusion, the repositioning relative pose generated by the self-developed AI algorithm, and the absolute position observation provided by the satellite navigation system to build and optimize the global pose graph. By solving the optimization problem, the system obtains the optimal estimate of the pose at all historical moments, updates the historical frame sub-map and the global map, eliminates long-term accumulated errors, and achieves globally consistent high-precision mapping. This optimization method ensures that the system has stable long-term positioning and mapping capabilities in complex environments.

[0091] ④ Positioning feedback and navigation guidance: Integrating positioning data with environmental perception information, this module provides real-time navigation guidance for rescue equipment, and supports relative positioning between multiple machines, serving battlefield collaboration and rescue scenarios. The system achieves accurate relative posture determination and path optimization through collaborative computing and communication between devices, combined with decision-making algorithms, and builds an efficient multi-device collaborative network. The decision-making algorithm prefers AI algorithms. The AI ​​decision-making algorithm can analyze multi-sensor data in real time, evaluate environmental complexity, dynamic obstacle distribution, and equipment operating status, and provide intelligent support for navigation guidance. For different scenarios, the AI ​​algorithm dynamically adjusts the navigation strategy. For example, in scenes with dense obstacles or smoke, it prioritizes obstacle avoidance paths; in life detection tasks, it optimizes equipment collaborative paths to quickly reach the target location. In addition, positioning information is fed back to the control center in real time through the communication module. The AI ​​algorithm conducts a comprehensive analysis of global information, generates real-time scheduling suggestions, and optimizes task allocation and collaborative paths between multiple devices. This function not only improves rescue efficiency, but also enhances the system's responsiveness in complex dynamic environments, providing accurate and reliable navigation and scheduling support for emergency tasks.

[0092] 3. Multi-machine and multi-source data fusion mapping module.

[0093] The multi-machine multi-source data fusion mapping module is responsible for processing the data of multiple sensors of single and multiple machines for fusion mapping, which mainly includes the construction of local two-dimensional temperature occupancy maps and the construction of multi-machine global three-dimensional environment fine maps on the server side. Through the deep fusion and real-time dynamic update of multi-source data, the module can provide emergency response personnel with accurate and comprehensive environmental perception, help them grasp the on-site layout and dynamic changes in real time, and provide strong support for decision-making, path planning and task scheduling.

[0094] ① Single-machine data fusion and local two-dimensional temperature occupancy map construction: On the single-machine side, the thermal imaging data captured by the thermal imaging camera and the local point cloud generated by the sensor data such as the inertial navigation system are integrated, and the lightweight algorithm is used to realize the efficient two-dimensional temperature occupancy map construction. The lightweight algorithm preferably uses the AI ​​algorithm. The map represents the spatial structure and temperature distribution information around the equipment in a rasterized form, and intuitively displays the fire source, life form and high temperature danger area. The lightweight AI algorithm analyzes the infrared thermal image and point cloud data in real time, intelligently filters out noise and optimizes the grid update rate to ensure the accuracy and real-time performance of map construction. In a dynamic environment, the AI ​​model can automatically adjust the perception weight according to environmental changes. For example, it can give priority to strengthening the processing of thermal image data under smoke or low light conditions, so that the generated occupancy map is more in line with the current scene requirements. This module not only improves the efficiency of occupancy map construction, but also significantly reduces the consumption of computing resources, ensuring that the equipment can continuously generate efficient and reliable local environmental information in complex emergency scenarios, providing strong support for local navigation, obstacle avoidance and path planning, and effectively improving the work efficiency and safety of front-line combatants.

[0095] In the absence of smoke and dust, the sensor module corresponding to the data fusion mapping module also includes a lidar module, which fuses the thermal imaging data captured by the thermal imaging camera and the local point cloud generated by the inertial navigation system and the lidar module data, and uses the algorithm model to realize the construction of a two-dimensional temperature occupancy map.

[0096] ② Construction of a global three-dimensional environmental temperature map of multiple machines on the server side: The server side integrates the sensor data of multiple devices and uses scene re-identification and multi-machine re-positioning technology to build a globally consistent three-dimensional fusion environment map. This module can dynamically update the static structure and dynamic targets in the three-dimensional map to achieve fine modeling of the global environment. The scene re-identification algorithm preferably uses an AI algorithm, which identifies and matches similar scene areas by analyzing multi-sensor data such as infrared thermal images from multiple devices, and realizes alignment of observation data between different devices. The multi-machine re-positioning detection technology combines the global feature descriptor generated by deep learning to quickly retrieve similar scenes in the historical point cloud database, and further verify the infrared image to determine the re-positioning matching frame. For complex scenes (such as smoke and dark environments), the system can dynamically adjust the perception strategy according to the scene state to ensure the robustness of the algorithm. The server uses the fused re-positioning constraint information to globally optimize the local maps of each device, eliminate the drift accumulation error through posture adjustment, and generate a global three-dimensional map with high consistency and wide coverage. Real-time dynamic update technology can accurately reflect changes in the scene, including new obstacles, dynamic target locations, and fire source spread. This module provides the command center with globally consistent three-dimensional environmental perception capabilities, provides complete environmental model support for multi-device collaborative rescue missions, optimizes task allocation and path planning, realizes global multi-machine collaboration and improves emergency response efficiency, and provides strong rear support for combatants.

[0097] The sensor modules corresponding to the server-side multi-machine global three-dimensional ambient temperature mapping and dynamic update also include lidar modules and visible light camera modules: in the specific steps, the scene re-identification algorithm analyzes the multi-sensor data of lidar point clouds, infrared thermal images and visible light images from multiple devices, identifies and matches similar scene areas, and realizes the alignment of observation data between different devices; the multi-machine relocation detection technology is combined with the global feature descriptor generated by deep learning, retrieves similar scenes in the historical point cloud database, and determines the relocation matching frame through further verification of visible light images and infrared images.

[0098] ③ Accurate segmentation of remote 3D maps: The global 3D environment map on the server side is accurately segmented through a segmentation algorithm. The segmentation algorithm preferably uses an AI segmentation algorithm to accurately divide the objects in the scene into key elements such as entrances and exits, walls, ground, obstacles, and living things. The deep learning-based segmentation model uses fused multi-source data (including LiDAR point clouds, infrared thermal imaging data, and visible light images, etc.) and combines global scene features to achieve high-precision semantic segmentation and object recognition. The results of accurate segmentation can help commanders quickly grasp the location and status of key areas in the scene, clarify safe paths, passable areas, and potential rescue targets, and provide refined environmental perception support for rescue missions. Through this algorithm, the system can dynamically update the segmentation results, promptly reflect scene changes, and provide more intelligent guidance for equipment navigation, path planning, and mutual rescue.

[0099] ④ Three-dimensional rendering of temperature map: The remote three-dimensional environmental map is further combined with the thermal imaging data of the thermal imaging camera, and the temperature map is rendered in three dimensions through the temperature mapping algorithm. The temperature mapping algorithm preferably uses an AI-driven algorithm. This function superimposes the temperature information acquired in real time onto the three-dimensional point cloud, and intuitively displays the temperature distribution in different areas of the scene, including the specific location and temperature change trend of the attack point, fire source, heat source, and living organisms. Three-dimensional temperature rendering can help commanders quickly identify high-temperature areas and potential directions for fire spread, and provide real-time reference for the dynamic adjustment of rescue missions and the division of safe areas. At the same time, this function provides combatants with three-dimensional visualization support for heat sources, improving the efficiency of emergency response and the accuracy of rescue decisions in complex environments.

[0100] like Figure 2 As shown, the equipment of the present invention is one or more, and the multiple equipment communicates with the control center and the server in real time through the relay base station. The communication adopts the end-to-cloud integrated cluster collaboration and multi-link stable transmission technology: the end-to-cloud data processing and cluster collaboration technology are adopted. The positioning information is fed back to the control center in real time through the communication module, and the global information is comprehensively analyzed to generate real-time scheduling suggestions to optimize the task allocation and collaboration paths between multiple devices.

[0101] Each device of the present invention supports multiple signal transmission modes such as 4G, WiFi, LoRA, etc., ensuring real-time transmission and efficient processing of information in extreme fire rescue environments. Especially in the case of severe smoke and dust, when 4G and WiFi signals are poor, multiple devices communicate with each other through Lora and calculate the floor difference and 2D horizontal distance between each other.

[0102] The real-time dispatch suggestions generated by the control center also include dispatching personnel from other equipment to rescue when a problem occurs to personnel of a certain device and they are trapped.

[0103] The multi-sensor fusion positioning, mapping and thermal imaging equipment described in the present invention is wearable equipment.

[0104] The multi-sensor fusion positioning, mapping and thermal imaging equipment described in the present invention is a handheld device.

[0105] The present invention further proposes a computer-readable medium, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement the working method of the above-mentioned equipment.

[0106] The present invention also proposes a fire emergency robot equipped with the multi-sensor fusion positioning mapping and thermal imaging equipment.

[0107] The present invention also proposes a fire emergency robot dog equipped with the multi-sensor fusion positioning mapping and thermal imaging equipment.

[0108] The present invention also proposes a fire emergency drone equipped with the multi-sensor fusion positioning mapping and thermal imaging equipment.

[0109] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any modifications made according to the spirit of the main technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. Multi-sensor fusion positioning, mapping and thermal imaging equipment, characterized by: include: A sensor module, including at least a thermal imaging camera and an inertial navigation system; The data fusion mapping module is responsible for processing multiple sensor data for fusion mapping, including the construction of local two-dimensional temperature occupancy maps.

2. Multi-sensor fusion positioning, mapping and thermal imaging equipment, characterized by: include: A sensor module, including at least a thermal imaging camera and an inertial navigation system; The data fusion mapping module is responsible for processing multiple sensor data for fusion mapping, including the construction of multi-machine global three-dimensional environmental temperature maps on the server side.

3. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 1, characterized in that: The fusion mapping of the data fusion mapping module also includes the construction of multi-machine global three-dimensional ambient temperature maps on the server side.

4. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 1, characterized in that: Also includes: The scene perception module perceives the dynamic and static environmental information in the emergency scene in real time, including the perception of the device's own movement, real-time tracking of dynamic targets, and detection of high and low temperature targets.

5. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 1, characterized in that: Also includes: The multi-sensor fusion positioning module fuses data from multiple sensors to generate device pose estimation.

6. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 4, characterized in that: The scene perception module perceives the device's own motion through the inertial navigation system combined with the self-motion perception model: the inertial navigation system monitors the device's posture changes through the data of the accelerometer and gyroscope; based on the thermal imaging camera and the inertial navigation system, the self-motion perception model optimizes data processing to ensure accurate identification of movement, falls, going up and down stairs / escalators / elevators, walking, and running; the perception of the device's own motion helps the system dynamically adjust the sensor strategy and provide motion parameters and decision support for path planning.

7. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 4, characterized in that: The inertial navigation system combines its own inertial measurement unit data and thermal imaging data, uses sensor fusion technology, optimizes motion estimation through Kalman filtering or particle filtering algorithm, reduces the impact of sensor noise, and improves positioning accuracy; specifically, the inertial navigation system collects the acceleration information of the device through the accelerometer, obtains angular velocity data through the gyroscope, and estimates the instantaneous speed and displacement of the device in combination with the sensor data fusion model; in order to ensure that the system can accurately distinguish between vertical and horizontal motion states, the self-motion perception model introduces a motion state classification algorithm in data processing, and adopts a combination of a threshold-based detection method, a machine learning algorithm, and a deep learning algorithm; the Z-axis data of the accelerometer is used to monitor whether there is an obvious vertical acceleration change, so as to determine whether it is a vertical motion state of going up and down stairs, escalators or elevators; and the data based on the gyroscope is used to determine whether it is a horizontal walking or running dynamic state.

8. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 7, characterized in that: The scene perception module's perception of the device's own motion enables the inertial navigation system to dynamically adjust the sensor's operating mode and strategy; according to the device's motion state, the inertial navigation system selects an adaptive sampling rate, adjusts the sensor's operating frequency, or switches to a high-precision mode or a low-power mode to ensure the efficiency and accuracy of data collection; By working in collaboration with the path planning module, the inertial navigation system provides real-time motion parameters and decision support for path planning, thereby achieving better positioning and navigation performance in complex environments.

9. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 8, characterized in that: For different motion states, the self-motion perception model further optimizes path planning and system parameter adjustment through joint analysis of inertial measurement unit data and thermal imaging data, ensuring that high positioning accuracy and robustness can be maintained in dynamic and complex environments.

10. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 6, characterized in that: The self-motion perception model adopts a deep neural network of multi-sensor information fusion; The self-motion perception model uses a convolutional neural network to extract heat source features in the environment from infrared images, and combines it with a recurrent neural network to process the time series sensor data provided by the inertial navigation system to track the motion trajectory of the device and predict its position changes. The convolutional neural network infers the motion state of the device by capturing the time dependency of sensor data, and can maintain stability even in the presence of noise or signal loss.

11. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 9, characterized in that: The self-motion perception model also designs a fusion judgment module with adaptive weight allocation; the fusion judgment module dynamically adjusts the weight based on the density of heat source distribution in the scene, thereby avoiding wrong judgment in areas with scarce infrared features or large interference, and ensuring the accuracy of motion state estimation and thermal imaging results.

12. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 4, characterized in that: The scene perception module tracks dynamic targets in real time and processes thermal imaging camera data through a lightweight dynamic target recognition model to achieve real-time perception of dynamic targets; The lightweight dynamic target recognition model analyzes multi-frame information, tracks moving people or equipment, removes noise points, and extracts key dynamic features, thereby improving the system's perception accuracy of dynamic targets and providing real-time dynamic information support.

13. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 12, characterized in that: The lightweight dynamic target recognition model extracts temporal features based on a multi-layer convolutional neural network; The lightweight dynamic target recognition model includes a shallow feature extraction part and a deep feature extraction part; The targets are divided into small targets and large targets. The shallow feature extraction part is used to detect small targets, while the deep feature extraction part is used to identify large targets. An interactive branch differential network is introduced into the lightweight dynamic target recognition model, and dynamic target indexing is performed in time series to further filter out misidentified targets and ensure the accuracy of target recognition and tracking.

14. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 4, characterized in that: The scene perception module detects high and low temperature targets and detects thermal radiation sources in the environment through a temperature analysis model based on thermal imaging camera data, identifying areas with abnormal temperatures; it performs in-depth analysis of infrared imaging data to generate a thermal distribution map, thereby locating the source of the fire or potential rescue targets.

15. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 4, characterized in that: The scene perception module also automatically identifies and marks entrances, exits and floors; the scene perception module automatically identifies and marks entrances, exits, floors and attack points in the environment through a deep model; The deep model automatically generates a spatial model of the building by analyzing the data from the thermal imaging camera, and identifies the locations of entrances, exits, floors, and attack points. The floor recognition and entrance and exit detection algorithms are combined with the inertial navigation system data to determine the floor location in real time.

16. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 15, characterized in that: The scene perception module automatically identifies and marks entrances, exits and floors, and the corresponding sensor module also includes a lidar module and / or a barometer and / or a visible light camera module.

17. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 16, characterized in that: The scene perception module adopts deep neural network representation, and automatically generates a spatial model of the building by analyzing the 3D point cloud data provided by the laser radar module and / or the barometer data and / or the image data captured by the visible light camera module, and identifies the entrances, floors and attack points. The scene perception module adopts a spatial convolutional neural network to extract local features from the laser radar module and / or the visible light camera module data, and further identifies the geometric form of these structures. Furthermore, a graph neural network is used to construct a spatial relationship graph, optimize the topological structure between spatial elements, and thus accurately identify the relative positions of entrances and stairs. The floor recognition and entrance detection algorithm combines the barometer and inertial navigation data, calculates the absolute height of the current floor of the building through deep feature representation, and determines the floor position in real time.

18. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 15, characterized in that: The scene perception module automatically identifies and marks entrances, exits, floors and attack points, and also manually marks or modifies them.

19. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 4, characterized in that: The scene perception module also performs smoke and dust identification, and the corresponding sensor module also includes a laser radar module and / or a visible light camera module; the scene perception module automatically identifies the smoke and dust environment by analyzing the data of the laser radar module and / or the visible light camera module; the laser radar module generates high-precision point cloud data of the surrounding environment by emitting lasers and receiving return signals; the scene perception module uses data preprocessing technology to determine whether there is smoke or dust through the sparsity of the point cloud, changes in reflection intensity and distance changes.

20. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 19, characterized in that: In the smoke and dust recognition of the scene perception module, the corresponding sensor module also includes a visible light camera module; the image data provided by the visible light camera module assists in identifying the environmental characteristics of smoke and dust; through a threshold-based detection method, a machine learning algorithm or a deep learning algorithm, the scene perception module extracts the visual characteristics of smoke or dust from the image, and judges the presence of smoke and dust in a dynamic environment by combining the image's lighting changes, edge detection and color difference characteristics.

21. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 19, characterized in that: In the smoke and dust recognition of the scene perception module, based on historical data and environmental models, the scene perception module gradually optimizes its recognition accuracy, responds in real time and provides accurate environmental perception information in a complex and dynamically changing environment.

22. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 5, characterized in that: The multi-sensor fusion positioning module, the corresponding sensor module also includes a laser radar module, and the multi-sensor fusion positioning module includes specific sub-modules: Data cleaning module: First, pre-process the data of thermal imaging camera, inertial navigation module and lidar module, filter out noise observations through noise suppression model, and synchronize multi-sensor data through sensor synchronization signal to ensure the time synchronization of observation data of different sensors; Odometer positioning module: It fuses multi-source data from thermal imaging cameras, inertial navigation modules, and lidar modules through filtering, optimization, or end-to-end state estimation methods, recursively estimates the device's posture, and builds a multi-sensor fusion odometer; The state estimation model is used to handle nonlinear motion situations in emergency scenarios, thereby ensuring that the device can still provide high-precision real-time positioning even when there is no satellite signal or the signal is interfered with.

23. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 22, characterized in that: The odometer positioning module filters the thermal imaging data collected by the thermal imaging camera, smoothes the stripe noise, and further extracts key radiation features based on the local radiation gradient; after feature extraction, feature association is performed based on the local radiation feature descriptor to ensure robust matching of heat sources in the infrared image; the local radiation feature descriptor is encoded by the local heat source area in the infrared image.

24. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 22, characterized in that: The sensor module corresponding to the multi-sensor fusion positioning module also includes a laser radar module. In order to balance the efficiency and accuracy of laser radar matching, a classification feature matching method is adopted: When the LiDAR data is identified as a surface feature, the point-to-surface distance matching method is used; for non-surface features, the point-to-distribution matching method is used, which uses the low dimensionality of surface features to reduce the amount of calculation. At the same time, the point-to-distribution constraints can also be used on non-surface features to enhance the robustness of the algorithm.

25. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 22, characterized in that: In order to cope with possible degradation phenomena in the environment, an adaptive point cloud downsampling and matching parameter adjustment strategy is introduced to ensure the accuracy and stability of matching when the point cloud density is low or the feature distribution is uneven. In the case of final degradation, a fast normal distribution transformation is used for point cloud registration to ensure that the algorithm can still provide accurate matching and positioning results even under extreme conditions.

26. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 22, characterized in that: The sensor module also includes a satellite navigation module, and the corresponding multi-sensor fusion positioning module includes sub-modules: Repositioning and global state optimization module: Use the global position information and repositioning constraints of the satellite navigation module to correct long-term posture errors and improve the positioning robustness and accuracy of the overall system.

27. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 22, characterized in that: The multi-sensor fusion positioning module also includes sub-modules: Positioning feedback and navigation guidance module: integrates positioning data and environmental perception information to provide real-time navigation guidance for rescue equipment, while supporting relative positioning between multiple machines.

28. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 26, characterized in that: The relocalization and global state optimization module corrects the long-term posture error, specifically including: Absolute position observation fusion of satellite positioning and navigation module: In outdoor scenes, the global position information provided by the satellite positioning and navigation module is used as an absolute position reference to reduce cumulative drift. Based on a multimodal data fusion strategy, the absolute position data of the satellite is used as a soft constraint for low-frequency updates, and is fused with the high-frequency data of the lidar module, thermal imaging camera, and inertial navigation to gradually correct the position. In addition, an adaptive data credibility adjustment mechanism is introduced. When the satellite signal is weak or blocked, the satellite data weight is automatically reduced, and the fusion result of the lidar, thermal imaging camera, and inertial navigation is used. Multi-sensor position re-identification: The corresponding sensor module also includes a visible light camera module; a deep re-localization algorithm is used to fuse multi-sensor data to solve the limitations of traditional re-localization algorithms in complex environments; first, a re-localization feature description algorithm is used to extract global feature descriptors from the LiDAR point cloud and build a historical point cloud database; candidate matching frames of the current point cloud in the historical database are retrieved and determined; then multi-modal matching of infrared images and / or visible light images is introduced, and a similarity model is used to calculate the matching similarity between the current image and the candidate image, and the candidate frame sorting is optimized; in smoke or low-light environments, the scene state is automatically judged through the scene perception module, and infrared image data is used first to ensure the robustness of the re-localization algorithm; after determining the re-localization constraint identifier, point cloud registration is performed to generate the re-localization relative pose as the input for global pose graph optimization; Global pose graph optimization: Combine the state estimation method of multi-sensor fusion, the generated repositioning relative pose, and the absolute position observation provided by the satellite navigation system to construct and optimize the global pose graph; by solving the optimization problem, obtain the optimal estimate of the pose at all historical moments, update the historical frame sub-map and the global map, eliminate long-term accumulated errors, and achieve globally consistent high-precision mapping.

29. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 27, characterized in that: The positioning feedback and navigation guidance module integrates positioning data and environmental perception information to provide real-time navigation guidance for rescue equipment and support relative positioning between multiple machines, including: Through collaborative computing and communication between devices, combined with decision-making algorithms, accurate relative posture determination and path optimization are achieved to build an efficient multi-device collaborative network; the decision-making algorithm analyzes multi-sensor data in real time, evaluates environmental complexity, dynamic obstacle distribution, and equipment operating status, and provides intelligent support for navigation guidance; for different scenarios, the decision algorithm dynamically adjusts the navigation strategy, and prioritizes obstacle avoidance paths in scenes with dense obstacles or smoke; in life detection tasks, the device collaborative path is optimized to quickly reach the target location.

30. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 1, characterized in that: The equipment also includes a communication module to achieve real-time communication between the device and the control center. The positioning information is fed back to the control center in real time through the communication module, which conducts a comprehensive analysis of the global information, generates real-time scheduling suggestions, and optimizes the task allocation and collaboration paths between multiple devices.

31. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 30, characterized in that: Multiple devices communicate with each other through Lora, 4G or WiFi. When the 4G and WiFi signals are poor, they communicate with each other through Lora and calculate the floor difference and 2D horizontal distance between each other.

32. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 30, characterized in that: The real-time dispatch suggestions generated by the control center also include dispatching personnel from other equipment to rescue when a problem occurs to personnel of a certain device and they are trapped.

33. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 30, characterized in that: The equipment may be one or more devices, and each device communicates with the control center and the server in real time through a relay base station.

34. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 1, characterized in that: The data fusion mapping module specifically includes: Single-machine data fusion and local two-dimensional temperature occupancy map construction: On the single-machine side, the thermal imaging data captured by the thermal imaging camera and the local point cloud generated by the inertial navigation system are integrated to realize the construction of a two-dimensional temperature occupancy map using an algorithm model. The two-dimensional temperature occupancy map represents the spatial structure and temperature distribution information around the equipment in a rasterized form, showing fire sources, living things and high-temperature hazardous areas. The algorithm model analyzes infrared thermal images and point cloud data in real time, intelligently filters out noise and optimizes the grid update rate to ensure the accuracy and real-time nature of map construction. In a dynamic environment, the algorithm model automatically adjusts the perception weight according to environmental changes.

35. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 34, characterized in that: The sensor module corresponding to the data fusion mapping module also includes a lidar module. Under smoke and dust-free conditions, the thermal imaging data captured by the thermal imaging camera and the local point cloud generated by the inertial navigation system and the lidar module data are integrated to realize the construction of a two-dimensional temperature occupancy map using an algorithm model.

36. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 3, characterized in that: The server-side multi-machine global three-dimensional ambient temperature map construction specifically includes: Server-side multi-machine global three-dimensional environmental temperature mapping and dynamic update: The server-side integrates the sensor data of multiple devices and uses scene re-identification and multi-machine relocalization technology to build a globally consistent three-dimensional fusion environment map, dynamically update the static structure and dynamic targets in the three-dimensional map, and realize fine modeling of the global environment; the scene re-identification algorithm analyzes infrared thermal imaging data from multiple devices to identify and match similar scene areas, thereby aligning the observation data between different devices; the multi-machine relocalization detection technology combines the global feature descriptors generated by deep learning to retrieve similar scenes in the historical point cloud database, and determines the relocalization matching frames through further verification of infrared images; the server uses the fused relocalization constraint information to globally optimize the local maps of each device, eliminate the drift accumulation error through posture adjustment, and generate a global three-dimensional map.

37. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 36, characterized in that: The sensor modules corresponding to the server-side multi-machine global three-dimensional ambient temperature mapping and dynamic update also include lidar modules and visible light camera modules: in the specific steps, the scene re-identification algorithm analyzes the multi-sensor data of lidar point clouds, infrared thermal images and visible light images from multiple devices, identifies and matches similar scene areas, and realizes the alignment of observation data between different devices; the multi-machine relocation detection technology is combined with the global feature descriptor generated by deep learning, retrieves similar scenes in the historical point cloud database, and determines the relocation matching frame through further verification of visible light images and infrared images.

38. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 36, characterized in that: The server-side multi-machine global three-dimensional ambient temperature map construction specifically includes: Precise segmentation of remote 3D maps: The server-side global 3D environment map is accurately segmented through a segmentation algorithm, and the objects in the scene are accurately divided into at least one of the following elements: entrances and exits, walls, ground, obstacles, and living things. The segmentation model uses fused multi-source data and combines global scene features to achieve semantic segmentation and object recognition.

39. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 36, characterized in that: The server-side multi-machine global three-dimensional ambient temperature map construction specifically includes: 3D rendering of temperature map: The remote 3D environmental map is further combined with the thermal imaging data of the thermal imaging camera to achieve 3D rendering of the temperature map through the temperature mapping algorithm. The real-time temperature information is superimposed on the 3D point cloud to intuitively display the temperature distribution of different areas in the scene, including the specific location and temperature change trend of at least one of the attack point, fire source, heat source and life form.

40. The multi-sensor fusion positioning mapping and thermal imaging equipment according to claim 1, characterized in that: The equipment adopts end-to-cloud integrated cluster collaboration and multi-link stable transmission technology: it adopts end-to-cloud data processing and cluster collaboration technology, supports multiple signal transmission methods and relay communication modes between end and cloud.

41. The multi-sensor fusion positioning mapping and thermal imaging equipment according to any one of claims 1 to 40, characterized in that: The equipment is wearable equipment.

42. The multi-sensor fusion positioning mapping and thermal imaging equipment according to any one of claims 1 to 40, characterized in that: The device is a handheld device.

43. A computer readable medium, characterized in that At least one computer program is stored in the computer-readable medium, and the at least one computer program is loaded and executed by the processor to enable the electronic device to implement the working method of the equipment according to any one of claims 1 to 40.

44. A fire emergency robot, characterized in that: Equipped with the multi-sensor fusion positioning, mapping and thermal imaging equipment according to any one of claims 1-40.

45. A fire emergency robot dog, characterized in that: Equipped with the multi-sensor fusion positioning, mapping and thermal imaging equipment according to any one of claims 1-40.

46. ​​A fire emergency drone, characterized in that: Equipped with the multi-sensor fusion positioning, mapping and thermal imaging equipment according to any one of claims 1-40.

Images