Target intelligent positioning control system and method based on infrared imaging, laser radar and sound directional detection

Abstract

The present invention relates to a target intelligent positioning control system based on infrared imaging, laser radar, and sound directional detection, wherein the system includes an infrared imaging module, a laser radar module, a sound directional detection module, a radio detection module, a gas detection module, a three-dimensional mapping and perception module, a three-dimensional fusion and processing module, a target positioning and tracking module, and a decision-making and early warning module. The present invention also relates to a corresponding method. The target intelligent positioning control system based on infrared imaging, laser radar, and sound directional detection and its method of the present invention maps each frame of three-dimensional point infrared imaging data, microphone sound source probability distribution, and wireless and gas detection information into a probability distribution of objects to be rescued in a three-dimensional scene, providing three-dimensional fusion-based environmental detection and object positioning capabilities, effectively and scientifically improving rescue and reconnaissance efficiency, significantly enhancing the efficiency and accuracy of locating trapped personnel, and having a relatively wide range of applications.

Description

Technical Field

[0001] The present invention relates to the field of emergency rescue technology, in particular to the field of data fusion and intelligent positioning technology, and specifically refers to a target intelligent positioning control system and method based on infrared imaging, laser radar and sound directional detection. Background Art

[0002] In the emergency rescue industry, accurately and quickly detecting and locating the target is crucial. For various reasons, trapped individuals may be in deep mountains, narrow valleys, caves, underground buildings, or even buried in ruins, mudslides, and other disaster sites. Accurately and quickly detecting and locating the target is crucial to the success of the rescue operation. However, due to complex environments, difficulty in detection, and insufficient on-site information, detecting and locating the target presents a significant challenge and bottleneck in the rescue industry.

[0003] Currently, several technical solutions exist for detecting and locating objects to be rescued. Traditional robotic detection technology suffers from poor environmental adaptability, low detection accuracy, and short operating times. While visual sensor-based detection technology offers certain advantages, it suffers from poor detection effectiveness in harsh environments like low illumination and haze. Sound-based detection technology can be applied in environments like deep mountains and narrow valleys, but suffers from complex signals and low accuracy. Radio and gas detection technologies, on the other hand, face limitations such as noise interference and short detection distances. Therefore, existing single-sensor positioning technologies struggle to comprehensively address the challenges of detecting and locating objects to be rescued in the emergency rescue industry. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a target intelligent positioning control system and method based on infrared imaging, laser radar and sound directional detection with high precision, high speed and high stability.

[0005] In order to achieve the above objectives, the target intelligent positioning control system and method based on infrared imaging, laser radar and sound directional detection of the present invention are as follows:

[0006] The main features of the target intelligent positioning control system based on infrared imaging, laser radar and sound directional detection are as follows:

[0007] The LiDAR module uses a high-precision laser rangefinder to scan the target and measure the distance, angle, and height information of surrounding points in real time. At the same time, it combines the relevant information obtained from the scan with the point cloud information processing module to process it.

[0008] The infrared imaging module uses a high-sensitivity infrared sensor to capture the target thermal radiation characteristics under the current rescue mission and generate high-resolution infrared image data; and uses the infrared image processing module to perform corresponding processing on the generated infrared image data.

[0009] The sound directional detection module captures the sound signals emitted by the target through a high-sensitivity microphone array, and determines the direction of the sound target and the probability distribution of the sound source through the signal processing algorithm in the sound signal processing module;

[0010] The radio detection module detects the target's radio signal through a high-sensitivity antenna and radio frequency receiver, and then identifies and locates the wireless communication device through the radio signal processing module;

[0011] The gas detection module monitors the gas composition around the target through a highly sensitive gas sensor, and detects possible harmful gases or gases that are indicative of the target through a gas detection processing module;

[0012] The three-dimensional modeling and perception module is connected to the laser radar module, and is also equipped with a laser radar data processing unit and a multi-source fusion pose solving unit, which are used to obtain the three-dimensional model and scene information of the monitored target and provide stereoscopic perception capabilities for target positioning; among them, the multi-source fusion pose solving unit achieves accurate pose estimation of the detection equipment or detection robot by fusing information from different data sources (such as laser radar, infrared camera, ultrasound, wireless detection, etc.), thereby improving the accuracy and robustness of three-dimensional modeling.

[0013] A three-dimensional fusion and processing module is connected to the three-dimensional modeling and perception module, infrared imaging module, sound directional detection module, radio detection module, and gas detection module. The module is internally provided with an infrared imaging data processing unit, a sound directional detection data processing unit, a wireless detection data processing unit, a gas detection data processing unit, a three-dimensional neural network probability field construction unit, and a sensor data fusion unit. By adopting a multi-sensor data fusion algorithm, the module performs three-dimensional fusion and processing on the data collected by the infrared imaging module, the lidar module, the sound directional detection module, the three-dimensional modeling and perception module, the radio detection module, and the gas detection module to improve the positioning accuracy and robustness of the target;

[0014] Among them, the three-dimensional neural network probability field construction unit uses deep learning technology to fuse multi-sensor data into a unified three-dimensional neural network probability field to effectively identify and locate targets and improve the accuracy and reliability of target positioning.

[0015] A target positioning and tracking module, connected to the 3D fusion and processing module, is used to locate and track the target in real time based on the target position information output by the 3D fusion and processing module to ensure that the target is not lost; and

[0016] The decision-making and early warning module is connected to the target positioning and tracking module, and is used to combine the data information output by the three-dimensional fusion and processing module and the target positioning and tracking module, and perform target behavior analysis and prediction. By formulating detection strategies and execution plans, it can provide real-time early warning prompts, potential risk assessments, and response measures.

[0017] The method for realizing intelligent target positioning control based on infrared imaging, laser radar and sound directional detection by using the above system is characterized in that the method comprises the following steps:

[0018] (1) Use the LiDAR module to obtain point cloud information of the surrounding environment in real time;

[0019] (2) The three-dimensional modeling and perception module uses the point cloud information obtained by the laser radar module to perform real-time three-dimensional modeling;

[0020] (3) Based on the trajectory information obtained by the laser radar module, a three-dimensional point cloud structure of the current environment is generated in real time and incrementally, and the sensor data of the infrared imaging module, the sound directional detection module, the radio detection module and the gas detection module received synchronously are fused and positioned and parameter calculated using the three-dimensional fusion and processing module;

[0021] (4) transmitting the calculated received data of the infrared imaging module, the sound directional detection module, the radio detection module, and the gas detection module to the three-dimensional fusion and processing module for fusion processing;

[0022] (5) The three-dimensional fusion and processing module calculates the current target's position and probability distribution based on the sensor data and the three-dimensional point cloud structure information provided by the three-dimensional modeling and perception module;

[0023] (6) The target positioning and tracking module performs real-time positioning and tracking processing based on the target position information and probability distribution output by the three-dimensional fusion and processing module to ensure that the target is not lost;

[0024] (7) The decision-making and warning module makes decisions based on the three-dimensional environmental information and target position information output by the three-dimensional fusion and processing module and the target positioning and tracking module, gives subsequent operation suggestions or conducts autonomous detection, and issues warnings in dangerous moments.

[0025] Preferably, the step (2) specifically includes the following steps:

[0026] (2.1) The detection equipment or detection robot emits a laser beam through a laser radar and measures its echo reflection time to obtain the location information of the target object in the current environment, including the time, angle, distance, and reflection intensity of the reflected laser beam, and calculates single-frame three-dimensional point cloud data, wherein the single-frame three-dimensional point cloud data includes the three-dimensional coordinates (x, y, z) and reflection time of each point, and uses this to establish a local three-dimensional point cloud map;

[0027] (2.2) Utilizing the data information acquired by the laser radar, extracting feature points from the three-dimensional point cloud of the laser radar, performing feature matching on the acquired point cloud data at adjacent moments, finding identical feature points, thereby generating a feature matching list, and recording the reference frame feature ID, target frame feature ID, and feature matching confidence of each feature point, thereby incrementally updating the three-dimensional motion pose trajectory of the detection equipment or detection robot in real time;

[0028] (2.3) Based on the acquired three-dimensional motion posture trajectory, the three-dimensional point cloud data of each frame are converted into the robot coordinate system, merged into a point cloud data set for voxel filtering, and the point cloud map is continuously updated according to the motion trajectory of the detection equipment or detection robot to achieve real-time modeling of the environment.

[0029] Preferably, the step (3) specifically includes the following steps:

[0030] (3.1) Pre-calibrate the relative displacements and matrices between the lidar and the infrared imaging, microphone array, radio detection, and gas detection sensors to form extrinsic parameters, which are represented as a 4×4 floating-point matrix.

[0031] (3.2) receiving synchronized infrared imaging data, microphone array data, radio detection data, and gas detection sensor data;

[0032] (3.3) Calculating infrared data information of the three-dimensional position within the visible range based on the motion posture and three-dimensional point cloud structure information of the currently described detection equipment or detection robot;

[0033] (3.3.1) Obtaining infrared imaging data: Obtaining a frame of infrared imaging data from the infrared imaging sensor;

[0034] (3.3.2) Extract corresponding lidar data: Extract the corresponding infrared imaging data from the lidar data according to the timestamp.

[0035] According to the corresponding single-frame three-dimensional point cloud data;

[0036] (3.3.3) Point cloud registration: Based on the external parameter matrix of the infrared imaging sensor and the lidar, the extracted lidar data and the infrared imaging data are registered, that is, the two data are aligned and the infrared imaging data is mapped to the point cloud.

[0037] Shoot into the three-dimensional point cloud data at the corresponding position;

[0038] (3.3.4) Point cloud fusion: Fuse the registered infrared imaging data with the original 3D point cloud data to form

[0039] New 3D point cloud data;

[0040] (3.4) Calculate the probability distribution of possible sound sources in the three-dimensional environment based on the motion posture and signal direction information of the current detection equipment or detection robot;

[0041] (3.4.1) Acquiring sound source data: Acquire sound wave intensity and sound source direction angle data from the microphone array;

[0042] (3.4.2) Convert to the robot coordinate system: According to the external parameter matrix of the microphone array and the lidar, the sound source direction angle data is converted to the sound source direction in the robot coordinate system. The direction contains two data: the three-dimensional starting point coordinates, the three-dimensional

[0043] direction vector;

[0044] (3.4.3) Fusion and storage of data: Starting from the sound source in the robot coordinate system, simulate the direction of sound wave propagation and record the sound wave probability. Acquire the first point along the direction vector and calculate the reflection direction along the normal vector of the first point. Acquire the next point from the reflection direction. Repeat this step until a list of 3D points within a certain reflection distance is obtained. Weights are assigned according to distance and sound source probability data is assigned to the 3D points in the list.

[0045] (3.4.4) Point cloud fusion: write the fused point cloud data back into the 3D point cloud information;

[0046] (3.5) recording the acquired radio and gas detection data to the location of the detection equipment or detection robot according to the current motion posture of the detection equipment or detection robot;

[0047] (3.5.1) Acquire radio and gas detection sensor data;

[0048] (3.5.2) Extract the radar pose corresponding to the current time from the lidar data based on the timestamp;

[0049] (3.5.3) Create a 3D marker based on the radar pose and record the wireless and gas detection sensors

[0050] data;

[0051] (3.5.4) Point cloud fusion: Write the fused point cloud data back into the 3D point cloud information.

[0052] Preferably, the step (4) specifically includes the following steps:

[0053] (4.1) Preprocessing the received data from the infrared imaging module, the sound directional detection module, the radio detection module, and the gas detection module, including aligning the data from different sensors according to timestamps and performing denoising and filtering on the raw data;

[0054] (4.2) Constructing a signed distance field based on the shortest path from the target point to each environmental point: Input the 3D point cloud information obtained through 3D modeling into the neural network for clustering, segmentation, and feature extraction. SDF features are calculated and the signed distance field is calculated through voxel sampling.

[0055] (4.3) Based on the pre-processed sensor data, establish the infrared radiation probability field, audio neural radiation probability field, radio neural radiation probability field and gas neural density probability field respectively;

[0056] (4.4) Use the back-propagation algorithm to train the multi-layer perceptron MLP to fuse the different probability fields.

[0057] Preferably, the step (4.3) of establishing the infrared radiation probability field specifically includes:

[0058] Sensor platform pose calculation: Obtain the pose of the sensor platform in the current environment and the intrinsic and extrinsic parameters of infrared imaging, and project them through a universal camera model to obtain the correspondence between the two-dimensional camera and the three-dimensional environment;

[0059] Two-dimensional infrared image projection: each pixel on the two-dimensional infrared image is ray-projected and passes through the space of the signed distance field;

[0060] Neural network optimization: Use the multi-layer perceptron (MLP) neural network to iteratively optimize and regress the currently obtained projection function to obtain the infrared neural network radiation field;

[0061] Volume density sampling: sampling the infrared neural network radiation field in blocks and obtaining the volume density at each point;

[0062] Infrared image rendering: A camera model is built at a preset position and projected. All volume density values on the ray are integrated and accumulated, and then rendered to obtain an infrared image.

[0063] The step (4.3) of establishing the audio neural radiation probability field specifically includes:

[0064] Sound signal processing: Use microphone arrays to collect sound signals in the environment and pre-process the sound signals;

[0065] Sound wave propagation and reflection model: Based on the sound wave propagation and reflection model, the audio neural radiation field is established, and the sound source position and sound wave arrival time are calculated using the sound wave's acoustic path and three-dimensional environmental information;

[0066] Neural network optimization: The audio radiation field is iteratively optimized and regressed through the multi-layer perceptron (MLP) neural network to obtain the final audio neural radiation probability field.

[0067] Preferably, the step (4.3) of establishing the radio neural radiation probability field specifically includes:

[0068] Radio signal processing: Use wireless sensors to collect radio signals in the environment and pre-process the signals;

[0069] Radio signal propagation model: Based on the propagation, reflection, and penetration models of radio signals of different frequencies, the radio neural radiation field is established and the signal propagation path and signal arrival time are calculated;

[0070] Neural network optimization: The radio radiation field is iteratively optimized and regressed through the multi-layer perceptron (MLP) neural network to obtain the final radio neural radiation probability field;

[0071] The step (4.3) of establishing the gas density probability field specifically includes:

[0072] Gas information processing: Use gas detectors to collect gas information in the environment and pre-process the gas information;

[0073] Gas diffusion model: Based on the gas diffusion model, the gas density field is established to calculate the gas density and expected distribution;

[0074] Neural network optimization: The gas density field is iteratively optimized and regressed through the multi-layer perceptron (MLP) neural network to obtain the final gas neural radiation probability field.

[0075] Preferably, the step (4.4) specifically includes the following steps:

[0076] (4.4.1) Input layer: takes the data of different probability fields as the input of the neural network;

[0077] (4.4.2) Hidden Layer: The correlation between different probability fields is learned through multiple hidden layers, where the number of neurons and activation functions in the hidden layers are adjusted according to the specific task and dataset;

[0078] (4.4.3) Output layer: Outputs the fused three-dimensional neural network probability field. The number of neurons in the output layer should be the same as that in the input layer to ensure that the output probability field has the same dimension.

[0079] (4.4.4) Weight Fusion: During the training process, the weights of the neural network are adjusted according to the prediction error. Through weight fusion, the neural network learns the correlation between different probability fields, thereby achieving effective fusion.

[0080] Preferably, the step (5) specifically includes the following steps:

[0081] (5.1) Using a single sensor, calculate the probability value of each point in the 3D point cloud structure corresponding to the infrared radiation probability field. When the probability value is greater than a preset threshold, the point is considered to be the location of the object to be rescued;

[0082] (5.2) Using multi-sensor fusion processing to combine the probability fields of different sensors to calculate the probability distribution;

[0083] (5.2.1) Voxel rendering: voxelize the current 3D model and convert the point cloud data in each voxel into

[0084] A format that can be processed by the neural network;

[0085] (5.2.2) Ray projection: The radiation field of each sensor obtained after the neural network processing is

[0086] Perform ray projection and calculate the intersection of each ray and voxel;

[0087] (5.2.3) Cumulative probability: For each intersection of a ray and a voxel, calculate the neural network radiation probability of the corresponding sensor.

[0088] The probability values in the shooting field are accumulated to obtain the fusion probability;

[0089] (5.2.4) Judgment: Set a threshold. When the fusion probability is greater than the threshold, the point is considered to be the location of the object to be rescued.

[0090] Preferably, the step (7) specifically includes the following steps:

[0091] (7.1) Risk Analysis: Based on the input 3D environmental information, the system conducts a comprehensive analysis of the various sensor data obtained to analyze the possible risks in the environment and identify risk areas.

[0092] (7.2) Path planning: Analyze the currently generated 3D environment model and calculate the shortest or safest path to the target location by combining risk area information;

[0093] (7.3) Target tracking: Based on the target location information provided by the target positioning and tracking module, the target is tracked in real time and the target position in the three-dimensional environment is updated in real time;

[0094] (7.4) Early warning: Based on the risk analysis results, corresponding early warning information is given in real time.

[0095] The target intelligent positioning control system and method based on infrared imaging, laser radar, and sound directional detection of the present invention are used. The laser radar emits a laser beam and measures its reflection time to obtain the position information of objects in the environment. At the same time, the real-time incremental establishment of a three-dimensional point cloud map technology solution can generate the three-dimensional coordinates of the object position in real time during the movement of the detection equipment or detection robot, thereby establishing a three-dimensional point cloud map of the environment. This technical solution has the advantages of high precision, high speed, and high stability. It can be applied to positioning and detection in various environments, effectively improve the efficiency of rescue and investigation, and reduce the difficulty of accurately locating the object to be rescued, thereby efficiently and scientifically improving the efficiency of rescue and investigation, significantly enhancing the efficiency and accuracy of locating trapped personnel, and has a wider range of applications compared to existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 The present invention is a flow chart of a target intelligent positioning control system and method based on infrared imaging, laser radar and sound directional detection.

[0097] Figure 2 This is a structural diagram of the target intelligent positioning control system based on infrared imaging, laser radar and sound directional detection of the present invention. DETAILED DESCRIPTION

[0098] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0099] Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, whereby a process, method, article, or apparatus comprising a list of elements includes not only those elements, but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0100] See also Figure 1 and Figure 2 As shown, the target intelligent positioning control system based on infrared imaging, laser radar and sound directional detection, wherein the system includes:

[0101] The LiDAR module uses a high-precision laser rangefinder to scan the target and measure the distance, angle, and height information of surrounding points in real time;

[0102] The infrared imaging module uses a highly sensitive infrared sensor to capture the target's thermal radiation characteristics during the current rescue mission and generate high-resolution infrared image data;

[0103] The sound directional detection module uses a high-sensitivity microphone array to capture the sound signals emitted by the target and uses a signal processing algorithm to determine the direction of the sound target and the probability distribution of the sound source;

[0104] The radio detection module detects the target's radio signal through a high-sensitivity antenna and radio frequency receiver to identify and locate wireless communication devices;

[0105] The gas detection module monitors the gas composition around the target through a highly sensitive gas sensor to detect possible harmful gases or gases that are indicative of the target;

[0106] A three-dimensional modeling and perception module, connected to the lidar module, is used to obtain a three-dimensional model and scene information of the monitored target, providing stereoscopic perception capabilities for target positioning;

[0107] a three-dimensional fusion and processing module connected to the three-dimensional modeling and perception module, infrared imaging module, sound directional detection module, radio detection module, and gas detection module, and performing three-dimensional fusion and processing on the data collected by the infrared imaging module, lidar module, sound directional detection module, three-dimensional modeling and perception module, radio detection module, and gas detection module by adopting a multi-sensor data fusion algorithm, so as to improve the positioning accuracy and robustness of the target;

[0108] A target positioning and tracking module, connected to the 3D fusion and processing module, is used to locate and track the target in real time based on the target position information output by the 3D fusion and processing module to ensure that the target is not lost; and

[0109] The decision-making and early warning module is connected to the target positioning and tracking module, and is used to combine the data information output by the three-dimensional fusion and processing module and the target positioning and tracking module, and perform target behavior analysis and prediction. By formulating detection strategies and execution plans, it can provide real-time early warning prompts, potential risk assessments, and response measures.

[0110] The system is used to implement a target intelligent positioning control method based on infrared imaging, laser radar and sound directional detection, wherein the method comprises the following steps:

[0111] (1) Use the LiDAR module to obtain point cloud information of the surrounding environment in real time;

[0112] (2) The three-dimensional modeling and perception module uses the point cloud information obtained by the laser radar module to perform real-time three-dimensional modeling;

[0113] (3) Based on the trajectory information obtained by the laser radar module, a three-dimensional point cloud structure of the current environment is generated in real time and incrementally, and the sensor data of the infrared imaging module, the sound directional detection module, the radio detection module and the gas detection module received synchronously are fused and positioned and parameter calculated using the three-dimensional fusion and processing module;

[0114] (4) transmitting the calculated received data of the infrared imaging module, the sound directional detection module, the radio detection module, and the gas detection module to the three-dimensional fusion and processing module for fusion processing;

[0115] (5) The three-dimensional fusion and processing module calculates the current target's position and probability distribution based on the sensor data and the three-dimensional point cloud structure information provided by the three-dimensional modeling and perception module;

[0116] (6) The target positioning and tracking module performs real-time positioning and tracking processing based on the target position information and probability distribution output by the three-dimensional fusion and processing module to ensure that the target is not lost;

[0117] (7) The decision-making and warning module makes decisions based on the three-dimensional environmental information and target position information output by the three-dimensional fusion and processing module and the target positioning and tracking module, gives subsequent operation suggestions or conducts autonomous detection, and issues warnings in dangerous moments.

[0118] As a preferred embodiment of the present invention, the step (2) specifically includes the following steps:

[0119] (2.1) The detection equipment or detection robot emits a laser beam through a laser radar and measures its echo reflection time to obtain the location information of the target object in the current environment, including the time, angle, distance, and reflection intensity of the reflected laser beam, and calculates single-frame three-dimensional point cloud data, wherein the single-frame three-dimensional point cloud data includes the three-dimensional coordinates (x, y, z) and reflection time of each point, and uses this to establish a local three-dimensional point cloud map;

[0120] (2.2) Utilizing the data information acquired by the laser radar, extracting feature points from the three-dimensional point cloud of the laser radar, performing feature matching on the acquired point cloud data at adjacent moments, finding identical feature points, thereby generating a feature matching list, and recording the reference frame feature ID, target frame feature ID, and feature matching confidence of each feature point, thereby incrementally updating the three-dimensional motion pose trajectory of the detection equipment or detection robot in real time;

[0121] (2.3) Based on the acquired three-dimensional motion posture trajectory, the three-dimensional point cloud data of each frame are converted into the robot coordinate system, merged into a point cloud data set for voxel filtering, and the point cloud map is continuously updated according to the motion trajectory of the detection equipment or detection robot to achieve real-time modeling of the environment.

[0122] As a preferred embodiment of the present invention, the step (3) specifically includes the following steps:

[0123] (3.1) Pre-calibrate the relative displacements and matrices between the lidar and the infrared imaging, microphone array, radio detection, and gas detection sensors to form extrinsic parameters, which are represented as a 4×4 floating-point matrix.

[0124] (3.2) receiving synchronized infrared imaging data, microphone array data, radio detection data, and gas detection sensor data;

[0125] (3.3) Calculating infrared data information of the three-dimensional position within the visible range based on the motion posture and three-dimensional point cloud structure information of the currently described detection equipment or detection robot;

[0126] (3.3.1) Obtaining infrared imaging data: Obtaining a frame of infrared imaging data from the infrared imaging sensor;

[0127] (3.3.2) Extract corresponding lidar data: Extract the corresponding infrared imaging data from the lidar data according to the timestamp.

[0128] According to the corresponding single-frame three-dimensional point cloud data;

[0129] (3.3.3) Point cloud registration: Based on the external parameter matrix of the infrared imaging sensor and the lidar, the extracted lidar data and the infrared imaging data are registered, that is, the two data are aligned and the infrared imaging data is mapped to the point cloud.

[0130] Shoot into the three-dimensional point cloud data at the corresponding position;

[0131] (3.3.4) Point cloud fusion: Fuse the registered infrared imaging data with the original 3D point cloud data to form

[0132] New 3D point cloud data;

[0133] (3.4) Calculate the probability distribution of possible sound sources in the three-dimensional environment based on the motion posture and signal direction information of the current detection equipment or detection robot;

[0134] (3.4.1) Acquiring sound source data: Acquire sound wave intensity and sound source direction angle data from the microphone array;

[0135] (3.4.2) Convert to the robot coordinate system: According to the external parameter matrix of the microphone array and the lidar, the sound source direction angle data is converted to the sound source direction in the robot coordinate system. The direction contains two data: the three-dimensional starting point coordinates, the three-dimensional

[0136] direction vector;

[0137] (3.4.3) Fusion and storage of data: Starting from the sound source in the robot coordinate system, simulate the direction of sound wave propagation and record the sound wave probability. Acquire the first point along the direction vector and calculate the reflection direction along the normal vector of the first point. Acquire the next point from the reflection direction. Repeat this step until a list of 3D points within a certain reflection distance is obtained. Weights are assigned according to distance and sound source probability data is assigned to the 3D points in the list.

[0138] (3.4.4) Point cloud fusion: write the fused point cloud data back into the 3D point cloud information;

[0139] (3.5) recording the acquired radio and gas detection data to the location of the detection equipment or detection robot according to the current motion posture of the detection equipment or detection robot;

[0140] (3.5.1) Acquire radio and gas detection sensor data;

[0141] (3.5.2) Extract the radar pose corresponding to the current time from the lidar data based on the timestamp;

[0142] (3.5.3) Create a 3D marker based on the radar pose and record the wireless and gas detection sensors

[0143] data;

[0144] (3.5.4) Point cloud fusion: Write the fused point cloud data back into the 3D point cloud information.

[0145] As a preferred embodiment of the present invention, the step (4) specifically includes the following steps:

[0146] (4.1) Preprocessing the received data from the infrared imaging module, the sound directional detection module, the radio detection module, and the gas detection module, including aligning the data from different sensors according to timestamps and performing denoising and filtering on the raw data;

[0147] (4.2) Constructing a signed distance field based on the shortest path from the target point to each environmental point: Input the 3D point cloud information obtained through 3D modeling into the neural network for clustering, segmentation, and feature extraction. SDF features are calculated and the signed distance field is calculated through voxel sampling.

[0148] (4.3) Based on the pre-processed sensor data, establish the infrared radiation probability field, audio neural radiation probability field, radio neural radiation probability field and gas neural density probability field respectively;

[0149] (4.4) Use the back-propagation algorithm to train the multi-layer perceptron MLP to fuse the different probability fields.

[0150] As a preferred embodiment of the present invention, the step (4.3) of establishing the infrared radiation probability field specifically includes:

[0151] Sensor platform pose calculation: Obtain the pose of the sensor platform in the current environment and the intrinsic and extrinsic parameters of infrared imaging, and project them through a universal camera model to obtain the correspondence between the two-dimensional camera and the three-dimensional environment;

[0152] Two-dimensional infrared image projection: each pixel on the two-dimensional infrared image is ray-projected and passes through the space of the signed distance field;

[0153] Neural network optimization: Use the multi-layer perceptron (MLP) neural network to iteratively optimize and regress the currently obtained projection function to obtain the infrared neural network radiation field;

[0154] Volume density sampling: sampling the infrared neural network radiation field in blocks and obtaining the volume density at each point;

[0155] Infrared image rendering: A camera model is built at a preset position and projected. All volume density values on the ray are integrated and accumulated, and then rendered to obtain an infrared image.

[0156] The step (4.3) of establishing the audio neural radiation probability field specifically includes:

[0157] Sound signal processing: Use microphone arrays to collect sound signals in the environment and pre-process the sound signals;

[0158] Sound wave propagation and reflection model: Based on the sound wave propagation and reflection model, the audio neural radiation field is established, and the sound source position and sound wave arrival time are calculated using the sound wave's acoustic path and three-dimensional environmental information;

[0159] Neural network optimization: The audio radiation field is iteratively optimized and regressed through the multi-layer perceptron (MLP) neural network to obtain the final audio neural radiation probability field.

[0160] As a preferred embodiment of the present invention, the step (4.3) of establishing the radio neural radiation probability field specifically includes:

[0161] Radio signal processing: Use wireless sensors to collect radio signals in the environment and pre-process the signals;

[0162] Radio signal propagation model: Based on the propagation, reflection, and penetration models of radio signals of different frequencies, the radio neural radiation field is established and the signal propagation path and signal arrival time are calculated;

[0163] Neural network optimization: The radio radiation field is iteratively optimized and regressed through the multi-layer perceptron (MLP) neural network to obtain the final radio neural radiation probability field;

[0164] The step (4.3) of establishing the gas density probability field specifically includes:

[0165] Gas information processing: Use gas detectors to collect gas information in the environment and pre-process the gas information;

[0166] Gas diffusion model: Based on the gas diffusion model, the gas density field is established to calculate the gas density and expected distribution;

[0167] Neural network optimization: The gas density field is iteratively optimized and regressed through the multi-layer perceptron (MLP) neural network to obtain the final gas neural radiation probability field.

[0168] As a preferred embodiment of the present invention, the step (4.4) specifically includes the following steps:

[0169] (4.4.1) Input layer: takes the data of different probability fields as the input of the neural network;

[0170] (4.4.2) Hidden Layer: The correlation between different probability fields is learned through multiple hidden layers, where the number of neurons and activation functions in the hidden layers are adjusted according to the specific task and dataset;

[0171] (4.4.3) Output layer: Outputs the fused three-dimensional neural network probability field. The number of neurons in the output layer should be the same as that in the input layer to ensure that the output probability field has the same dimension.

[0172] (4.4.4) Weight Fusion: During the training process, the weights of the neural network are adjusted according to the prediction error. Through weight fusion, the neural network learns the correlation between different probability fields, thereby achieving effective fusion.

[0173] As a preferred embodiment of the present invention, the step (5) specifically includes the following steps:

[0174] (5.1) Using a single sensor, calculate the probability value of each point in the 3D point cloud structure corresponding to the infrared radiation probability field. When the probability value is greater than a preset threshold, the point is considered to be the location of the object to be rescued;

[0175] (5.2) Using multi-sensor fusion processing to combine the probability fields of different sensors to calculate the probability distribution;

[0176] (5.2.1) Voxel rendering: voxelize the current 3D model and convert the point cloud data in each voxel into

[0177] A format that can be processed by the neural network;

[0178] (5.2.2) Ray projection: The radiation field of each sensor obtained after the neural network processing is

[0179] Perform ray projection and calculate the intersection of each ray and voxel;

[0180] (5.2.3) Cumulative probability: For each intersection of a ray and a voxel, calculate the neural network radiation probability of the corresponding sensor.

[0181] The probability values in the shooting field are accumulated to obtain the fusion probability;

[0182] (5.2.4) Judgment: Set a threshold. When the fusion probability is greater than the threshold, the point is considered to be the location of the object to be rescued.

[0183] As a preferred embodiment of the present invention, the step (7) specifically includes the following steps:

[0184] (7.1) Risk Analysis: Based on the input 3D environmental information, the system conducts a comprehensive analysis of the various sensor data obtained to analyze the possible risks in the environment and identify risk areas.

[0185] (7.2) Path planning: Analyze the currently generated 3D environment model and calculate the shortest or safest path to the target location by combining it with risk area information;

[0186] (7.3) Target tracking: Based on the target location information provided by the target positioning and tracking module, the target is tracked in real time and the target position in the three-dimensional environment is updated in real time;

[0187] (7.4) Early warning: Based on the risk analysis results, corresponding early warning information is given in real time.

[0188] This technical solution establishes a three-dimensional point cloud map of the rescue environment through real-time incremental laser three-dimensional pose calculation, and integrates and stores infrared imaging, microphone array, wireless and gas detection information with the three-dimensional point cloud map, effectively solving the defects of insufficient stability of a single sensor, large errors, and the difficulty in joint analysis of positioning direction and distance.

[0189] In practical applications, this technical solution uses LiDAR to incrementally establish the three-dimensional motion pose of detection equipment or detection robots in real time and calculate and generate three-dimensional point cloud information, specifically including:

[0190] (1) The detection equipment or detection robot emits a laser beam through a laser radar and measures its reflection time to obtain the position information of objects in the environment and establish a local three-dimensional point cloud map;

[0191] (2) Using lidar data, the three-dimensional motion trajectory of the detection equipment or detection robot is updated incrementally in real time;

[0192] (3) Based on trajectory calculation, the three-dimensional point cloud information of the environment is generated in real time and incrementally, realizing real-time detection and modeling of the environment, and providing basic data for the intelligent positioning of the objects to be rescued.

[0193] Furthermore, the above step (1) establishes a local three-dimensional point cloud map, specifically including:

[0194] The system described in (1.1) uses a laser radar to emit a laser beam in real time, directs the laser beam to a target object, and collects echo information; the collected echo information is then converted into a single frame of laser radar raw data, and a single frame of three-dimensional point cloud data is obtained through calculation.

[0195] (1.2) The single-frame laser radar raw data includes: the time, angle, distance, and reflection intensity of the reflected laser; the single-frame 3D point cloud data includes a list of 3D points, where each point includes the 3D coordinates (x, y, z) and the reflection time;

[0196] Furthermore, the above step (2) of updating the three-dimensional motion posture trajectory of the detection equipment or detection robot specifically includes:

[0197] (2.1) Feature point extraction: Extract some representative feature points from the lidar point cloud, such as corner points, plane points, etc.

[0198] (2.2) Feature Matching: Feature matching is performed on point cloud data at adjacent moments to find common feature points and calculate the distance and angle between them. This matching is specifically a feature matching list that records the reference frame feature ID, the target frame feature ID, and the feature matching confidence.

[0199] (2.3) Motion Estimation: Using the feature point information obtained by feature matching, the motion posture of the detection equipment or detection robot is estimated, including translation and rotation, to obtain the relative posture, which is represented as a 4×4 floating-point matrix.

[0200] Furthermore, the above step (3) of calculating the three-dimensional point cloud information of the real-time incremental generation environment specifically includes:

[0201] (3.1) Convert the 3D point cloud of each frame to the robot coordinate system;

[0202] (3.2) Merge the 3D point cloud data of each frame into a point cloud dataset and perform voxel filtering;

[0203] (3.3) The point cloud data is registered and the point cloud map is continuously updated according to the robot's motion trajectory to achieve real-time modeling of the environment.

[0204] In practical applications, this technology integrates infrared imaging, microphone array, wireless and gas detection sensor data with 3D pose and 3D point cloud information to form structured 3D data information, including:

[0205] (1) Pre-calibrate the relative displacement and matrix of the lidar and infrared imaging, microphone array, wireless and gas detection sensors to form external parameters, which are represented as a 4×4 floating point matrix;

[0206] (2) Receive synchronized infrared imaging, microphone array, wireless, and gas detection sensor data;

[0207] (3) Calculate the infrared data information of the three-dimensional position within the visible range based on the current posture and three-dimensional point cloud structure information;

[0208] (4) Calculate the probability distribution of possible sound sources in the three-dimensional environment based on the current posture and signal orientation information;

[0209] (5) Record the wireless and gas detection data to the location of the detection equipment or detection robot according to the current posture.

[0210] Furthermore, it receives synchronized infrared imaging, microphone array, wireless and gas detection sensor data, specifically:

[0211] Infrared imaging data is a single-channel infrared imaging image with constant resolution, and its pixel value represents the infrared intensity;

[0212] Microphone array data, which includes sound wave intensity and sound source directional angle data. The directional angle is the clockwise angle between the array reference orientation and the estimated sound source direction.

[0213] Wireless and gas detection sensor data are floating-point data. Before entering the aforementioned system, the data is converted into floating-point numbers in the range of [0,1] according to traditional mapping methods. A number close to 1 indicates that it is close to the dangerous alarm value, and reaching 1 indicates that the sensor data at that location has reached the dangerous alarm value.

[0214] Furthermore, the infrared data information of the three-dimensional position is calculated, specifically:

[0215] (1) Obtaining infrared imaging data: obtaining a frame of infrared imaging data from the infrared imaging sensor;

[0216] (2) Extracting corresponding lidar data: extracting single-frame 3D point cloud data corresponding to the infrared imaging data from the lidar data according to the timestamp;

[0217] (3) Point cloud registration: Based on the extrinsic parameter matrix of the infrared imaging sensor and the lidar, the extracted lidar data is registered with the infrared imaging data, that is, the two data are aligned and the infrared imaging data is mapped to the three-dimensional point cloud data of the corresponding position;

[0218] (4) Point cloud fusion: The registered infrared imaging data is fused with the original three-dimensional point cloud data to form a new three-dimensional point cloud data, which contains the infrared imaging data.

[0219] Furthermore, the probability distribution of possible sound sources in the three-dimensional environment is calculated, specifically:

[0220] (1) Acquiring sound source data: Acquiring sound wave intensity and sound source direction angle data from the microphone array;

[0221] (2) Conversion to the robot coordinate system: According to the external parameter matrix of the microphone array and the lidar, the sound source direction angle data is converted to the sound source direction in the robot coordinate system. The direction contains two data: the three-dimensional starting point coordinates and the three-dimensional direction vector.

[0222] (3) Fusion and storage of data: Starting from the starting point of the sound source in the robot coordinate system, simulate the direction of sound wave propagation and record the sound wave probability. Obtain the first point along the direction vector, and calculate the reflection direction along the normal vector of the point, and then obtain the next point in this direction. Repeat the above steps in sequence to obtain a list of three-dimensional points within a certain reflection distance, assign weights according to the distance, and assign sound source probability data to the three-dimensional points in the list. The sum of the source probabilities assigned to all points in the three-dimensional point list is 1. The first reflection point has the highest probability, and the probability decreases in sequence.

[0223] (4) Point cloud fusion: write the fused point cloud data back into the 3D point cloud information.

[0224] Furthermore, the possible wireless and gas detection sensor data in the three-dimensional environment are calculated, specifically:

[0225] (1) Acquire wireless and gas detection sensor data;

[0226] (2) Extract the radar pose corresponding to the time from the lidar data according to the timestamp;

[0227] (3) Create a three-dimensional marker point based on the aforementioned position and record the wireless and gas detection sensor data;

[0228] (4) Point cloud fusion: write the fused point cloud data back into the 3D point cloud information.

[0229] In practical applications, this technical solution uses multi-source fusion pose solution technology to improve the accuracy of pose estimation and three-dimensional mapping. It includes: multi-source fusion pose solution integrates data from lidar, IMU, wheel speed meter and other sensors to achieve accurate pose estimation of detection equipment or detection robots. Using factor graph optimization and filtering algorithms, different pose information is fused to improve the accuracy and robustness of three-dimensional modeling. At the same time, by matching the landmark data of infrared imaging, microphone array, wireless and gas detection sensors in the three-dimensional map, the pose can be further calibrated and 3D mapping can be assisted. The specific steps are as follows:

[0230] (1) Obtain point cloud data through lidar and build a local map;

[0231] (2) Use IMU and wheel speed meter to obtain robot motion information;

[0232] (3) Fuse motion information with lidar data to estimate the robot’s initial pose;

[0233] (4) Incorporating data from infrared imaging, microphone arrays, wireless, and gas detection sensors as landmarks into a three-dimensional map;

[0234] (5) Use data association algorithms (such as KD-Tree algorithm) to search for matching landmarks in the map;

[0235] (6) Construct a factor graph, which includes pose nodes, landmark nodes and corresponding edges, representing the constraint relationship between poses and landmarks;

[0236] (7) Through factor graph optimization algorithms (such as G2O, iSAM2, etc.), the errors of all edges in the factor graph are minimized to obtain the globally optimal pose and landmark estimation;

[0237] (8) Use filtering algorithms (such as Kalman filtering, particle filtering, etc.) to smooth the pose estimation and eliminate the influence of noise;

[0238] (9) The optimized pose results are applied to the 3D mapping process to achieve accurate 3D modeling.

[0239] Furthermore, the sensors used above are: An IMU (Inertial Measurement Unit) is a sensor that integrates an accelerometer, gyroscope, and magnetometer, capable of measuring the robot's acceleration, angular velocity, and magnetic field information in real time. By integrating the data from the accelerometer and gyroscope, the robot's speed and rotation can be obtained, thereby estimating the robot's position and posture. A wheel speedometer measures the speed of the detection equipment or the robot's wheels. Combined with the robot's geometric parameters, the robot's displacement and rotation can be calculated. Wheel speedometer data can be used as auxiliary information and integrated with IMU and lidar data to improve the accuracy of pose estimation.

[0240] Graph optimization technology plays a key role in multi-source fusion pose solution. Graph optimization technology represents the constraints between poses and landmarks by constructing a factor graph consisting of nodes and edges. Nodes represent the robot's pose or landmarks, and edges represent constraints between poses or between poses and landmarks. By minimizing the errors of all edges in the factor graph, a globally optimal pose and landmark estimate can be obtained. The following is a specific optimization method using existing lidar poses, IMUs, and wheel speedometers:

[0241] Preprocess the data:

[0242] (1) Convert the lidar point cloud data into a local coordinate system;

[0243] (2) Align the timestamps of IMU and wheel speed meter data to ensure data consistency;

[0244] (3) Use IMU data to compensate for motion distortion of lidar point cloud data.

[0245] Initial pose estimation:

[0246] (1) Calculate the initial transformation matrix between adjacent frames by matching the front and back frames of the LiDAR (e.g., ICP algorithm);

[0247] (2) By integrating the accelerometer and gyroscope data of the IMU, the displacement and rotation between adjacent moments are obtained;

[0248] (3) Use the wheel tachometer data to calculate the robot's displacement and rotation, combined with the robot's geometric parameters.

[0249] Multi-source data fusion:

[0250] (1) Convert the pose estimates of the lidar, IMU, and wheel speedometer into nodes in the factor graph;

[0251] (2) Use the constraints between the lidar poses, the IMU constraints, and the wheel speedometer constraints to construct the edges of the factor graph;

[0252] (3) For other sensors (such as infrared imaging, microphone arrays, wireless and gas detection sensors), their data are stored as landmarks in the factor graph and edges are constructed between them and the pose nodes.

[0253] Optimization method:

[0254] (1) Optimize the factor graph using the Maximum A Posteriori (MAP) estimation method. Gradient descent, Gauss-Newton method, or Levenberg-Marquardt algorithm can be used.

[0255] (2) Use sparse linear algebra techniques (such as sparse Cholesky decomposition) to improve optimization speed and adapt to large-scale data processing.

[0256] Result fusion and update:

[0257] (1) Update the robot pose and landmark positions based on the optimized factor graph;

[0258] (2) Apply the optimized pose information to the 3D mapping process to achieve accurate 3D modeling.

[0259] Through this series of operations, the existing lidar pose, IMU and wheel speed meter can be used for graph optimization to improve the accuracy and robustness of pose estimation.

[0260] In practical applications, the 3D neural network probability field construction unit uses neural network technology to fuse and analyze data from multiple sensors to construct a probability field that can be used for intelligent target positioning. The specific steps are as follows:

[0261] This unit includes the following key components: signed distance field calculation, establishment and rendering of infrared radiation environment field, construction of audio neural radiation field, establishment of radio neural radiation field, establishment of gas density field and fusion of probability field.

[0262] 1. Signed distance field calculation:

[0263] A signed distance field is calculated for the 3D model using a neural network. A signed distance field is the shortest path from the target point to various environmental points, and it contains both directionality and distance information. The point cloud information obtained through 3D modeling is input into a neural network for clustering, segmentation, and feature extraction. SDF features are calculated, and voxelized sampling is used to generate a signed distance field. Constructing a signed distance field provides the foundation for establishing environmental fields for various sensors. The specific steps are as follows:

[0264] (1) Before calculating the SDF, the point cloud is clustered and divided into different regions, each of which represents an original feature;

[0265] (2) In each region, a region-based segmentation algorithm is used to divide the point cloud into different blocks, each of which represents a local feature;

[0266] (3) In each block, the edge feature points and sharp feature points in the point cloud are extracted based on the feature weight, and a minimum spanning tree is constructed to connect these feature points;

[0267] (4) Input the feature points into a randomly initialized neural network, which consists of an encoder and a decoder. The encoder maps the three-dimensional coordinates to a low-dimensional vector, and the decoder maps the low-dimensional vector to a directed distance.

[0268] (5) Define a self-supervised loss function that uses the distance from the nearest point to the target point in a batch of actively sampled query points to constrain the predicted directed distance;

[0269] (6) Use the gradient descent method to update the parameters of the neural network to minimize the loss function;

[0270] (7) Use voxel sampling to convert the output of the neural network into a signed distance field. 2. Establishment and rendering of infrared radiation environment field:

[0271] (1) Sensor platform pose calculation: The pose of the sensor platform in the environment and the internal and external parameters of the infrared imaging are projected through a general camera model to obtain the correspondence between the two-dimensional camera and the three-dimensional environment.

[0272] The sensor platform refers to a device that carries the aforementioned sensors, such as a drone or robot, on which the sensors are firmly fixed and their relative positional relationship remains unchanged. Intrinsic parameters include intrinsic parameters such as the focal length and principal point coordinates of the camera. Extrinsic parameters refer to the transformation relationship between the camera and the world coordinate system, including the rotation matrix (R) and the translation matrix (T). The rotation matrix describes the rotation relationship of the camera coordinate system relative to the world coordinate system, while the translation matrix describes the coordinates of the camera's optical center in the world coordinate system. The camera model is a transformation from a two-dimensional image space to a three-dimensional world coordinate system. Specifically, the projection process from a point (X, Y, Z) in the three-dimensional world coordinate system to a point (u, v) on the camera imaging plane can be expressed by the following formula: [u, v, 1] T =K×[R|T]×[X,Y,Z,1] T Where K is the camera intrinsic parameter matrix composed of intrinsic parameters, and [R|T] is the camera extrinsic parameter matrix composed of extrinsic parameters. After the above projection process, we can obtain the correspondence between the points on the camera's 2D image and the points in the 3D environment.

[0273] (2) 2D infrared image projection: Each pixel in the 2D infrared image is projected ray-wise through the space of a signed distance field. A signed distance field is a function that implicitly represents a 3D geometric shape. It gives the signed distance from any point in space to the nearest surface. Ray-wise projection is a method for converting a 2D image into a set of rays in 3D space. It determines the direction and starting point of each ray corresponding to each pixel based on the camera model and internal and external parameters.

[0274] (3) Neural Network Optimization: This projection function is iteratively optimized and regressed using a multi-layer perceptron (MLP) neural network to obtain an infrared neural network radiation field. The infrared neural network radiation field is a method that uses a neural network to represent the distribution of infrared radiation in three-dimensional space. It takes any point in space as input and outputs the volume density and emission spectrum of that point. Neural network optimization is a method that uses algorithms such as gradient descent to adjust the parameters of the neural network to minimize the loss function, which is usually the difference between the predicted value and the true value. In this method, the loss function is the mean squared error between the projection function and the true two-dimensional infrared image.

[0275] (4) Volume density sampling: The infrared neural network radiation field is sampled in blocks to obtain the volume density at each point. Volume density is a quantity that describes the degree of material presence in a certain area of space. It reflects the area's ability to absorb or scatter light. Block sampling is a method that discretizes a continuous space into several small blocks and takes several sampling points in each small block. It can improve sampling efficiency and accuracy.

[0276] (5) Infrared image rendering: A camera model is built at a given location for projection. The infrared image is rendered by integrating and accumulating all volume density values along the ray. Infrared image rendering is a method that uses the infrared neural network radiation field to generate a two-dimensional infrared image from a new perspective. It simulates the propagation of light in three-dimensional space. Integration and accumulation is a method that calculates the influence of all substances encountered by the ray on its intensity and color. It performs numerical calculations based on physical laws such as Bell's law.

[0277] 3. Establishing the audio neural radiation field:

[0278] (1) Sound signal processing: A microphone array is used to collect sound signals in the environment and process and preprocess the sound signals, including denoising, filtering, and voice separation. The purpose of this step is to extract the characteristics and direction of the sound source and eliminate irrelevant noise and interference.

[0279] (2) Sound wave propagation and reflection model: Based on the sound wave propagation and reflection model, an audio neural radiation field is established. This model takes into account the propagation and reflection of sound waves in the air, and uses the sound path and three-dimensional environmental information of the sound wave to calculate the sound source position and the arrival time of the sound wave. The audio neural radiation field is a fully connected neural network whose input is a continuous 5D coordinate (spatial position (x, y, z) and viewing direction (θ, φ)), and its output is the volume density and viewing angle-related emission radiation of the spatial position. The audio neural radiation field can represent the distribution and changes of sound in complex three-dimensional scenes.

[0280] (3) Neural Network Optimization: The audio radiation field is iteratively optimized and regressed through a multi-layer perceptron (MLP) neural network to obtain the final audio neural radiation field. This radiation field can be used for audio localization and orientation. The optimization goal of the neural network is to use a loss function to reconstruct the input sound signal, while taking into account the exponential attenuation of sound waves with distance and the reflection characteristics of sound waves when they encounter a directed distance field.

[0281] 4. Establishment of radio neural radiation field:

[0282] (1) Radio signal processing: Use wireless sensors to collect radio signals in the environment and process and preprocess the signals, including denoising, filtering, signal identifier recognition, etc.

[0283] (2) Radio signal propagation model: Based on the propagation, reflection, and penetration models of radio signals of different frequencies, a radio neural radiation field is established. This model takes into account the propagation and reflection of radio signals in the air, as well as the penetration and reflectivity of three-dimensional environmental information, to calculate the signal propagation path and signal arrival time.

[0284] (3) Neural network optimization: The radio radiation field is iteratively optimized and regressed through a multi-layer perceptron (MLP) neural network to obtain the final radio neural radiation field. This radiation field can be used to locate and orient radio signals. The goal of neural network optimization is to make the radio neural radiation field reproduce the distribution of radio signals in the real environment to the greatest extent possible while maintaining a certain degree of smoothness and consistency. The loss function of neural network optimization includes reconstruction loss, regularization loss, and perspective consistency loss.

[0285] 5. Establishment of gas density field:

[0286] (1) Gas information processing: Use gas detectors to collect gas information in the environment, and process and preprocess the gas information, including denoising, filtering, gas identification, etc.

[0287] (2) Gas diffusion model: Based on the gas diffusion model, a gas density field is established. This model takes into account the diffusivity and propagation of gas in the air, and calculates the density and expected distribution of the gas. Diffusion refers to the tendency of gas to be evenly distributed in space due to molecular thermal motion and concentration differences; propagation refers to the tendency of gas to move in a certain direction in space due to wind or other external forces. Depending on the type of gas and environmental conditions, an appropriate diffusion equation or transmission equation is selected to describe the law of change of the gas density field over time and space, and the solution is obtained by numerical or analytical means.

[0288] (3) Neural network optimization: The gas density field is iteratively optimized and regressed through the multi-layer perceptron (MLP) neural network to obtain the final gas neural radiation field.

[0289] 6. Fusion of Probability Fields

[0290] To achieve effective fusion, we can use the methods of neural network backpropagation and weight fusion, as well as the methods of prior probability and posterior probability.

[0291] Neural network back propagation and weight fusion:

[0292] Neural network backpropagation is a supervised learning algorithm that adjusts neural network weights by minimizing prediction errors. In constructing a 3D neural network probability field, we can use the backpropagation algorithm to train a multilayer perceptron (MLP) to fuse different probability fields. The specific steps are as follows:

[0293] (1) Input layer: The data of different probability fields are used as the input of the neural network, such as infrared neural network radiation field, audio neural radiation field, radio neural radiation field and gas neural radiation field.

[0294] (2) Hidden layer: Multiple hidden layers are used to learn the correlation between different probability fields. The number of neurons and activation functions in the hidden layer can be adjusted according to the specific task and dataset.

[0295] (3) Output layer: Outputs the fused three-dimensional neural network probability field. The number of neurons in the output layer should be the same as that in the input layer to ensure that the output probability field has the same dimension.

[0296] (4) Weight fusion: During the training process, the weights of the neural network are adjusted according to the prediction error. Through weight fusion, the neural network can learn the correlation between different probability fields, thus achieving effective fusion.

[0297] Prior and posterior probability methods:

[0298] The prior probability and posterior probability methods are based on Bayesian theory, which uses prior knowledge and observation data to update the probability distribution. In the construction of a three-dimensional neural network probability field, we can use the prior probability and posterior probability methods to fuse different probability fields. The specific steps are as follows:

[0299] (1) Define prior probability distribution: Based on the properties of different probability fields and prior knowledge, define a prior probability distribution for each probability field.

[0300] (2) Observation data: The observation data from sensors such as infrared imaging, audio, radio, and gas detection are used as conditional probabilities.

[0301] (3) Calculate the posterior probability: According to the Bayesian formula, the posterior probability is calculated by combining the prior probability and the conditional probability. The posterior probability represents the relative weight of each probability field given the observed data.

[0302] The following are the specific implementation steps for constructing a probability field that can be used for intelligent target positioning:

[0303] 1. Input data preprocessing:

[0304] Before constructing the 3D neural network probability field, the input sensor data needs to be preprocessed. This includes:

[0305] (1) Data alignment: aligning data from different sensors based on timestamps;

[0306] (2) Data filtering: De-noising and filtering are performed on the original data to improve data quality.

[0307] 2. Calculation of signed distance field:

[0308] Based on a neural network, a directed distance field is calculated for the 3D model. A directed distance field is the shortest path from the target point to various environmental points. It contains directionality and distance information and can be used to establish environmental fields for different sensors.

[0309] 3. Establishment of environmental field:

[0310] Based on the pre-processed sensor data, the infrared radiation environment field, audio neural radiation field, radio neural radiation field, and gas density field are established respectively. The specific steps are described above.

[0311] 4. Fusion of Probability Fields

[0312] The different environmental probability fields are fused. The specific steps are described above.

[0313] 5. Optimization and update:

[0314] (1) Update the 3D neural network probability field in real time based on the latest sensor data;

[0315] (2) Optimize the neural network probability field to improve its accuracy and robustness.

[0316] 6. Output results:

[0317] The constructed three-dimensional neural network probability field is provided to the target positioning and tracking module for subsequent target positioning, tracking and decision-making.

[0318] In actual applications, by integrating the previous data, the distribution probability of the total number of objects to be rescued is calculated. The specific implementation method is as follows:

[0319] Each point in the point cloud has a probability value ranging from 0 to 1. We will discuss this in two parts: a single sensor approach and a multi-sensor fusion approach.

[0320] Single sensor approach:

[0321] For a single-sensor approach, we can directly use the corresponding probability field as the distribution probability of the target object. For example, if we only use an infrared imaging sensor for detection, we can directly use the infrared neural network radiation field as the distribution probability of the target object. First, for each point in the point cloud, we calculate its probability value in the infrared neural network radiation field. Then, we set a threshold, such as 0.5. Only when the probability value is greater than the threshold, we consider the point as a possible location of the target object.

[0322] Multi-sensor fusion approach:

[0323] For multi-sensor fusion, we need to combine the probability fields of different sensors to calculate the distribution probability of the objects to be rescued. The specific method is as follows:

[0324] (1) Voxelized rendering: First, we voxelize the three-dimensional space and convert the point cloud data in each voxel into a format that can be processed by the neural network.

[0325] (2) Ray Projection: For each sensor, we perform ray projection on its corresponding neural network radiation field. Specifically, we send rays from the sensor’s position through the 3D neural network radiation field space and calculate the intersection of each ray with the voxel.

[0326] (3) Cumulative Probability: For each intersection of a ray and a voxel, we calculate the probability of its presence in the corresponding sensor’s neural network radiation field and add them up to obtain the fusion probability. For example, for an infrared imaging sensor and a microphone array sensor, we can calculate the probability of each intersection in the infrared neural network radiation field and the audio neural network radiation field and add them together to obtain the fusion probability.

[0327] (4) Judgment: Similarly, a threshold is set, for example, 0.5. Only when the fusion probability is greater than the threshold, we consider that the point may be the location of the object to be rescued.

[0328] In practical applications, the decision-making and warning module is responsible for making decisions based on the 3D environmental information and target location information output by the 3D Fusion and Processing Module and the Target Positioning and Tracking Module. This module's primary purpose is to provide follow-up operational recommendations to operators or autonomous detection systems and issue warnings in the event of danger. By analyzing 3D environmental information and target location information, the decision-making and warning module can provide useful information for search and rescue operations, improving search efficiency and safety. The main functions of the decision-making and warning module include:

[0329] Risk Analysis: Based on the input 3D environmental information, this module analyzes potential risks within the environment. These risks may include unstable building structures, toxic gas leaks, fire sources, and more. This module identifies these risk areas through comprehensive analysis of various sensor data, allowing operators or autonomous detection systems to avoid them and reduce risk during the search and rescue process.

[0330] Path Planning: Based on input 3D environmental information and target location information, this module plans the optimal path for search and rescue teams or autonomous detection equipment. This module analyzes the 3D environmental model and combines it with risk zone information to calculate the shortest or safest path to the target location. Path planning algorithms include A* and Dijkstra, which can generate paths based on different optimization objectives (such as shortest distance, minimum risk, etc.).

[0331] Target Tracking: Based on the target location information provided by the Target Positioning and Tracking module, the module tracks the target in real time. The target may be a person, animal, or other critical material to be rescued. This module updates the target's position in the 3D environment in real time based on the target location information, providing real-time target information to operators or autonomous detection systems. Target tracking algorithms can include Kalman filters and particle filters, which can effectively handle the uncertainty and noise of target location information.

[0332] Early warning: Based on the risk analysis results, early warning information is given in real time. The early warning information may include dangerous gases, high temperatures, collapse warnings, etc.

[0333] In a specific embodiment of the present invention, we use a rescue robot in tunnel collapse rescue as the background to further explain this technical solution in detail:

[0334] When a rescue mission is received, the rescue robot first activates the LiDAR module to acquire real-time point cloud information about the surrounding environment. The LiDAR module uses a high-precision laser rangefinder to scan the target and measure the distance, angle, and height of surrounding points in real time.

[0335] The rescue robot's 3D modeling and perception module uses point cloud information acquired by the LiDAR module for real-time 3D mapping. A multi-source fusion pose calculation unit enables precise pose estimation of the rescue robot, improving the accuracy and robustness of 3D modeling.

[0336] The infrared imaging module also detects thermal radiation in the environment to identify trapped individuals. It operates normally in low-light conditions and boasts high detection sensitivity. Hazardous gas leaks and fires also generate significant amounts of thermal radiation. This thermal radiation can be used to locate the fire source and facilitate rescue efforts.

[0337] The sound directional detection module receives sound signals and locates the general direction of the trapped person. The sound directional detection module can identify sound sources such as human voices and cries for help, and use sound source localization algorithms to estimate the direction of the sound source.

[0338] The wireless communication module receives radio signals that may be emitted by the trapped person, such as mobile phone signals, walkie-talkie signals, etc. By analyzing the signal strength and frequency, the wireless communication module can estimate the approximate direction of the signal source.

[0339] The gas detection module detects gas components in the environment, such as carbon monoxide, methane and other toxic gases. By monitoring the gas concentration in real time, the rescue robot can avoid entering dangerous areas and ensure its own safety.

[0340] By fusing information from infrared imaging, microphone arrays, wireless, and gas detection modules with point cloud information from the LiDAR module, the rescue robot can intelligently locate the target. In this example, we'll use a tunnel collapse rescue application as an example. Suppose the rescue robot discovers a collapsed area in the tunnel, potentially burying someone nearby.

[0341] The rescue robot first integrates the point cloud information acquired by the LiDAR module with the point cloud information from the previous few seconds, updating the 3D map in real time to create a 3D point cloud map of the environment. Using the 3D modeling and perception modules, the rescue robot calculates its own 3D position and the 3D structure of the environment surrounding the collapse point in real time.

[0342] The rescue robot then integrates the information acquired by the infrared imaging, microphone array, wireless, and gas detection modules with the 3D map. Using a multi-source fusion pose calculation unit, the rescue robot estimates its own pose, the position of the trapped person, and the location of the collapse point. Simultaneously, by calculating the probability of the position distribution of the target in 3D space, the rescue robot determines the most likely location of the target.

[0343] The rescue robot autonomously explores areas where victims may be trapped, following a pre-set search strategy. It uses a lidar module to scan its surroundings in real time. It also monitors the surroundings through infrared imaging, microphone arrays, wireless, and gas detection modules, continuously updating the three-dimensional map and the probability of the trapped person's location.

[0344] When the rescue robot locates a trapped person, the infrared imaging module detects thermal radiation to determine their location. The microphone array module uses a sound source localization algorithm to determine the victim's direction. The wireless communication module analyzes signal strength and frequency to determine the victim's location. Through this comprehensive analysis, the rescue robot can pinpoint the victim's specific location and initiate rescue operations.

[0345] The rescue robot in this embodiment utilizes data fusion technologies from multiple sensors, including lidar, infrared imaging, microphone arrays, wireless sensors, and gas detection, to intelligently locate the target being rescued and model the collapsed area. The rescue robot can use the point cloud information acquired by the lidar module to perform three-dimensional reconstruction. By fusing point cloud information acquired at different times, a three-dimensional model of the collapsed area can be created. The rescue robot can use the 3D model to explore the collapsed area and, based on the information in the model, plan the optimal path to avoid dangerous areas.

[0346] When a rescue robot detects a victim, it uses the fused information from multiple sensors to calculate a probability distribution for the victim's location. This probability distribution allows the robot to determine the reliability of the victim's location and select the optimal rescue plan. For example, if a victim is buried deep within, the robot can use its infrared imaging module and microphone array module to determine their approximate location. It can then use its gas detection module to monitor the concentration of toxic gases in the environment, avoid entering dangerous areas, select the optimal rescue path, and successfully rescue the victim.

[0347] In addition, we use the application of a rescue robot in the context of a dangerous gas leak and fire in a pipeline corridor.

[0348] Given the complexity of the on-site situation, the rescue robot must fuse data from multiple sensors to locate the target and conduct rescue operations. To do this, the rescue robot first activates its LiDAR module to acquire real-time point cloud information about the surrounding environment. Leveraging the LiDAR module's high-precision ranging capabilities, the rescue robot can quickly assess the fire's development and create real-time 3D modeling based on this real-time point cloud information, providing an accurate environmental model for subsequent positioning and rescue operations.

[0349] The rescue robot's gas detection module can also monitor the composition of gases in the environment in real time, such as carbon monoxide and methane. By monitoring gas concentrations in real time, the rescue robot can avoid entering dangerous areas and ensure its own safety. Furthermore, the rescue robot's infrared imaging module can also detect possible fire sources in the environment, effectively preventing further damage.

[0350] When a trapped person is found, the rescue robot receives sound signals through the microphone array module and locates the victim's approximate direction. Using the sound source localization algorithm in the microphone array module, the rescue robot estimates the direction of the sound source and matches the trapped person's position with the environmental point cloud information obtained by the lidar module.

[0351] In addition, the rescue robot's wireless communication module can also receive radio signals that may be emitted by the trapped person, such as mobile phone signals and walkie-talkie signals. By analyzing the signal strength and frequency, the wireless communication module can estimate the approximate direction of the signal source, thereby helping the rescue robot locate the trapped person more quickly.

[0352] After locating the trapped victim, the rescue robot can use multiple actuators to perform rescue operations, such as using a robotic arm to retrieve objects or a sprinkler to extinguish fires. Based on the actual situation, the rescue robot can integrate data from multiple sensors to perform multi-angle positioning and rescue operations, maximizing the efficiency and success rate of rescue operations.

[0353] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0354] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0355] Those skilled in the art will understand that all or part of the steps of the method for implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0356] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0357] Throughout this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "embodiment" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0358] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0359] The target intelligent positioning control system and method based on infrared imaging, laser radar, and sound directional detection of the present invention are used. The laser radar emits a laser beam and measures its reflection time to obtain the position information of objects in the environment. At the same time, the real-time incremental establishment of a three-dimensional point cloud map technology solution can generate the three-dimensional coordinates of the object position in real time during the movement of the detection equipment or detection robot, thereby establishing a three-dimensional point cloud map of the environment. This technical solution has the advantages of high precision, high speed, and high stability. It can be applied to positioning and detection in various environments, effectively improve the efficiency of rescue and investigation, and reduce the difficulty of accurately locating the object to be rescued, thereby efficiently and scientifically improving the efficiency of rescue and investigation, significantly enhancing the efficiency and accuracy of locating trapped personnel, and has a wider range of applications compared to existing technologies.

[0360] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A target intelligent positioning control system based on infrared imaging, laser radar and sound directional detection, characterized in that: The system comprises: The LiDAR module uses a high-precision laser rangefinder to scan the target and measure the distance, angle, and height information of surrounding points in real time; The infrared imaging module uses a highly sensitive infrared sensor to capture the target's thermal radiation characteristics during the current rescue mission and generate high-resolution infrared image data; The sound directional detection module uses a high-sensitivity microphone array to capture the sound signals emitted by the target and uses a signal processing algorithm to determine the direction of the sound target and the probability distribution of the sound source; The radio detection module detects the target's radio signal through a high-sensitivity antenna and radio frequency receiver to identify and locate wireless communication devices; The gas detection module monitors the gas composition around the target through a highly sensitive gas sensor to detect possible harmful gases or gases that are indicative of the target; A three-dimensional modeling and perception module, connected to the lidar module, is used to obtain a three-dimensional model and scene information of the monitored target, providing stereoscopic perception capabilities for target positioning; a three-dimensional fusion and processing module connected to the three-dimensional modeling and perception module, infrared imaging module, sound directional detection module, radio detection module, and gas detection module, and performing three-dimensional fusion and processing on the data collected by the infrared imaging module, lidar module, sound directional detection module, three-dimensional modeling and perception module, radio detection module, and gas detection module by adopting a multi-sensor data fusion algorithm, so as to improve the positioning accuracy and robustness of the target; A target positioning and tracking module, connected to the 3D fusion and processing module, is used to locate and track the target in real time based on the target position information output by the 3D fusion and processing module to ensure that the target is not lost; and The decision-making and early warning module is connected to the target positioning and tracking module, and is used to combine the data information output by the three-dimensional fusion and processing module and the target positioning and tracking module, and perform target behavior analysis and prediction. By formulating detection strategies and execution plans, it can provide real-time early warning prompts, potential risk assessments, and response measures.

2. A method for intelligent target positioning control based on infrared imaging, laser radar and sound directional detection using the system of claim 1, characterized in that: The method comprises the following steps: (1) Use the LiDAR module to obtain point cloud information of the surrounding environment in real time; (2) The three-dimensional modeling and perception module uses the point cloud information obtained by the laser radar module to perform real-time three-dimensional modeling; (3) Based on the trajectory information obtained by the laser radar module, a three-dimensional point cloud structure of the current environment is generated in real time and incrementally, and the sensor data of the infrared imaging module, the sound directional detection module, the radio detection module and the gas detection module received synchronously are fused and positioned and parameter calculated using the three-dimensional fusion and processing module; (4) transmitting the calculated received data of the infrared imaging module, the sound directional detection module, the radio detection module, and the gas detection module to the three-dimensional fusion and processing module for fusion processing; (5) The three-dimensional fusion and processing module calculates the current target's position and probability distribution based on the sensor data and the three-dimensional point cloud structure information provided by the three-dimensional modeling and perception module; (6) The target positioning and tracking module performs real-time positioning and tracking processing based on the target position information and probability distribution output by the three-dimensional fusion and processing module to ensure that the target is not lost; (7) The decision-making and warning module makes decisions based on the three-dimensional environmental information and target position information output by the three-dimensional fusion and processing module and the target positioning and tracking module, gives subsequent operation suggestions or conducts autonomous detection, and issues warnings in dangerous moments.

3. The target intelligent positioning control method based on infrared imaging, laser radar and sound directional detection according to claim 2 is characterized in that: The step (2) specifically includes the following steps: (2.1) The detection equipment or detection robot emits a laser beam through a laser radar and measures its echo reflection time to obtain the location information of the target object in the current environment, including the time, angle, distance, and reflection intensity of the reflected laser beam, and calculates single-frame three-dimensional point cloud data, wherein the single-frame three-dimensional point cloud data includes the three-dimensional coordinates (x, y, z) and reflection time of each point, and uses this to establish a local three-dimensional point cloud map; (2.2) Utilizing the data information acquired by the laser radar, extracting feature points from the three-dimensional point cloud of the laser radar, performing feature matching on the acquired point cloud data at adjacent moments, finding identical feature points, thereby generating a feature matching list, and recording the reference frame feature ID, target frame feature ID, and feature matching confidence of each feature point, thereby incrementally updating the three-dimensional motion pose trajectory of the detection equipment or detection robot in real time; (2.3) Based on the acquired three-dimensional motion posture trajectory, the three-dimensional point cloud data of each frame are converted into the robot coordinate system, merged into a point cloud data set for voxel filtering, and the point cloud map is continuously updated according to the motion trajectory of the detection equipment or detection robot to achieve real-time modeling of the environment.

4. The target intelligent positioning control method based on infrared imaging, laser radar and sound directional detection according to claim 3 is characterized in that: The step (3) specifically includes the following steps: (3.1) Pre-calibrate the relative displacements and matrices between the lidar and the infrared imaging, microphone array, radio detection, and gas detection sensors to form extrinsic parameters, which are represented as a 4×4 floating-point matrix. (3.2) receiving synchronized infrared imaging data, microphone array data, radio detection data, and gas detection sensor data; (3.3) Calculating infrared data information of the three-dimensional position within the visible range based on the motion posture and three-dimensional point cloud structure information of the detection equipment or detection robot; (3.3.1) Obtaining infrared imaging data: Obtaining a frame of infrared imaging data from the infrared imaging sensor; (3.3.2) Extracting corresponding lidar data: Extracting single-frame 3D point cloud data corresponding to the infrared imaging data from the lidar data based on the timestamp; (3.3.3) Point Cloud Registration: Based on the extrinsic parameter matrices of the infrared imaging sensor and the lidar, the extracted lidar data is registered with the infrared imaging data, i.e., the two data are aligned, and the infrared imaging data is mapped to the 3D point cloud data at the corresponding position; (3.3.4) Point cloud fusion: Fuse the registered infrared imaging data with the original 3D point cloud data to form a new 3D point cloud data; (3.4) Calculate the probability distribution of possible sound sources in the three-dimensional environment based on the motion posture and signal direction information of the current detection equipment or detection robot; (3.4.1) Acquiring sound source data: Acquire sound wave intensity and sound source direction angle data from the microphone array; (3.4.2) Conversion to the robot coordinate system: Based on the extrinsic matrix of the microphone array and the lidar, the sound source direction angle data is converted to the sound source direction in the robot coordinate system. This direction contains two data: the three-dimensional starting point coordinates and the three-dimensional direction vector; (3.4.3) Fusion and storage of data: Starting from the sound source in the robot coordinate system, simulate the direction of sound wave propagation and record the sound wave probability. Acquire the first point along the direction vector and calculate the reflection direction along the normal vector of the first point. Acquire the next point from the reflection direction. Repeat this step until a list of 3D points within a certain reflection distance is obtained. Weights are assigned according to distance and sound source probability data is assigned to the 3D points in the list. (3.4.4) Point cloud fusion: write the fused point cloud data back into the 3D point cloud information; (3.5) recording the acquired radio and gas detection data to the location of the detection equipment or detection robot according to the current motion posture of the detection equipment or detection robot; (3.5.1) Acquire radio and gas detection sensor data; (3.5.2) Extract the radar pose corresponding to the current time from the lidar data based on the timestamp; (3.5.3) Create a three-dimensional marker based on the radar pose and record wireless and gas detection sensor data; (3.5.4) Point cloud fusion: Write the fused point cloud data back into the 3D point cloud information.

5. The target intelligent positioning control method based on infrared imaging, laser radar and sound directional detection according to claim 2 is characterized in that: The step (4) specifically includes the following steps: (4.1) Preprocessing the received data from the infrared imaging module, the sound directional detection module, the radio detection module, and the gas detection module, including aligning the data from different sensors according to timestamps and performing denoising and filtering on the raw data; (4.2) Constructing a signed distance field based on the shortest path from the target point to each environmental point: Input the 3D point cloud information obtained through 3D modeling into the neural network for clustering, segmentation, and feature extraction. SDF features are calculated and the signed distance field is calculated through voxel sampling. (4.3) Based on the pre-processed sensor data, establish the infrared radiation probability field, audio neural radiation probability field, radio neural radiation probability field and gas neural density probability field respectively; (4.4) Use the back-propagation algorithm to train the multi-layer perceptron MLP to fuse the different probability fields.

6. The target intelligent positioning control method based on infrared imaging, laser radar and sound directional detection according to claim 5 is characterized in that: The step (4.3) of establishing the infrared radiation probability field specifically includes: Sensor platform pose calculation: Obtain the pose of the sensor platform in the current environment and the intrinsic and extrinsic parameters of infrared imaging, and project them through a universal camera model to obtain the correspondence between the two-dimensional camera and the three-dimensional environment; Two-dimensional infrared image projection: each pixel on the two-dimensional infrared image is ray-projected and passes through the space of the signed distance field; Neural network optimization: Use the multi-layer perceptron (MLP) neural network to iteratively optimize and regress the currently obtained projection function to obtain the infrared neural network radiation field; Volume density sampling: sampling the infrared neural network radiation field in blocks and obtaining the volume density at each point; Infrared image rendering: A camera model is built at a preset position and projected. All volume density values on the ray are integrated and accumulated, and then rendered to obtain an infrared image. The step (4.3) of establishing the audio neural radiation probability field specifically includes: Sound signal processing: Use microphone arrays to collect sound signals in the environment and pre-process the sound signals; Sound wave propagation and reflection model: Based on the sound wave propagation and reflection model, the audio neural radiation field is established, and the sound source position and sound wave arrival time are calculated using the sound wave's acoustic path and three-dimensional environmental information; Neural network optimization: The audio radiation field is iteratively optimized and regressed through the multi-layer perceptron (MLP) neural network to obtain the final audio neural radiation probability field.

7. The target intelligent positioning control method based on infrared imaging, laser radar and sound directional detection according to claim 5 is characterized in that: The step (4.3) of establishing the radio neural radiation probability field specifically includes: Radio signal processing: Use wireless sensors to collect radio signals in the environment and pre-process the signals; Radio signal propagation model: Based on the propagation, reflection, and penetration models of radio signals of different frequencies, the radio neural radiation field is established and the signal propagation path and signal arrival time are calculated; Neural network optimization: The radio radiation field is iteratively optimized and regressed through the multi-layer perceptron (MLP) neural network to obtain the final radio neural radiation probability field; The step (4.3) of establishing the gas density probability field specifically includes: Gas information processing: Use gas detectors to collect gas information in the environment and pre-process the gas information; Gas diffusion model: Based on the gas diffusion model, the gas density field is established to calculate the gas density and expected distribution; Neural network optimization: The gas density field is iteratively optimized and regressed through the multi-layer perceptron (MLP) neural network to obtain the final gas neural radiation probability field.

8. The target intelligent positioning control method based on infrared imaging, laser radar and sound directional detection according to claim 5 is characterized in that: The step (4.4) specifically includes the following steps: (4.4.1) Input layer: takes the data of different probability fields as the input of the neural network; (4.4.2) Hidden Layer: The correlation between different probability fields is learned through multiple hidden layers, where the number of neurons and activation functions in the hidden layers are adjusted according to the specific task and dataset; (4.4.3) Output layer: Outputs the fused three-dimensional neural network probability field. The number of neurons in the output layer should be the same as that in the input layer to ensure that the output probability field has the same dimension. (4.4.4) Weight Fusion: During the training process, the weights of the neural network are adjusted according to the prediction error. Through weight fusion, the neural network learns the correlation between different probability fields, thereby achieving effective fusion.

9. The target intelligent positioning control method based on infrared imaging, laser radar and sound directional detection according to claim 6 is characterized in that: The step (5) specifically includes the following steps: (5.1) Using a single sensor, calculate the probability value of each point in the 3D point cloud structure corresponding to the infrared radiation probability field. When the probability value is greater than a preset threshold, the point is considered to be the location of the object to be rescued; (5.2) Using multi-sensor fusion processing to combine the probability fields of different sensors to calculate the probability distribution; (5.2.1) Voxelized rendering: Voxelize the current 3D model and convert the point cloud data in each voxel into a format that can be processed by the neural network; (5.2.2) Ray projection: Perform ray projection on the radiation field of each sensor after the neural network processing and calculate the intersection point of each ray with the voxel; (5.2.3) Cumulative probability: For each intersection of a ray and a voxel, calculate its probability value in the neural network radiation field of the corresponding sensor and accumulate it to obtain the fusion probability; (5.2.4) Judgment: Set a threshold. When the fusion probability is greater than the threshold, the point is considered to be the location of the object to be rescued.

10. The target intelligent positioning control method based on infrared imaging, laser radar and sound directional detection according to any one of claims 2 to 9, characterized in that: The step (7) specifically includes the following steps: (7.1) Risk Analysis: Based on the input 3D environmental information, the system conducts a comprehensive analysis of the various sensor data obtained to analyze the possible risks in the environment and identify risk areas. (7.2) Path planning: Analyze the currently generated 3D environment model and calculate the shortest or safest path to the target location by combining it with risk area information; (7.3) Target tracking: Based on the target location information provided by the target positioning and tracking module, the target is tracked in real time and the target position in the three-dimensional environment is updated in real time; (7.4) Early warning: Based on the risk analysis results, corresponding early warning information is given in real time.

Images