Multi-sensor fusion life detection system based on hand-held display control terminal

Abstract

The application relates to the technical field of life detection, and discloses a multi-sensor fusion life detection system based on a handheld display control terminal, which comprises a sensor group used for collecting vital sign signals and environmental parameters, the sensor group comprising an infrared thermal imager, an audio vibration sensor, a radar wave detector and an environmental sensor; and a data processing module connected with the sensor group and used for carrying out noise filtering and feature extraction on the data collected by the sensor group. The application adopts a multi-sensor cooperative detection and data fusion technology, and achieves the technical effect of real-time and accurate detection of vital signs in a complex environment. Compared with the detection method of a single sensor in the prior art, the application can complement and fuse the data of multiple sensors at a disaster site, significantly improves the reliability and accuracy of detection, and solves the problem of a detection blind area of a traditional method under environmental noise and interference.

Description

Technical Field

[0001] This invention relates to the field of life detection technology, specifically a multi-sensor fusion life detection system based on a handheld display and control terminal. Background Technology

[0002] With the acceleration of urbanization, the complexity of disaster relief missions is constantly increasing. Timely and effective life detection and rescue operations are crucial after a disaster. However, traditional detection technologies often face problems such as environmental noise, sensor malfunction, and blind spots, leading to low search and rescue efficiency and increased loss of life. Therefore, how to accurately and in real-time locate trapped personnel in complex environments has become a key issue in improving rescue efficiency.

[0003] Currently, common technical solutions in the rescue field include single-sensor detection methods, such as infrared thermal imagers for temperature detection and audio sensors for detecting vital signs. However, these methods typically rely on the judgment of a single sensor, lack data complementarity, and are prone to misjudgment or missed detection in complex environments. Furthermore, most existing technologies employ a centralized data processing model, leading to limited information transmission and the risk of single points of failure.

[0004] The main problem with existing technologies is that a single sensor cannot provide comprehensive detection data, and traditional data processing and transmission methods lack flexibility and real-time performance. This limitation can severely impact rescue efficiency, especially in complex disaster environments. Therefore, addressing data transmission delays, sensor malfunctions, and misjudgments through multi-sensor data fusion, collaborative detection networks, and augmented reality interaction technologies has become a key challenge in current rescue technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multi-sensor fusion life detection system based on a handheld display and control terminal. This system solves the problems that existing technologies cannot provide comprehensive detection data from a single sensor, and that traditional data processing and transmission methods lack flexibility and real-time performance. In particular, in complex disaster environments, these limitations can seriously affect rescue efficiency.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multi-sensor fusion life detection system based on a handheld display and control terminal, comprising:

[0007] A sensor array is used to collect vital signs signals and environmental parameters. The sensor array includes an infrared thermal imager, an audio vibration sensor, a radar wave detector, and an environmental sensor.

[0008] A data processing module, connected to the sensor group, is used to perform noise filtering and feature extraction on the data collected by the sensor group.

[0009] A multi-source data fusion engine, connected to the data processing module, is used to perform fusion analysis on preprocessed multimodal data;

[0010] A handheld display and control terminal is connected to the multi-source data fusion engine to receive fusion results and display vital sign location information;

[0011] An edge collaboration network, connected to the handheld display and control terminal, is used to connect multiple handheld display and control terminals and drone nodes to achieve data sharing and collaborative analysis.

[0012] An augmented reality interaction module, integrated into the handheld display terminal, is used to overlay vital sign location information on the real-time screen and provide tactile feedback.

[0013] Preferably, the data processing module includes:

[0014] A noise filtering unit, connected to the sensor group, is used to remove sudden noise and smooth time-series data.

[0015] A cross-calibration unit, connected to the noise filtering unit, is used to correct the depth error of the infrared thermal imager and verify whether the heat source is a living organism.

[0016] The output of the cross-calibration unit is connected to the multi-source data fusion engine, and the processed data is transmitted to the multi-source data fusion engine.

[0017] Preferably, the multi-source data fusion engine includes a dynamic weight adjustment module, configured as follows:

[0018] It is connected to the data processing module to receive preprocessed data;

[0019] Real-time monitoring of environmental parameters, and dynamic allocation of weights for each sensor based on these parameters;

[0020] The output of the dynamic weight adjustment module is connected to the signal analysis unit, and outputs the weight allocation result.

[0021] Preferably, the multi-source data fusion engine further includes a signal analysis unit, configured as follows:

[0022] It connects to the dynamic weight adjustment module and receives the weight allocation results;

[0023] Joint analysis of multimodal data is performed to extract spatial and time series features;

[0024] By using a cross-modal attention mechanism, the temperature difference region of infrared thermal imaging is correlated with the heart rhythm characteristics of audio vibration signals;

[0025] The output of the signal analysis unit is connected to the handheld display and control terminal to output fused analysis data.

[0026] Preferably, the edge collaborative network is configured as follows:

[0027] It connects to the handheld display and control terminal to receive fused analysis data;

[0028] Connect multiple handheld display and control terminals and drone nodes, and cross-verify the detection results of multiple terminals;

[0029] The output of the edge collaborative network is connected to the handheld display and control terminal to output collaborative detection results.

[0030] Preferably, the augmented reality interaction module is configured as follows:

[0031] It connects to the handheld display and control terminal to receive collaborative detection results;

[0032] Vital signs location information is overlaid on the real-time image from the terminal camera;

[0033] Tactile feedback is generated based on the intensity of vital signs signals;

[0034] The output of the augmented reality interaction module is connected to the display unit of the handheld display terminal to display vital sign location information.

[0035] Preferably, the dynamic weight adjustment module uses evidence theory to fuse multi-sensor data, specifically including:

[0036] It is connected to the data processing module to receive preprocessed data;

[0037] A sensor reliability matrix is ​​constructed based on environmental parameters, and the fusion result is corrected using an evidence conflict detection algorithm.

[0038] The output of the dynamic weight adjustment module is connected to the signal analysis unit, and outputs the corrected fusion result.

[0039] Preferably, the signal analysis unit performs incremental updates through federated learning, and its configuration is as follows:

[0040] It connects to the multi-source data fusion engine to receive fused and analyzed data;

[0041] Multiple terminals share anonymized data and upload it to the cloud training platform;

[0042] Privacy protection technologies are employed to ensure data security, and local models are fine-tuned to adapt to different disaster scenarios.

[0043] The output of the signal analysis unit is connected to the multi-source data fusion engine, and outputs the updated feature extraction results.

[0044] Preferably, the radar wave detector is configured as follows:

[0045] It connects to the data processing module and outputs probe data;

[0046] Detecting chest cavity rise and fall signals in living organisms;

[0047] The output of the radar wave detector is connected to the noise filtering unit of the data processing module.

[0048] Preferably, the model update process of the federated learning includes:

[0049] It is connected to the signal analysis unit to receive feature extraction results;

[0050] Retain the feature extraction weights related to vital sign recognition in the global model parameters;

[0051] Ensure model generalization performance through parameter update constraint techniques;

[0052] The output of the federated learning module is connected to the signal analysis unit, and outputs the updated model parameters.

[0053] This invention provides a multi-sensor fusion life detection system based on a handheld display terminal. It has the following beneficial effects:

[0054] 1. This invention employs multi-sensor collaborative detection and data fusion technology, achieving real-time and accurate detection of vital signs in complex environments. Compared to existing single-sensor detection methods, this invention significantly improves the reliability and accuracy of detection at disaster sites through the complementary and fusion of data from multiple sensors, solving the detection blind spot problem of traditional methods under environmental noise and interference.

[0055] 2. This invention achieves real-time data sharing and collaboration among multiple terminals through an edge collaborative detection network. Using a mesh self-organizing network and the AODV protocol, each terminal can quickly transmit data, ensuring smooth information transmission. Compared to traditional centralized networks, this approach avoids single points of failure, greatly improving the system's adaptability and stability in disaster scenarios, and ensuring efficient data flow in complex and dynamic environments.

[0056] 3. This invention utilizes augmented reality (AR) technology to visualize detection results in real time, providing intuitive information feedback and safety guidance. Unlike traditional text or graphic annotations, this invention uses an AR system to directly overlay the detected target and risk area onto the on-site video footage, enabling rescue personnel to quickly identify high-risk areas and react accordingly. Through the combination of tactile feedback, on-site personnel can maintain efficient response under pressure, reducing operational delays caused by unclear information.

[0057] 4. This invention improves the system's adaptability and accuracy by dynamically adjusting sensor weights and fusion decisions. By continuously correcting the weights among sensors, this system can adjust data contributions in real time according to changes in the field environment. Compared with traditional fixed-weight solutions, this invention avoids misjudgments caused by sensor failures or environmental changes, providing a more flexible and reliable solution for detecting vital signs. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the overall system architecture of the present invention;

[0059] Figure 2 This is a flowchart of the data processing and fusion analysis process of the present invention;

[0060] Figure 3 This is a flowchart of the collaborative detection and augmented reality interaction process of the present invention. Detailed Implementation

[0061] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Please see the appendix Figure 1-3 This invention provides a multi-sensor fusion life detection system based on a handheld display terminal, comprising:

[0063] In the technical solution of this invention, sensor data acquisition, data processing, fusion analysis, edge computing, and interactive feedback all rely on a reasonable system architecture and reliable hardware support. Life detection systems in complex environments need to possess multimodal sensing, real-time computing, and human-computer interaction capabilities to adapt to different environmental conditions and ensure data accuracy and stability. This invention employs a multi-sensor collaborative working approach and is equipped with a high-performance data processing module to achieve accurate life detection.

[0064] The system's hardware architecture needs to meet the requirements of high integration, low power consumption, and high reliability. Especially in extreme environments such as earthquake ruins and fire scenes, the equipment should be shock-resistant, waterproof, and dustproof to ensure long-term stable operation. Specifically, the core hardware components of the system mainly include sensor groups, data processing units, edge computing terminals, wireless communication modules, and handheld display and control terminals.

[0065] In this embodiment, the sensor array is used to acquire vital sign data and environmental parameters of the target area, encompassing an infrared thermal imager, an audio vibration sensor, a radar wave detector, and an environmental sensor. These different types of sensors complement each other, improving the system's adaptability in complex scenarios.

[0066] Infrared thermal imagers are used to detect the thermal radiation signals of living organisms. Since the body temperature of living organisms is higher than the ambient temperature, infrared thermal imaging can detect the distribution of the target's temperature field within a certain distance. This infrared thermal imager uses an uncooled infrared detector with a resolution of 640×480 pixels and a frame rate of 30fps, with a temperature measurement range of -20℃ to 150℃ and a temperature difference recognition accuracy of ±0.1℃. The data output of the infrared thermal imager can include both the height and width of the image. To improve measurement accuracy, this system employs a temperature averaging filtering method.

[0067] ;

[0068] in, This is the filtered temperature value; The first infrared thermal imager to collect Frame temperature value; This is the size of the filtering window.

[0069] In this embodiment, an audio vibration sensor is used to detect weak low-frequency signals, including the heartbeat and respiration of a living organism. The typical frequency of a heartbeat is 0.5Hz-2Hz, and the frequency of a respiration signal is 0.1Hz-0.5Hz. A MEMS miniature vibration sensor is used, with a sensitivity of -40dB and a frequency response range of 5Hz-500Hz.

[0070] Specifically, audio vibration signals can be represented as:

[0071] ;

[0072] in, It is a vibration signal. The amplitude of vibration. For signal frequency This invention uses Short-Time Fourier Transform (STFT) to perform time-frequency analysis on audio signals, and the calculation formula is as follows:

[0073] ;

[0074] in, It is a time-frequency distribution; For window functions; For discrete-time indexing; For frequency components.

[0075] The radar wave detector uses a 24GHz millimeter-wave radar, which can penetrate obstacles to detect minute movements of targets. In this embodiment, the radar detection range can reach 0.5-10 meters, with an accuracy of ±1cm.

[0076] Radar wave detection is based on the Doppler effect, calculating the target's velocity by measuring the frequency shift caused by the target's motion.

[0077] ;

[0078] in, For target speed; The speed of light; This is the radar wave carrier frequency.

[0079] In this embodiment, radar waves are used to detect the micro-motion characteristics of the chest cavity, and the presence of respiratory signals in the target is determined by measuring periodic motion.

[0080] Environmental sensors are used to monitor external factors that affect life detection, such as temperature, humidity, electromagnetic interference, and noise levels. Environmental sensors include:

[0081] Temperature and humidity sensor: Measurement range -40℃ to 85℃, humidity range 0-100%RH.

[0082] Noise sensor: Detection range 30-130dB, used to assess the interference of ambient noise on audio sensors.

[0083] Barometric pressure sensor: Measurement range 300hPa-1100hPa, used to calculate elevation changes.

[0084] In this embodiment, the handheld display and control terminal is used to process, display, and interact with the probe data. Generally, the terminal needs to have high computing power and a good human-computer interaction experience to support real-time data processing and feedback. In this embodiment, the terminal is configured as follows:

[0085] It features a 7-inch touchscreen with a resolution of 1280×720 and supports touch control in glove mode.

[0086] Multi-core processor with a clock speed of ≥2GHz, supporting local computing and edge collaborative computing.

[0087] It has 4GB of RAM and 64GB of storage space, which can store long-term probe data and training model parameters.

[0088] With IP68 protection, it supports waterproof, dustproof and shock-resistant design, making it suitable for harsh environments.

[0089] The terminals communicate wirelessly via Wi-Fi 6, Bluetooth 5.0, and the Mesh self-organizing network protocol. Data sharing can be achieved between terminals, and multi-device collaborative detection is supported.

[0090] In this embodiment, the wireless communication module ensures smooth data transmission across all parts of the system. Wireless communication in complex environments requires interference resistance to guarantee stable data transmission. Mesh self-organizing network technology is employed, using the AODV protocol for multi-hop communication to ensure information synchronization among multiple terminals.

[0091] Specifically, the communication latency of a mesh network

[0092] The calculation formula is as follows:

[0093] ;

[0094] in, For the terminal To the terminal Data transmission delay; For the first The length of the data packets in the link; Link bandwidth; This represents the number of relay nodes along the transmission path.

[0095] In summary, the system architecture and hardware configuration in this example provide technical support for multi-sensor fusion, real-time computing, wireless collaboration, and human-computer interaction. Through collaborative detection by multiple sensors, dynamic data processing, and intelligent interaction, the system can achieve efficient vital sign detection in complex environments, providing technical support for emergency rescue.

[0096] Data Processing Module: In a multi-sensor fusion life detection system, the data processing module plays a crucial role. Its main task is to effectively preprocess data from various sensors, including noise removal and signal calibration, to ensure the accuracy and reliability of the data ultimately sent to subsequent modules. The effectiveness of data processing directly determines the detection accuracy and response speed of the entire system. Its core objective is to improve signal quality, filter unnecessary noise, and correct signals to ensure that subsequent fusion and analysis are based on accurate information.

[0097] The first key operation of the data processing module is noise filtering. Sensor data is often affected by ambient noise, causing signal distortion. To eliminate this unwanted interference, this system employs two main noise reduction techniques.

[0098] First, wavelet packet decomposition was used to remove high-frequency noise, such as mechanical vibration and environmental noise, from audio vibration sensor data. Through wavelet packet transform, the signal can be decomposed into multiple frequency bands, and high-frequency noise can be selectively removed based on these bands. Second, for radar waveform data, Kalman filtering was employed. Kalman filtering can adaptively adjust its noise estimation parameters, enabling better signal smoothing and preservation of original characteristics in dynamic environments.

[0099] Specifically, the state estimation formula for Kalman filtering is as follows:

[0100] ;

[0101] ;

[0102] in, For state variables; This is the state transition matrix; For control vectors; This is process noise; For observation vectors; The observation matrix; To observe noise.

[0103] Kalman filtering can smooth radar data and avoid interference from sudden data changes.

[0104] After noise filtering, the next step is cross-calibration. Alternatively, data from the infrared thermal imager and radar detector need to be cross-calibrated to ensure consistency and accuracy. For example, the temperature distribution acquired by the infrared thermal imager may be deviated from due to environmental interference (such as sunlight, flames, etc.), and the ranging data from the radar detector can be used to correct these errors. When the radar fails to detect periodic life movement, if an abnormal heat source (temperature difference greater than 1°C) appears in the infrared image, the system will automatically identify it as a non-living source and issue an alarm. This cross-calibration method ensures the accuracy of infrared thermal image and radar data under multiple interference conditions.

[0105] Multi-source data fusion engine: In this invention, the multi-source data fusion engine is responsible for intelligently fusing multimodal sensor data after preprocessing by the data processing module. Since the data from each sensor exhibits dynamic differences and environmental conditions are complex, the fusion engine aims to achieve automatic data adjustment and joint analysis, ensuring accurate and stable detection results. This module provides unified and clear detection information to the handheld display and control terminal, and is a crucial guarantee for the overall performance of the system.

[0106] In this embodiment, the multi-source data fusion engine first includes a dynamic weight adjustment module. Different sensors behave differently under environmental interference. The dynamic weight adjustment module collects environmental parameters in real time and constructs a sensor reliability matrix. The reliability index output by each sensor is defined as:

[0107] ;

[0108] in, Indicates sensor Reliability; This represents a vector of environmental parameters (such as temperature, noise, and electromagnetic interference). For sensors The signal quality metrics were then calculated. Subsequently, the sensor weights were calculated using a normalization method:

[0109] ;

[0110] in, For sensors Dynamic weights; This indicates the total number of sensors.

[0111] This formula ensures that the data contribution of each sensor is automatically adjusted in variable environments, reducing the impact of noise.

[0112] In this embodiment, an evidence fusion method is also used to handle data conflicts. The confidence levels output by each sensor are inconsistent. Each sensor... On the proposition The confidence level is expressed as a function This indicates that, combining the outputs of the two sensors, the formula for combining the evidence is:

[0113] ;

[0114] in, The conflict coefficient is defined as:

[0115] ;

[0116] Here, and These represent the subsets of sensor 1 and sensor 2, respectively. and The confidence level is determined. This formula can be extended to multiple sensors, ensuring that the fusion results are more reliable when different data conflict.

[0117] In this embodiment, the signal analysis unit employs a cross-modal attention mechanism to perform joint feature extraction on the fused data. Features captured by different sensors are complementary. By employing the cross-modal attention mechanism, the system can focus on key feature regions.

[0118] Let query matrix Key matrix and numerical matrix The formula for calculating attention is:

[0119] ;

[0120] in, For query matrix; The key matrix; It is a value matrix; denoted as the dimension of the key vector. This mechanism enables the system to automatically focus on biometric information in multimodal data such as infrared, radar, and audio during joint analysis, thereby improving the overall accuracy of feature extraction.

[0121] In this embodiment, the output of the multi-source data fusion engine includes target location information, confidence indices of contributions from each sensor, and joint feature information. This output is transmitted to a handheld display terminal via a wireless communication module for real-time display and feedback. The entire fusion process considers both environmental influences and the individual advantages of each sensor, achieving intelligent adjustment and efficient integration of data.

[0122] Overall, the multi-source data fusion engine in this embodiment achieves efficient joint analysis of multi-sensor data in complex environments through dynamic weight adjustment, evidence fusion, and cross-modal attention mechanisms.

[0123] Edge Collaborative Detection Network: In this invention, the edge collaborative detection network ensures rapid data sharing and processing through multi-terminal cooperation. By integrating multiple handheld display terminals and other mobile nodes, such as drones, the system can achieve comprehensive vital sign detection. In the aforementioned data processing and fusion stage, the data from each sensor undergoes preprocessing and intelligent analysis, and is ultimately detected in real-time by the edge computing network. The core task of this module is to provide an efficient multi-node collaborative platform to ensure the accuracy, real-time performance, and reliability of the detection results.

[0124] In this embodiment, the edge collaborative detection network employs Mesh self-organizing network technology, enabling multiple terminals and devices to flexibly connect and share data. Each terminal in the system possesses communication and computing capabilities, allowing for wireless communication with other nodes. Each terminal establishes a multi-hop network via the Mesh protocol, ensuring that even if some nodes are far apart, data can still be transmitted to the final destination through intermediate nodes. In this way, the system's communication range is effectively extended, ensuring its broad applicability at disaster sites.

[0125] Specifically, once a handheld display terminal completes a vital sign detection, it broadcasts the detection results, including the target's location information and detection confidence level, to other terminals in the network. Other terminals cross-verify the received data to confirm the target's authenticity. Typically, the Mesh network's communication protocol allows data sharing and collaboration among all terminals and drone nodes. Each terminal transmits data packets on demand via the AODV protocol. This ensures real-time synchronization of detection information across nodes, avoiding the single point of failure problem inherent in traditional centralized network architectures.

[0126] In this embodiment, the collaborative localization algorithm between nodes is based on the principle of triangulation. When multiple terminals detect the same target, they will locate it from different directions. By calculating the phase difference or signal strength difference between these nodes, the system can accurately calculate the target's position. The triangulation method can provide high positioning accuracy, with a positioning error of less than 0.5 meters in most application scenarios.

[0127] Assuming nodes , and They are located at known positions and are respectively relative to the target point. The distances are respectively , and Target point coordinates It can be calculated using the following system of equations:

[0128] ;

[0129] ;

[0130] ;

[0131] These equations form the basis of triangulation, which calculates the precise location of a target using three known points and their corresponding distances.

[0132] For target location confirmation, this system employs a confidence-based voting mechanism. That is, if more than 70% of the nodes confirm the existence of the target, the system considers the target's vital signs valid. This mechanism significantly improves the system's accuracy and robustness in complex environments.

[0133] For each detected vital sign, the system confirms its existence through a confidence-based voting mechanism among nodes. If more than 70% of the nodes confirm the target's existence, the system considers the target valid. The formula can be expressed as:

[0134] ;

[0135] in, It is the first The confidence value of each node for target T; This is the number of nodes participating in the vote. When If the value is greater than a certain set threshold (such as 0.7), the target is considered valid.

[0136] Furthermore, the edge collaborative detection network offers powerful data fusion and multi-terminal collaboration capabilities. Each node can share local detection data in real time, and other nodes can perform real-time corrections based on the received data to ensure the accuracy and consistency of the final report. Under this dynamic collaborative framework, data exchange between multiple terminals not only improves system efficiency but also avoids detection blind spots caused by single device failure or data loss.

[0137] Edge collaborative detection networks fully leverage the advantages of multi-terminal collaboration. At disaster sites, multiple handheld display terminals and drone nodes can work together rapidly, significantly improving the accuracy and response speed of vital sign detection. Compared to traditional single-node detection modes, this invention significantly enhances the system's adaptability and reliability through multi-terminal collaborative operation.

[0138] In practical applications, this network offers high flexibility in configuration and operation. The number of terminals and drone nodes can be dynamically adjusted according to actual needs, ensuring efficient operation of the system at disaster sites of varying scales. Furthermore, the system can be further optimized in its network topology to improve data transmission efficiency and collaboration accuracy based on different scenario requirements.

[0139] Through the above methods, the edge collaborative detection network in this embodiment ensures efficient collaboration and accurate positioning of the entire detection system in disaster relief, solves the limitations and bottlenecks of traditional single sensor and centralized processing modes, and enhances the stability and adaptability of the system in complex environments.

[0140] Augmented Reality Interaction: In this invention, the augmented reality interaction module presents the detection results obtained after data processing, fusion, and collaborative positioning to the site in an intuitive manner using graphics, icons, and feedback, compensating for the information deficiencies of traditional display methods. Seamless data integration between modules provides real-time assistance for on-site rescue. This module works closely with the aforementioned data processing and network collaboration components to ensure efficient data flow from detection to presentation.

[0141] In this embodiment, the augmented reality interaction module first captures live video using a camera and spatially registers the sensor data. Generally, virtual data must precisely match the real scene. As an option, this module uses the projection transformation formula:

[0142] ;

[0143] in, Indicates the target's position in the world coordinate system; The model-view matrix; The projection matrix; Here are the projected coordinates of the target on the display screen. This formula ensures accurate alignment of the digital information with the surrounding environment.

[0144] In this embodiment, a layered overlay technique is also used to achieve graphic display. The display layers are divided into background, target markers, and warning icons. The processed data is overlaid on the real-time video as colored areas or icons. Red markers are used for high-confidence targets, and yellow markers are used for targets that need to be confirmed. This method facilitates on-site operators to quickly identify key information.

[0145] In this embodiment, the augmented reality interaction module also provides haptic feedback. The detection results are not only presented visually but also accompanied by vibration alerts via a handheld terminal. Different vibration modes correspond to different confidence levels: low-confidence targets receive slight vibrations, while high-confidence targets receive strong feedback. This feedback mechanism helps rescue personnel quickly assess the importance of the target. The augmented reality interaction module synchronously processes real-time video data, sensor data, and location information.

[0146] The system utilizes a unified timestamp to enable multi-data source linkage, ensuring that all information is presented almost simultaneously, meeting the demands for high-speed on-site response. The augmented reality interaction module employs technologies such as real-time video capture, virtual-real data overlay, projection transformation, graphic layering, and haptic feedback to achieve an intuitive display of detection results. This module effectively overcomes the limitations of traditional display methods, improving the efficiency and accuracy of on-site decision-making.

[0147] Example 1:

[0148] Scenario Description: Following a powerful earthquake, large-scale building collapses occurred in multiple areas of the city. The disaster site environment is extremely complex, and there may be trapped people within the rubble. To efficiently conduct life detection and determine the location of trapped individuals, rescue teams deployed a multi-sensor detection system.

[0149] Implementation steps:

[0150] Step 1: Sensor Setup and Data Acquisition

[0151] Infrared thermal imagers, millimeter-wave radars, and audio vibration sensors were deployed at key locations in the ruins.

[0152] Infrared thermal imagers are responsible for capturing the temperature distribution of the surrounding environment, especially possible signs of life; millimeter-wave radar detects the movement of tiny targets; and audio vibration sensors are used to sense the heartbeat and breathing of living organisms.

[0153] Sensors collect data in real time and transmit it to nearby handheld terminals via wireless network. Low data transmission latency ensures that rescuers receive feedback quickly.

[0154] Step 2: Data Processing and Fusion

[0155] The data collected by each sensor will be processed by the data processing module to reduce noise and correct signals, removing external environmental noise and interference.

[0156] The data fusion engine analyzes data from infrared thermal imaging, audio, and radar, and performs a comprehensive evaluation based on sensor weights. Weights are automatically adjusted to ensure the accuracy of data contributions in the face of environmental changes.

[0157] Edge collaborative detection networks ensure rapid data sharing and collaborative operation among multiple handheld terminals, further improving the accuracy of target positioning.

[0158] Step 3: Target Positioning and Confirmation:

[0159] Using multi-node positioning algorithms (such as triangulation), the possible location of trapped personnel can be determined through multiple sensors.

[0160] Through a confidence-based voting mechanism, if more than 70% of the terminals confirm that a certain area is the location of vital signs, then that location is determined to be the target.

[0161] Step 4: Augmented Reality Interaction and Display

[0162] The augmented reality system overlays the real-time detected target location information with on-site video images, displaying the marker in the trapped area to alert rescue personnel.

[0163] Through the graphical interface on the handheld device, rescuers can clearly see the specific location and confidence level of each target, and receive further prompts through haptic feedback.

[0164] Step 5: Rescue Execution

[0165] Based on information provided by the augmented reality interaction system, rescuers acted quickly, prioritizing locations with confirmed strong vital signs for rescue.

[0166] Example 2:

[0167] Scenario Description: During a fire, the temperature inside the fire scene is extremely high and the smoke is dense, making it difficult for traditional detection methods to penetrate the smoke and flames. In this situation, it is crucial to use multi-sensor collaborative detection technology to detect vital signs and combine it with augmented reality technology to provide real-time safety guidance to on-site rescue personnel.

[0168] Implementation steps:

[0169] Step 1: Sensor Setup and Data Acquisition

[0170] Infrared thermal imagers, audio vibration sensors, and gas detectors were deployed at the fire scene.

[0171] Infrared thermal imagers are used to detect temperature differences, especially the thermal radiation signals of living organisms; audio vibration sensors are used to capture heartbeats or breathing sounds; gas detectors monitor the concentration of harmful gases.

[0172] These sensors transmit data wirelessly to the command center and rescue personnel's handheld devices, ensuring information synchronization.

[0173] Step 2: Data Processing and Fusion

[0174] The data processing module performs real-time noise filtering and signal processing to remove interference signals caused by flames and smoke.

[0175] Through a multi-source data fusion engine, the system combines infrared, audio, and gas detection data to assess the confidence level of each sensor in the presence of the target.

[0176] Edge collaborative detection networks ensure that all sensors and devices on site work together in real time, guaranteeing stable transmission of information.

[0177] Step 3: Target Identification and Localization

[0178] The system analyzes the combined information from various sensors to determine areas where vital signs may exist.

[0179] The augmented reality system marks the locations of high-confidence vital signs in red and displays them on the device screen in real time.

[0180] Step 4: Augmented Reality Interaction and Evacuation Guidance

[0181] Rescuers used augmented reality equipment to see a superimposed display of real-time video footage and target locations, marking out danger zones and safe evacuation routes.

[0182] If a high-risk area is detected, the augmented reality system will highlight it in red and alert people to avoid it as soon as possible through vibration feedback.

[0183] Step 5: Rescue and Evacuation

[0184] Based on AR prompts, rescuers prioritize entering areas with visible vital signs for rescue operations, while ensuring the rapid and safe evacuation of other personnel.

[0185] Example 3:

[0186] Scenario Description: In the event of a gas leak at a chemical plant, the site faces not only the potential risk of explosion but also the hazard of toxic gas leakage. To accurately identify risks and guide rescue efforts, the system combines temperature, gas, and audio sensors for multimodal detection, while employing augmented reality technology for on-site risk visualization.

[0187] Implementation steps:

[0188] Step 1: Sensor Setup and Data Acquisition

[0189] Temperature sensors, gas detectors, vibration sensors, etc. are installed at key locations in the chemical plant.

[0190] Temperature sensors are used to monitor temperature changes in an area, gas detectors are used to monitor the leakage of harmful gases in real time, and vibration sensors monitor structural changes in equipment or pipelines.

[0191] Step 2: Data Processing and Fusion

[0192] The data processing module processes the raw data acquired from each sensor, including noise filtering, signal correction, and data synchronization.

[0193] The multi-source data fusion engine integrates data from various sensors and uses an automatic weight adjustment method to ensure that each sensor provides the most accurate data under different environments.

[0194] Edge collaborative detection networks ensure rapid synchronization and sharing of data across multiple devices, avoiding information lag.

[0195] Step 3: Risk Assessment and Positioning

[0196] The system analyzes data from multimodal sensors to assess on-site gas leakage concentration, temperature changes, and vibration conditions, and performs real-time risk assessment.

[0197] Based on the analysis results, the system will mark high-risk areas and verify dangerous areas through a confidence-based voting mechanism.

[0198] Step 4: Augmented Reality Interaction and Risk Visualization:

[0199] The augmented reality system overlays real-time sensor data and risk assessment results, marking hazardous areas of toxic gas leaks and displaying safe evacuation routes.

[0200] Rescuers can see a clear AR interface through handheld devices, understand the on-site environment, risk areas and the location of vital signs, and receive tactile feedback.

[0201] Step 5: Decision-making and Action

[0202] Based on the real-time information displayed on the AR interface, the command center and on-site personnel made rapid decisions and guided rescuers to avoid dangerous areas and quickly enter the areas in need of assistance to carry out rescue operations.

[0203] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-sensor fusion life detection system based on a handheld display and control terminal, characterized in that, include: A sensor array is used to collect vital signs signals and environmental parameters. The sensor array includes an infrared thermal imager, an audio vibration sensor, a radar wave detector, and an environmental sensor. A data processing module, connected to the sensor group, is used to perform noise filtering and feature extraction on the data collected by the sensor group. A multi-source data fusion engine, connected to the data processing module, is used to perform fusion analysis on preprocessed multimodal data; A handheld display and control terminal is connected to the multi-source data fusion engine to receive fusion results and display vital sign location information; An edge collaboration network, connected to the handheld display and control terminal, is used to connect multiple handheld display and control terminals and drone nodes to achieve data sharing and collaborative analysis. An augmented reality interaction module, integrated into the handheld display terminal, is used to overlay vital sign location information on the real-time screen and provide tactile feedback; The data processing module includes: A noise filtering unit, connected to the sensor group, is used to remove sudden noise and smooth time-series data. A cross-calibration unit, connected to the noise filtering unit, is used to correct the depth error of the infrared thermal imager and verify whether the heat source is a living organism. The output of the cross-calibration unit is connected to the multi-source data fusion engine, and the processed data is transmitted to the multi-source data fusion engine. The multi-source data fusion engine includes a dynamic weight adjustment module, which is configured as follows: It is connected to the data processing module to receive preprocessed data; Real-time monitoring of environmental parameters, and dynamic allocation of weights for each sensor based on these parameters; The output of the dynamic weight adjustment module is connected to the signal analysis unit, and outputs the weight allocation result. The multi-source data fusion engine also includes a signal analysis unit, which is configured as follows: It connects to the dynamic weight adjustment module and receives the weight allocation results; Joint analysis of multimodal data is performed to extract spatial and time series features; By using a cross-modal attention mechanism, the temperature difference region of infrared thermal imaging is correlated with the heart rhythm characteristics of audio vibration signals; The output of the signal analysis unit is connected to the handheld display and control terminal, and outputs fused analysis data. The multi-source data fusion engine first includes a dynamic weight adjustment module; different sensors perform differently under environmental interference; the dynamic weight adjustment module collects environmental parameters in real time and constructs a sensor reliability matrix; the reliability index output by each sensor is defined as: ； in, This indicates the reliability of sensor i. For environmental parameter vectors, For sensor i, the signal quality index is used. Subsequently, the sensor weights are calculated using a normalization method: ； in, The dynamic weight of sensor i; Indicates the total number of sensors; Ensure that the data contribution of each sensor is automatically adjusted in variable environments to reduce the impact of noise; Evidence fusion is used to handle data conflicts; inconsistencies exist in the confidence levels output by different sensors; the confidence level of each sensor i for proposition A is expressed as a function. This indicates that, combining the outputs of the two sensors, the formula for combining the evidence is: ； in, The conflict coefficient is defined as: ； Here, and These represent the confidence levels of sensor 1 and sensor 2 for subsets B and C, respectively; they can be generalized to multiple sensors, ensuring that the fusion results are more reliable when different data conflict. The signal analysis unit employs a cross-modal attention mechanism to perform joint feature extraction on the fused data; the features captured by different sensors are complementary; after adopting the cross-modal attention mechanism, the system can focus on key feature regions; Let query matrix Key matrix and numerical matrix The formula for calculating attention is: ； in, The dimension of the key vector; this mechanism enables the system to automatically focus on biometric information in multimodal data such as infrared, radar, and audio during joint analysis, thereby improving the accuracy of overall feature extraction; The edge collaboration network is configured as follows: It connects to the handheld display and control terminal to receive fused analysis data; Connect multiple handheld display and control terminals and drone nodes, and cross-verify the detection results of multiple terminals; The output of the edge collaboration network is connected to the handheld display and control terminal, and outputs the collaborative detection results. The augmented reality interaction module is configured as follows: It connects to the handheld display and control terminal to receive collaborative detection results; Vital signs location information is overlaid on the real-time image from the terminal camera; Tactile feedback is generated based on the intensity of vital signs signals; The output of the augmented reality interaction module is connected to the display unit of the handheld display terminal to display vital sign location information; The augmented reality interaction module uses the projection transformation formula: ； in, Indicates the target's position in the world coordinate system; The model-view matrix; The projection matrix; The formula represents the projected coordinates of the target on the display screen, ensuring accurate alignment of digital information with the surrounding environment. It also employs layered overlay technology to achieve graphic display; the display layers are divided into background, target markers, and warning icons; the processed data is overlaid in the real-time video as colored areas or icons; red markers are used for high-confidence targets, and yellow markers are used for targets to be confirmed; this method facilitates on-site operators to quickly identify key information; The dynamic weight adjustment module uses evidence theory to fuse multi-sensor data, specifically including: It is connected to the data processing module to receive preprocessed data; A sensor reliability matrix is ​​constructed based on environmental parameters, and the fusion result is corrected using an evidence conflict detection algorithm. The output of the dynamic weight adjustment module is connected to the signal analysis unit, and outputs the corrected fusion result. The signal analysis unit performs incremental updates through federated learning, and its configuration is as follows: It connects to the multi-source data fusion engine to receive fused and analyzed data; Multiple terminals share anonymized data and upload it to the cloud training platform; Privacy protection technologies are employed to ensure data security, and local models are fine-tuned to adapt to different disaster scenarios. The output of the signal analysis unit is connected to the multi-source data fusion engine, and outputs the updated feature extraction results. The radar wave detector is configured as follows: It connects to the data processing module and outputs probe data; Detecting chest cavity rise and fall signals in living organisms; The output terminal of the radar wave detector is connected to the noise filtering unit of the data processing module; The model update process of the federated learning includes: It is connected to the signal analysis unit to receive feature extraction results; Retain the feature extraction weights related to vital sign recognition in the global model parameters; Ensure model generalization performance through parameter update constraint techniques; The output of the federated learning module is connected to the signal analysis unit, and outputs the updated model parameters.

Images