Multi-sensor fusion life detection system based on handheld display and control terminal

Abstract

The invention relates to the technical field of life detection, and discloses a multi-sensor fusion life detection system based on a handheld display control terminal, and the system comprises a sensor group which is used for collecting vital sign signals and environmental parameters, and comprises an infrared thermal imager, an audio vibration sensor, a radar wave detector, and an environmental sensor; and the data processing module is connected with the sensor group and is used for carrying out noise filtering and feature extraction on the data acquired by the sensor group. According to the invention, a multi-sensor cooperative detection and data fusion technology is adopted, and the technical effect of real-time and accurate detection of vital signs in a complex environment is achieved. Compared with a detection method of a single sensor in the prior art, the method can remarkably improve the reliability and accuracy of detection through data complementation and fusion of a plurality of sensors in a disaster site, and solves the problem of detection blind areas of a traditional method under environmental noise and interference.

Description

Technical Field

[0001] The present invention relates to the technical field of life detection, and in particular to a multi-sensor fusion life detection system based on a handheld display and control terminal. Background Art

[0002] With the acceleration of urbanization, the complexity of disaster rescue missions continues to increase. After a disaster, timely and effective life detection and rescue work is crucial. However, traditional detection technologies often face problems such as environmental noise, sensor failure, and detection blind spots, resulting in low search and rescue efficiency and increased loss of life. Therefore, how to accurately and real-time locate trapped people in complex environments has become a key issue in improving rescue efficiency.

[0003] At present, common technical solutions in the rescue field include single sensor detection methods, such as infrared thermal imagers for temperature detection and audio sensors for detecting vital signs. However, these methods usually rely on the judgment of a single sensor, lack data complementarity, and are prone to misjudgment or missed judgment in complex environments. In addition, most existing technologies adopt a centralized data processing mode, which leads to limited information transmission and the risk of single point failure.

[0004] The main problem with existing technologies is that a single sensor cannot provide comprehensive detection data, and traditional data processing and transmission methods lack flexibility and real-time performance. This limitation will seriously affect rescue efficiency, especially in complex disaster environments. Therefore, how to solve the problems of data transmission delay, sensor failure and misjudgment through multi-sensor data fusion, collaborative detection network and augmented reality interaction technology has become a key challenge in current rescue technology. Summary of the invention

[0005] In view of the shortcomings of the existing technology, the present invention provides a multi-sensor fusion life detection system based on a handheld display and control terminal, which solves the problem that a single sensor in the existing technology cannot provide comprehensive detection data, and the traditional data processing and transmission methods lack flexibility and real-time performance. Especially in complex disaster environments, this limitation will seriously affect the rescue efficiency.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a multi-sensor fusion life detection system based on a handheld display and control terminal, comprising: A sensor group, used to collect vital signs signals and environmental parameters, the sensor group includes an infrared thermal imager, an audio vibration sensor, a radar wave detector and an environmental sensor; A data processing module, connected to the sensor group, for performing noise filtering and feature extraction on the data collected by the sensor group; A multi-source data fusion engine, connected to the data processing module, for fusion analysis of the pre-processed multi-modal data; A handheld display and control terminal is connected to the multi-source data fusion engine and is used to receive fusion results and display vital sign positioning information; An edge collaborative network, connected to the handheld display and control terminal, is used to connect multiple handheld display and control terminals and drone nodes to achieve data sharing and collaborative analysis; The augmented reality interaction module is integrated in the handheld display and control terminal, and is used to overlay vital sign positioning information on the real-time image and provide tactile feedback.

[0007] Preferably, the data processing module includes: A noise filtering unit, connected to the sensor group, for removing burst noise and performing smoothing on time series data; a cross calibration unit, connected to the noise filtering unit, for correcting the depth error of the infrared thermal imager and verifying whether the heat source is a biological body; The output end of the cross calibration unit is connected to the multi-source data fusion engine to transmit the processed data to the multi-source data fusion engine.

[0008] Preferably, the multi-source data fusion engine includes a dynamic weight adjustment module, which is configured as follows: Connecting to the data processing module to receive pre-processed data; Monitor environmental parameters in real time and dynamically assign weights to each sensor based on the environmental parameters; The output end of the dynamic weight adjustment module is connected to the signal analysis unit to output the weight distribution result.

[0009] Preferably, the multi-source data fusion engine further includes a signal analysis unit, which is configured as follows: Connect with the dynamic weight adjustment module to receive the weight distribution result; Jointly analyze multimodal data to extract spatial and time series features; The temperature difference area of ​​infrared thermal imaging and the heart rhythm characteristics of audio vibration signals are associated through a cross-modal attention mechanism; The output end of the signal analysis unit is connected to the handheld display and control terminal to output fusion analysis data.

[0010] Preferably, the edge collaboration network is configured as follows: Connecting with the handheld display and control terminal to receive fusion analysis data; Connect multiple handheld display and control terminals and drone nodes, and cross-verify the detection results of multiple terminals; The output end of the edge collaborative network is connected to the handheld display and control terminal to output collaborative detection results.

[0011] Preferably, the augmented reality interaction module is configured as follows: Connecting with the handheld display and control terminal to receive collaborative detection results; Overlay vital signs positioning information on the real-time image of the terminal camera; generating tactile feedback based on the strength of vital sign signals; The output end of the augmented reality interaction module is connected to the display unit of the handheld display and control terminal to display vital sign positioning information.

[0012] Preferably, the dynamic weight adjustment module uses evidence theory to fuse multi-sensor data, specifically including: Connecting to the data processing module to receive pre-processed data; Construct a sensor reliability matrix based on environmental parameters and correct the fusion results through evidence conflict detection algorithm; The output end of the dynamic weight adjustment module is connected to the signal analysis unit to output the corrected fusion result.

[0013] Preferably, the signal analysis unit is incrementally updated through federated learning, and its configuration is as follows: Connecting to the multi-source data fusion engine to receive fusion analysis data; Multiple terminals share anonymized data and upload it to the cloud training platform; Privacy protection technology is used to ensure data security, and local models are fine-tuned to adapt to different disaster scenarios; The output end of the signal analysis unit is connected to the multi-source data fusion engine to output the updated feature extraction result.

[0014] Preferably, the radar wave detector is configured as follows: Connecting to the data processing module to output detection data; Detect chest rise and fall signals of living organisms; The output end of the radar wave detector is connected to the noise filtering unit of the data processing module.

[0015] Preferably, the model updating process of the federated learning includes: Connecting to the signal analysis unit to receive feature extraction results; The feature extraction weights related to vital sign recognition in the global model parameters are retained; Ensure model generalization performance through parameter update restriction technology; The output end of the federated learning module is connected to the signal analysis unit to output updated model parameters.

[0016] The present invention provides a multi-sensor fusion life detection system based on a handheld display and control terminal. It has the following beneficial effects: 1. The present invention adopts multi-sensor collaborative detection and data fusion technology to achieve the technical effect of real-time and accurate detection of vital signs in complex environments. Compared with the detection method of a single sensor in the prior art, the present invention can significantly improve the reliability and accuracy of detection through data complementation and fusion of multiple sensors at the disaster site, and solve the problem of detection blind spots of traditional methods under environmental noise and interference.

[0017] 2. The present invention realizes real-time data sharing and collaboration among multiple terminals through the edge collaborative detection network. Through the Mesh self-organizing network and AODV protocol, each terminal can quickly transmit data, ensuring smooth information transmission. Compared with the traditional centralized network, this method avoids the problem of single point failure, greatly improves the adaptability and stability of the system at the disaster site, and ensures that data can flow efficiently in a complex and dynamic environment.

[0018] 3. The present invention uses augmented reality technology to visualize the detection results in real time, providing intuitive information feedback and safety guidance. Unlike traditional text or graphic annotations, the present invention uses the AR system to directly superimpose the detection targets and risk areas on the on-site video screen, allowing rescue personnel to quickly identify high-risk areas and respond. Through the combination of tactile feedback, on-site personnel can maintain efficient response in stressful environments, reducing operational delays caused by unclear information.

[0019] 4. The present invention improves the system's adaptability and accuracy by dynamically adjusting sensor weights and fusion decisions. By continuously correcting the weights between sensors, the system can adjust data contributions in real time according to changes in the on-site environment. Compared with the traditional fixed weight technical solution, the present invention avoids misjudgments caused by sensor failure or environmental changes, and provides a more flexible and reliable solution for vital sign detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the overall system architecture of the present invention; Figure 2 It is a flow chart of data processing and fusion analysis of the present invention; Figure 3 This is a flowchart of the collaborative detection and augmented reality interaction of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Please refer to the attached Figure 1-3 The embodiment of the present invention provides a multi-sensor fusion life detection system based on a handheld display and control terminal, comprising: In the technical solution of the present invention, sensor data acquisition, data processing, fusion analysis, edge computing and interactive feedback all rely on a reasonable system architecture and reliable hardware support. The life detection system in a complex environment needs to have multimodal sensing, real-time computing and human-computer interaction capabilities to adapt to different environmental conditions and ensure the accuracy and stability of the data. The present invention adopts a collaborative working mode of multiple sensors and is equipped with a high-performance data processing module to achieve accurate life detection.

[0023] The hardware structure of the system needs to meet the requirements of high integration, low power consumption and high reliability. Especially in extreme environments such as earthquake ruins and fire scenes, the equipment should be impact-resistant, waterproof and dustproof to ensure long-term stable operation. Specifically, the core hardware of the system mainly includes sensor groups, data processing units, edge computing terminals, wireless communication modules and handheld display and control terminals.

[0024] In this embodiment, the sensor group is used to obtain vital sign data and environmental parameters of the target area, including infrared thermal imagers, audio vibration sensors, radar wave detectors and environmental sensors. Different types of sensors complement each other to improve the system's adaptability in complex scenarios.

[0025] Infrared thermal imagers are used to detect thermal radiation signals of living organisms. The body temperature of living organisms is higher than the ambient temperature, so infrared thermal imaging can detect the distribution of the target temperature field within a certain distance range. The infrared thermal imager uses an uncooled infrared detector with a resolution of 640×480 pixels and a frame rate of 30fps. The temperature measurement range is -20℃ to 150℃, and the temperature difference recognition accuracy reaches ±0.1℃. The data output form of the infrared thermal imager can be the height and width of the image. In order to improve the measurement accuracy, this system uses the temperature mean filtering method: ; in, is the temperature value after filtering; The infrared thermal imager collected Frame temperature value; is the filter window size.

[0026] In this embodiment, the audio vibration sensor is used to detect low-frequency weak signals, including the heartbeat and breathing of living organisms. The typical frequency of heartbeat is 0.5Hz-2Hz, and the frequency of breathing signal is 0.1Hz-0.5Hz. A MEMS micro vibration sensor is used, with a sensitivity of -40dB and a frequency response range of 5Hz-500Hz.

[0027] Specifically, the audio vibration signal can be expressed as: ; in, is the vibration signal, is the vibration amplitude, is the signal frequency The present invention uses short-time Fourier transform (STFT) to perform time-frequency analysis on the audio signal, and the calculation formula is as follows: ; in, is the time-frequency distribution; is the window function; is the discrete time index; is the frequency component.

[0028] The radar wave detector uses a 24GHz millimeter wave radar, which can penetrate obstacles to detect the slight movement of the target. In this embodiment, the radar detection distance can reach 0.5-10 meters, with an accuracy of ±1cm.

[0029] The principle of radar wave detection is based on the Doppler effect, which calculates the target's speed by measuring the frequency shift caused by the target's movement: ; in, is the target speed; is the speed of light; is the radar wave carrier frequency.

[0030] In this embodiment, radar waves are used to detect micro-motion characteristics of the chest cavity, and by measuring periodic motion, it is determined whether the target has a breathing signal.

[0031] Environmental sensors are used to monitor external factors that affect life detection, such as temperature and humidity, electromagnetic interference, and noise levels. Environmental sensors include: Temperature and humidity sensor: measurement range -40℃ to 85℃, humidity range 0-100%RH.

[0032] Noise sensor: detection range 30-130dB, used to evaluate the interference of environmental noise on audio sensors.

[0033] Air pressure sensor: Measuring range 300hPa-1100hPa, used to calculate elevation changes.

[0034] In this embodiment, the handheld display and control terminal is used to process, display and interact with the detection data. In general, the terminal needs to have high computing power and good human-computer interaction experience to support real-time processing and feedback of data. In this embodiment, the terminal is configured as follows: 7-inch touch screen, resolution 1280×720, supports glove mode touch.

[0035] Multi-core processor, main frequency ≥ 2GHz, supports local computing and edge collaborative computing.

[0036] 4GB memory and 64GB storage space can store long-term detection data and training model parameters.

[0037] IP68 protection, supports waterproof, dustproof and impact-resistant design, suitable for harsh environments.

[0038] The terminals communicate wirelessly via Wi-Fi6, Bluetooth 5.0, and Mesh self-organizing network protocols. Data can be shared between terminals, and multi-device collaborative detection is supported.

[0039] In this embodiment, the wireless communication module is used to ensure smooth data transmission in various parts of the system. Wireless communication in complex environments needs to have anti-interference capabilities to ensure stable data transmission. Mesh self-organizing network technology is used to perform multi-hop communication through the AODV protocol to ensure information synchronization between multiple terminals.

[0040] Specifically, the communication delay of the Mesh network The calculation formula is as follows: ; in, For Terminal To the terminal Data transmission delay; For the The length of the data packet on the link; is the link bandwidth; is the number of relay nodes on the transmission path.

[0041] In summary, the system structure and hardware configuration in this example provide technical support for multi-sensor fusion, real-time computing, wireless collaboration, and human-computer interaction. Through the collaborative detection of multiple sensors, dynamic data processing, and intelligent interaction, the system can achieve efficient vital sign detection in complex environments and provide technical support for emergency rescue.

[0042] Data processing module: In the multi-sensor fusion life detection system, the data processing module plays a vital role. Its main task is to effectively pre-process the data from each sensor, including noise removal, signal calibration and other steps, to ensure that the data sent to the subsequent modules is accurate and reliable. The effect of data processing directly determines the detection accuracy and response speed of the entire system. Its core goal is to improve the quality of the signal, filter unnecessary noise, and correct the signal to ensure that the subsequent fusion and analysis work can be based on accurate information.

[0043] The first key operation of the data processing module is noise filtering. Sensor data is often affected by environmental noise, resulting in signal distortion. In order to eliminate this unnecessary interference, this system uses two main noise processing techniques.

[0044] First, wavelet packet decomposition is used to remove high-frequency noise, such as mechanical vibration and environmental noise, from audio vibration sensor data. Through wavelet packet transform, the signal can be decomposed into multiple frequency bands, and then high-frequency noise can be selectively removed according to the frequency band. Secondly, for radar waveform data, Kalman filtering technology is used. Kalman filtering can adaptively adjust its noise estimation parameters, so that it can better smooth the signal and maintain its original characteristics in a dynamic environment.

[0045] Specifically, the state estimation formula of Kalman filter is: ; ; in, is the state variable; is the state transfer matrix; is the control vector; is the process noise; is the observation vector; is the observation matrix; is the observation noise.

[0046] Through Kalman filtering, radar data can be smoothed to avoid interference from sudden changes in data.

[0047] After noise filtering, the next step is cross-calibration. As an option, the data from the infrared thermal imager and the radar wave detector need to be calibrated against each other to ensure the consistency and accuracy of the data. For example, the temperature distribution obtained by the infrared thermal imager may be deviated due to environmental interference (such as sunlight, flames, etc.), and the ranging data of the radar wave detector can be used to correct these errors. When the radar fails to detect the periodic movement of life, if an abnormal heat source appears in the infrared image (temperature difference greater than 1°C), the system will automatically determine it as a non-life source and issue an alarm. This cross-calibration method ensures the accuracy of infrared thermal imaging and radar data under multiple interference conditions.

[0048] Multi-source data fusion engine: In the present invention, the multi-source data fusion engine is responsible for intelligently fusing the multi-modal sensor data pre-processed by the data processing module. There are dynamic differences in the data of each sensor and the environmental conditions are complex. The fusion engine aims to achieve automatic adjustment and joint analysis of the data to ensure accurate and stable detection results. This module provides unified and clear detection information for the handheld display and control terminal, which is an important guarantee for the overall performance of the system.

[0049] In this embodiment, the multi-source data fusion engine first includes a dynamic weight adjustment module. Different sensors behave differently under environmental interference. The dynamic weight adjustment module collects environmental parameters in real time and builds a sensor reliability matrix. The reliability index of each sensor output is defined as: ; in, Indicates sensor Reliability; is the environmental parameter vector (such as temperature, noise, electromagnetic interference); For sensor The signal quality indicator is then calculated by normalization: ; in, For sensor Dynamic weight of Indicates the total number of sensors.

[0050] This formula ensures that the data contribution of each sensor is automatically adjusted in a changing environment to reduce the impact of noise.

[0051] In this embodiment, the evidence fusion method is also used to deal with data conflicts. There is inconsistency in the confidence of each sensor output. Topic The confidence level is a function of Represents. Combining the outputs of the two sensors, the evidence combination formula is: ; in, is the conflict coefficient, defined as: ; Here, and Respectively represent the subsets of sensor 1 and sensor 2 and This formula can be extended to multiple sensors to ensure that the fusion result is more reliable when there is conflict between different data.

[0052] In this embodiment, the signal analysis unit adopts a cross-modal attention mechanism to perform joint feature extraction on the fused data. The features captured by different sensors are complementary. After adopting the cross-modal attention mechanism, the system can focus on key feature areas.

[0053] Let the query matrix , key matrix And the numerical matrix ,The attention calculation formula is: ; in, is the query matrix; is the key matrix; is the value matrix; is the dimension of the key vector. This mechanism enables the system to automatically focus on the biometric information in multimodal data such as infrared, radar, and audio during joint analysis, thereby improving the accuracy of overall feature extraction.

[0054] In this embodiment, the output of the multi-source data fusion engine includes target location information, confidence indicators contributed by each sensor, and joint feature information. The output is transmitted to the handheld display and control terminal through the wireless communication module to achieve real-time display and feedback. The entire fusion process takes into account both environmental impact and the advantages of each sensor, achieving intelligent adjustment and efficient integration of data.

[0055] In general, the multi-source data fusion engine in this embodiment realizes efficient joint analysis of multi-sensor data in complex environments through dynamic weight adjustment, evidence fusion and cross-modal attention mechanism.

[0056] Edge collaborative detection network: In the present invention, the edge collaborative detection network ensures rapid sharing and processing of data through multi-terminal collaboration. By integrating multiple handheld display and control terminals and other mobile nodes, such as drones, the system can achieve all-round vital signs detection. In the aforementioned data processing and fusion stage, the data of each sensor is preprocessed and intelligently analyzed, and finally the edge computing network performs real-time collaborative detection. The core task of this module is to provide an efficient multi-node collaborative platform to ensure the accuracy, real-time and reliability of the detection results.

[0057] In this embodiment, the edge collaborative detection network adopts Mesh self-organizing network technology, which enables multiple terminals and devices to flexibly connect and share data. Each terminal in the system has communication and computing capabilities and can communicate wirelessly with other nodes. Each terminal establishes a multi-hop network through the Mesh protocol, so that even if the distance between some nodes is far, the data can still be transmitted to the final destination through the intermediate nodes. In this way, the communication range of the system is effectively expanded, and the wide applicability of the system at the disaster site is guaranteed.

[0058] Specifically, when a handheld display and control terminal completes the vital sign detection, it broadcasts the detection results, including the target's location information and detection confidence, to other terminals in the network. Other terminals cross-verify the received data to confirm the authenticity of the target. Generally, the communication protocol of the Mesh network allows data sharing and collaboration between all terminals and drone nodes. Each terminal implements on-demand transmission of data packets through the AODV protocol. In this way, the system can ensure real-time synchronization of detection information of each node, avoiding the single point failure problem in the traditional centralized network architecture.

[0059] In this embodiment, the collaborative positioning algorithm between nodes is based on the principle of triangulation positioning. When multiple terminals detect the same target, they will locate from different directions. By calculating the phase difference or signal strength difference between these nodes, the system can accurately calculate the position of the target. The three-point positioning method can provide high positioning accuracy, and the positioning error is less than 0.5 meters in most application scenarios.

[0060] Assume Node , and They are located at known positions and are respectively close to the target point The distances are , and , target point Coordinates It can be calculated by the following equations: ; ; ; These equations are the basis for triangulation, which calculates the precise location of an object from three known points and their corresponding distances.

[0061] For the confirmation of the target location, this system adopts a confidence voting mechanism. That is, if more than 70% of the nodes confirm the existence of the target, the system will consider the target's vital signs to be valid. This mechanism greatly improves the accuracy and robustness of the system in complex environments.

[0062] For each detected vital sign, the system confirms it through a confidence voting mechanism between nodes. If more than 70% of the nodes confirm that the target exists, the system will consider the target to be valid. The formula can be expressed as: ; in, It is The confidence value of each node for the target T; is the number of nodes participating in the vote. If it is greater than a certain threshold (such as 0.7), the target is considered effective.

[0063] In addition, the edge collaborative detection network also provides powerful functions of data fusion and multi-terminal collaboration. Each node can share local detection data in real time, and other nodes can make real-time corrections based on the received data to ensure the accuracy and consistency of the final report. Under this dynamic collaborative framework, data exchange between multiple terminals can not only improve the work efficiency of the system, but also avoid detection blind spots caused by single device failure or data loss.

[0064] The edge collaborative detection network fully utilizes the advantages of multi-terminal collaboration. At the disaster site, multiple handheld display and control terminals and drone nodes can work together quickly, greatly improving the accuracy and response speed of vital sign detection. Compared with the traditional single-node detection mode, the present invention significantly improves the adaptability and reliability of the system through the collaborative work of multiple terminals.

[0065] In actual applications, the network's configuration and working mode are highly flexible. The number of terminals and drone nodes can be dynamically adjusted according to actual needs, ensuring that the system can operate efficiently at disaster sites of different sizes. The system can also further optimize the network topology according to the needs of different scenarios to improve data transmission efficiency and collaboration accuracy.

[0066] Through the above method, the edge collaborative detection network in this embodiment ensures the efficient collaboration and precise positioning of the entire detection system in disaster rescue, solves the limitations and bottleneck problems of the traditional single sensor and centralized processing mode, and enhances the stability and adaptability of the system in complex environments.

[0067] Augmented reality interaction: In the present invention, the augmented reality interaction module presents the detection results obtained after data processing, fusion and collaborative positioning in the form of intuitive graphics, logos and feedback, making up for the lack of information in traditional display methods. The data between each module is seamlessly connected to provide real-time assistance for on-site rescue. This module is closely linked with the aforementioned data processing and network collaboration parts to ensure efficient flow of data from detection to presentation.

[0068] In this embodiment, the augmented reality interaction module first captures the live video through the camera and registers the sensor data in space. Generally, the virtual data must accurately match the real scene. As an option, this module uses the projection transformation formula: ; in, Indicates the position of the target in the world coordinate system; is the model view matrix; is the projection matrix; is the projection coordinate of the target on the display screen. This formula ensures that the digital information is accurately aligned with the live environment.

[0069] In this embodiment, a layered overlay technique is also used to realize graphic display. The display layers are divided into background, target identification and warning icons. The processed data is superimposed on the real-time video in the form of colored areas or icons. Red marks are used for high-confidence targets, and yellow marks are used for targets to be confirmed. This method makes it easy for on-site operators to quickly identify key information.

[0070] In this embodiment, the augmented reality interaction module also provides tactile feedback. The detection results are not only presented visually, but also generate vibration prompts through the handheld terminal. Different confidence levels correspond to different vibration modes: low-confidence targets are given slight vibrations, and high-confidence targets are given strong feedback. This feedback mechanism helps rescuers quickly judge the importance of the target. The augmented reality interaction module synchronously processes real-time video data, sensor data, and location information.

[0071] The system uses a unified timestamp to realize the linkage of multiple data sources, ensuring that all information is presented almost simultaneously, meeting the needs of high-speed response on site. The augmented reality interaction module realizes the intuitive display of detection results through technical means such as real-time video capture, virtual and real data superposition, projection transformation, graphic layering and tactile feedback. This module effectively makes up for the limitations of traditional display methods and improves the efficiency and accuracy of on-site decision-making.

[0072] Embodiment 1: Scenario description: After a strong earthquake, large-scale building collapse occurred in many areas of the city. The disaster site environment is extremely complex, and there may be trapped people in the ruins. In order to efficiently detect life and determine the location of trapped people, the rescue team deployed a multi-sensor detection system.

[0073] Implementation steps: Step 1: Sensor layout and data collection: Thermal imaging cameras, millimeter-wave radars and audio vibration sensors are placed at key locations in the ruins.

[0074] The infrared thermal imager is responsible for capturing the temperature distribution of the surrounding environment, especially possible vital signs, the millimeter-wave radar detects tiny target movements, and the audio vibration sensor is used to sense the heartbeat and breathing of living organisms.

[0075] The sensors collect data in real time and send it to a nearby handheld terminal via a wireless network. The low latency of data transmission ensures that rescuers get feedback quickly.

[0076] Step 2: Data processing and fusion: The data collected by each sensor will be processed by the data processing module for noise reduction and signal correction to remove external environmental noise and interference.

[0077] The data fusion engine combines infrared, audio and radar data for analysis and makes a comprehensive assessment based on sensor weights. The weights are automatically adjusted to ensure the accuracy of data contribution when the environment changes.

[0078] The edge collaborative detection network ensures fast data sharing and collaborative work among multiple handheld terminals, further improving the accuracy of target positioning.

[0079] Step 3: Target positioning and confirmation: Using multi-node positioning algorithms (such as triangulation positioning), multiple sensors are used to determine the possible location of the trapped person.

[0080] Through the confidence voting mechanism, if more than 70% of the terminals confirm that a certain area is the location where vital signs exist, then the location is determined to be the target.

[0081] Step 4: Augmented reality interaction and display: The augmented reality system superimposes the real-time detected target location information with the on-site video image, displaying the logo in the trapped area to alert rescue workers.

[0082] Through the graphical interface on the handheld device, rescuers can clearly see the specific location and confidence level of each target, and get further prompts through tactile feedback.

[0083] Step 5: Rescue execution: Based on the information provided by the augmented reality interactive system, rescuers act quickly and prioritize going to locations with confirmed strong vital signs for rescue.

[0084] Embodiment 2: Scenario description: When a fire occurs, the temperature inside the fire scene is extremely high and the smoke is dense. Traditional detection methods are difficult to penetrate the smoke and flames. In this case, it is crucial to use multi-sensor collaborative detection technology to detect vital signs and combine it with augmented reality technology to provide real-time safety guidance for on-site rescue personnel.

[0085] Implementation steps: Step 1: Sensor layout and data collection: Deploy infrared thermal imagers, audio vibration sensors and gas detectors at the fire scene.

[0086] Infrared thermal imagers are used to detect temperature differences, especially the thermal radiation signals of living organisms; audio vibration sensors are used to capture heartbeats or breathing sounds; and gas detectors monitor the concentration of harmful gases.

[0087] These sensors transmit data to the command center and rescue workers' handheld devices via wireless communication to ensure information synchronization.

[0088] Step 2: Data processing and fusion: The data processing module performs real-time noise filtering and signal processing to remove interference signals caused by flames and smoke.

[0089] Through a multi-source data fusion engine, the system combines infrared, audio and gas detection data to assess each sensor's confidence in the presence of a target.

[0090] The edge collaborative detection network ensures that various sensors and devices on site work together in real time to ensure the stable transmission of information flow.

[0091] Step 3: Target identification and positioning: The system analyzes the comprehensive information from each sensor to determine areas where vital signs may exist.

[0092] The augmented reality system marks high-confidence vital sign locations in red and displays them in real time on the device screen.

[0093] Step 4: Augmented reality interaction and evacuation guidance: Rescuers can see real-time video footage and an overlay of the target location through augmented reality devices, identifying dangerous areas and safe evacuation routes.

[0094] If a high-risk area is detected, the augmented reality system will highlight it in red and use vibration feedback to remind people to avoid it as soon as possible.

[0095] Step 5: Rescue and evacuation: According to the AR prompts, rescuers give priority to entering areas with obvious vital signs for rescue, while ensuring that other people are evacuated quickly and safely.

[0096] Embodiment three: Scenario description: When a gas leak occurs in a chemical plant, the site is not only faced with the risk of possible explosion, but also the danger of toxic gas leakage. In order to accurately identify risks and guide rescue work, the system combines temperature sensors, gas sensors and audio sensors for multi-modal detection, and uses augmented reality technology to visualize on-site risks.

[0097] Implementation steps: Step 1: Sensor layout and data collection: Temperature sensors, gas detectors, vibration sensors, etc. are deployed at key locations in chemical plants.

[0098] Temperature sensors are used to monitor temperature changes in an area, gas detectors are used to monitor harmful gas leaks in real time, and vibration sensors monitor structural changes in equipment or pipelines.

[0099] Step 2: Data processing and fusion: The data processing module processes the raw data obtained from each sensor, including noise filtering, signal correction and data synchronization.

[0100] The multi-source data fusion engine integrates the data from each sensor and ensures that each sensor provides the most accurate data in different environments through automatic weight adjustment methods.

[0101] The edge collaborative detection network ensures fast synchronization and sharing of data among multiple devices to avoid information lag.

[0102] Step 3: Risk assessment and positioning: The system analyzes data from multimodal sensors to evaluate gas leakage concentration, temperature changes and vibration conditions at the site, and conducts real-time risk assessment.

[0103] Based on the analysis results, the system will mark high-risk areas and verify dangerous areas through a confidence voting mechanism.

[0104] Step 4: Augmented reality interaction and risk visualization: The augmented reality system overlays real-time sensor data and risk assessment results, marking dangerous areas of hazardous gas leakage and displaying safe evacuation routes.

[0105] Rescuers can see a clear AR interface through handheld devices to understand the on-site environment, risk areas, and the location of vital signs, while receiving tactile feedback.

[0106] Step 5: Decision and Action: Based on the real-time information in the AR interface, the command center and on-site personnel make quick decisions and guide rescue workers to avoid dangerous areas and quickly enter areas in need of assistance to carry out rescue.

[0107] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-sensor fusion life detection system based on a handheld display and control terminal, characterized in that: include: A sensor group, used to collect vital signs signals and environmental parameters, the sensor group includes an infrared thermal imager, an audio vibration sensor, a radar wave detector and an environmental sensor; A data processing module, connected to the sensor group, for performing noise filtering and feature extraction on the data collected by the sensor group; A multi-source data fusion engine, connected to the data processing module, for fusion analysis of the pre-processed multi-modal data; A handheld display and control terminal is connected to the multi-source data fusion engine and is used to receive fusion results and display vital sign positioning information; An edge collaborative network, connected to the handheld display and control terminal, is used to connect multiple handheld display and control terminals and drone nodes to achieve data sharing and collaborative analysis; The augmented reality interaction module is integrated in the handheld display and control terminal, and is used to overlay vital sign positioning information on the real-time image and provide tactile feedback.

2. The multi-sensor fusion life detection system based on a handheld display and control terminal according to claim 1 is characterized in that: The data processing module comprises: A noise filtering unit, connected to the sensor group, for removing burst noise and performing smoothing on time series data; a cross calibration unit, connected to the noise filtering unit, for correcting the depth error of the infrared thermal imager and verifying whether the heat source is a biological body; The output end of the cross calibration unit is connected to the multi-source data fusion engine to transmit the processed data to the multi-source data fusion engine.

3. The multi-sensor fusion life detection system based on a handheld display and control terminal according to claim 1 is characterized in that: The multi-source data fusion engine includes a dynamic weight adjustment module, which is configured as follows: Connecting to the data processing module to receive pre-processed data; Monitor environmental parameters in real time and dynamically assign weights to each sensor based on the environmental parameters; The output end of the dynamic weight adjustment module is connected to the signal analysis unit to output the weight distribution result.

4. The multi-sensor fusion life detection system based on a handheld display and control terminal according to claim 1 is characterized in that: The multi-source data fusion engine also includes a signal analysis unit, which is configured as follows: Connect with the dynamic weight adjustment module to receive the weight distribution result; Jointly analyze multimodal data to extract spatial and time series features; The temperature difference area of ​​infrared thermal imaging and the heart rhythm characteristics of audio vibration signals are associated through a cross-modal attention mechanism; The output end of the signal analysis unit is connected to the handheld display and control terminal to output fusion analysis data.

5. The multi-sensor fusion life detection system based on a handheld display and control terminal according to claim 1 is characterized in that: The edge collaboration network configuration is: Connecting with the handheld display and control terminal to receive fusion analysis data; Connect multiple handheld display and control terminals and drone nodes, and cross-verify the detection results of multiple terminals; The output end of the edge collaborative network is connected to the handheld display and control terminal to output collaborative detection results.

6. The multi-sensor fusion life detection system based on a handheld display and control terminal according to claim 1 is characterized in that: The augmented reality interaction module is configured as follows: Connecting with the handheld display and control terminal to receive collaborative detection results; Overlay vital signs positioning information on the real-time image of the terminal camera; generating tactile feedback based on the strength of vital sign signals; The output end of the augmented reality interaction module is connected to the display unit of the handheld display and control terminal to display vital sign positioning information.

7. The multi-sensor fusion life detection system based on a handheld display and control terminal according to claim 3 is characterized in that: The dynamic weight adjustment module uses evidence theory to fuse multi-sensor data, specifically including: Connecting to the data processing module to receive pre-processed data; Construct a sensor reliability matrix based on environmental parameters and correct the fusion results through evidence conflict detection algorithm; The output end of the dynamic weight adjustment module is connected to the signal analysis unit to output the corrected fusion result.

8. The multi-sensor fusion life detection system based on a handheld display and control terminal according to claim 4 is characterized in that: The signal analysis unit is incrementally updated through federated learning, and its configuration is: Connecting to the multi-source data fusion engine to receive fusion analysis data; Multiple terminals share anonymized data and upload it to the cloud training platform; Privacy protection technology is used to ensure data security, and local models are fine-tuned to adapt to different disaster scenarios; The output end of the signal analysis unit is connected to the multi-source data fusion engine to output the updated feature extraction result.

9. The multi-sensor fusion life detection system based on a handheld display and control terminal according to claim 1 is characterized in that: The radar wave detector is configured as follows: Connecting to the data processing module to output detection data; Detect chest rise and fall signals of living organisms; The output end of the radar wave detector is connected to the noise filtering unit of the data processing module.

10. The multi-sensor fusion life detection system based on a handheld display and control terminal according to claim 8, characterized in that: The model updating process of the federated learning includes: Connecting to the signal analysis unit to receive feature extraction results; The feature extraction weights related to vital sign recognition in the global model parameters are retained; Ensure model generalization performance through parameter update restriction technology; The output end of the federated learning module is connected to the signal analysis unit to output updated model parameters.

Images