Firefighter physiological safety monitoring system based on multi-modal data

Abstract

The invention relates to the technical field of safety monitoring, and discloses a fireman physiological safety monitoring system based on multi-modal data. According to the fireman physiological safety monitoring system based on the multi-modal data, by integrating real-time acquisition and analysis of physiological data, motion data and environment data, the system can comprehensively monitor the physiological state of a fireman and can evaluate the change of a fire scene environment in real time; the multi-modal data fusion method enables the system to perform fire behavior prediction and firefighter safety early warning more accurately, and particularly provides more accurate and timely alarm feedback in the face of rapid diffusion, complex fire scenes and high-difficulty fire extinguishing situations. By generating the alertness score, the attention distribution score and the operation response time of the firefighter, the system can effectively improve the safety awareness of the firefighter and enhance the ability of the firefighter to deal with emergencies, thereby significantly improving the safety and efficiency of the fire extinguishing action.

Description

Technical Field

[0001] The present invention relates to the technical field of safety monitoring, and particularly to a physiological safety monitoring system for firefighters based on multi-modal data. Background Art

[0002] Fire accidents have always been one of the major dangerous disasters that seriously threaten people's lives, health and property safety, and threaten social development. In recent years, with the multi-dimensional and comprehensive development of the economic society, the urban population has expanded rapidly, and magnificent urban buildings have sprung up. The urban landscape has become increasingly dense and charming. People's pursuit of outdoor natural landscapes has been continuously expanding. In this high-speed development trend, once a forest landscape catches fire, due to problems such as the fast spread speed of the fire, complex fire scene conditions and great difficulty in extinguishing the fire, the requirements for timely, rapid and accurate information perception in fire fighting and rescue have reached a new height.

[0003] Through statistical analysis of the casualties of Chinese firefighters in recent years, it is found that during the period from 2005 to 2013, a total of 70 casualties occurred to firefighters in fire fighting and rescue operations, with an average of about 27 firefighters suffering casualties each year. And in recent years, the number of casualties has shown an increasing trend year by year. At the same time, the statistical results also show that the number of casualties of new firefighters who have been engaged in fire fighting and rescue work for less than four years is the largest, accounting for 64.71% of the total number. Thus, it can be seen that the shorter the time of engaging in fire fighting and rescue work, the greater the probability of firefighters' death, and the more serious the death situation. Therefore, work experience and rescue ability are important factors affecting the casualties of firefighters.

[0004] The existing physiological monitoring and auxiliary technologies for firefighters can only monitor the physiological state or fire scene data of firefighters singly, and cannot predict the fire situation of the fire scene and give safety warnings to firefighters based on the current fire scene situation and the physiological conditions of firefighters. It is very difficult to achieve the purpose of safety warning and command for firefighters in the face of situations such as fast spread speed, complex fire scene conditions and great difficulty in extinguishing the fire. Summary of the Invention

[0005] (I) Technical Problems to be Solved

[0006] In view of the deficiencies of the prior art, the present invention provides a physiological safety monitoring system for firefighters based on multi-modal data. By integrating the real-time collection and analysis of physiological data, motion data, and environmental data, this system can not only comprehensively monitor the physiological state of firefighters but also evaluate the changes in the fire scene environment in real time. This method of multi-modal data fusion enables the system to more accurately predict the fire situation and provide safety warnings for firefighters. Especially in the face of rapidly spreading, complex fire scenes, and high-difficulty fire extinguishing situations, it provides more accurate and timely alarm feedback. By generating alertness scores, attention allocation scores, and operation reaction times for firefighters, the system can effectively enhance the safety awareness of firefighters and strengthen their ability to respond to emergencies, thus significantly improving the safety and efficiency of fire extinguishing operations.

[0007] (II) Technical Solution

[0008] To achieve the above object, the present invention provides the following technical solution: A physiological safety monitoring system for firefighters based on multi-modal data, comprising a data acquisition module, a data preprocessing module, a feature extraction module, a data fusion module, a state evaluation module, and an alarm feedback module;

[0009] The data acquisition module collects real-time physiological data of firefighters, motion data of firefighters, and environmental data of the firefighters' location through sensors and high-frame-rate cameras when firefighters are performing tasks. The physiological data of firefighters includes electrocardiogram, pulse, skin temperature, and blood oxygen saturation. The motion data of firefighters records the motion state of firefighters through attitude sensors and obtains their mental state through head-eye movement gaze tracking and facial expression analysis. The environmental data of the firefighters' location monitors the changes in the fire scene environment through visible light, near-infrared, and mid- and far-infrared images;

[0010] The data preprocessing module filters the data obtained by the data acquisition module to remove noise, performs data standardization processing, and then aligns the data from different sensors according to the time stamp and sends it to the feature extraction module;

[0011] The feature extraction module extracts key features from the preprocessed data, including physiological heart rate change rate, pulse waveform features, acceleration, motion posture change features, position features of the fixation point, expression duration, and facial emotion change features;

[0012] The data fusion module comprehensively processes the data from different modalities, and uses weighted average to fuse the extracted features into a unified data set and sends it to the state evaluation module;

[0013] The state evaluation module generates alertness scores, attention allocation scores, and operation reaction times for firefighters based on the fused data, for real-time evaluation of the physiological state and potential safety risks of firefighters;

[0014] The alarm feedback module sends real-time alarms to firefighters through a vibration device according to the output of the status evaluation module.

[0015] Preferably, the formula for data filtering to remove noise is as follows:

[0016]

[0017] In the formula, H(f) represents the transfer function of the filter, f represents the frequency of the input signal, f c represents the cut-off frequency, and j represents the imaginary unit.

[0018] Preferably, the formula for data alignment is as follows:

[0019]

[0020] In the formula, D aligned (t) represents the data value after alignment at time t, D1(t) represents the data value of the first sensor at time t, t1 and t2 are the timestamps of the first and second sensors respectively, and D2(t2) represents the data value of the second sensor at time t2.

[0021] Preferably, the formula for calculating the physiological heart rate change rate is as follows:

[0022] HRV = σ(RR)

[0023] In the formula, HRV represents heart rate variability, σ represents the standard deviation function, and RR represents the time interval between adjacent heartbeats in the electrocardiogram.

[0024] Preferably, the formula for extracting pulse waveform features is as follows:

[0025] P feature = T peak - T trough

[0026] In the formula, P feature represents the pulse waveform feature, T peak represents the peak time of the pulse wave, T trough represents the trough time of the pulse wave.

[0027] Preferably, the formula for calculating acceleration is as follows:

[0028]

[0029] In the formula, a(t) represents the acceleration at time t, v(t) represents the velocity at time t, and Δt represents the time interval.

[0030] Preferably, the formula for extracting the characteristics of motion posture changes is as follows:

[0031] Δθ = θ current - θ previous

[0032] In the formula, Δθ represents the angular change between the current and previous postures, θ current represents the current posture angle, and θ previous represents the previous posture angle.

[0033] Preferably, the formula for extracting the position characteristics of the fixation point is as follows:

[0034]

[0035] In the formula, Gaze avg represents the average fixation point position, N represents the number of fixations, Gaze i represents the position of the i-th fixation point, and i represents the counting subscript.

[0036] Preferably, the formula for calculating the duration is as follows:

[0037] Duration = T end - T start

[0038] In the formula, Duration represents the duration, T end represents the end time, and T start represents the start time.

[0039] Preferably, the formula for extracting the characteristics of facial emotional changes is as follows:

[0040] Emotion feature = P(emotion1), P(emotion2),..., P(emotion n )

[0041] In the formula, Emotion feature represents the emotional feature vector, and P(emotion i ) represents the probability of the i-th emotion.

[0042] Compared with the prior art, the present invention provides a physiological safety monitoring system for firefighters based on multi-modal data, having the following beneficial effects:

[0043] By integrating the real-time collection and analysis of physiological data, motion data, and environmental data, this system can not only comprehensively monitor the physiological state of firefighters but also evaluate the changes in the fire scene environment in real time. This method of multi-modal data fusion enables the system to more accurately predict the fire situation and provide early warnings for the safety of firefighters. Especially in the face of rapidly spreading, complex fire scenes, and high-difficulty firefighting situations, it provides more accurate and timely alarm feedback. By generating alertness scores, attention allocation scores, and operation reaction times for firefighters, the system can effectively enhance the safety awareness of firefighters and their ability to respond to emergencies, thus significantly improving the safety and efficiency of firefighting operations. Brief Description of the Drawings

[0044] Figure 1 It is a schematic diagram of the system flow of the present invention. Detailed Embodiment

[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0046] In view of the fact that the existing physiological monitoring and assistance technologies for firefighters can only monitor the physiological state of firefighters or fire scene data singly, and cannot predict the fire situation of the current fire scene and the physiological situation of firefighters, and it is difficult to achieve the purpose of early warning and command for firefighters' safety in the face of situations such as fast spreading speed, complex fire scene conditions, and high firefighting difficulty. Therefore, a physiological safety monitoring system for firefighters based on multi-modal data is proposed. Please refer to Figure 1 , this system includes a data collection module, a data preprocessing module, a feature extraction module, a data fusion module, a state evaluation module, and an alarm feedback module;

[0047] The data collection module can, through a variety of advanced sensor technologies and high-frame-rate cameras, collect multi-level physiological data, motion data, and environmental data in real time and accurately when firefighters are performing tasks. Specifically, the physiological data of firefighters includes electrocardiogram (ECG), pulse, skin temperature, and blood oxygen saturation. These data are collected through physiological monitoring devices (such as wearable heart rate monitors and pulse oximeters). These devices use photoplethysmography (PPG) technology and electrode sensors to monitor cardiac activity and blood oxygen levels to ensure the real-time and accuracy of the data;

[0048] In terms of motion data, the system records the motion state of firefighters through highly sensitive attitude sensors (such as accelerometers and gyroscopes), providing three-dimensional motion trajectories and attitude information. In addition, by using head-eye movement gaze tracking technology (such as infrared trackers) and facial expression analysis technology (such as deep learning-based image recognition algorithms), the psychological state of firefighters, such as attention concentration and emotional changes, can be effectively obtained. The combination of these technologies can capture the comprehensive performance of firefighters in a high-pressure environment;

[0049] The data of the environment where firefighters are located is monitored through a variety of imaging technologies, including high-definition images collected by visible light cameras, thermal energy distribution monitored by near-infrared cameras, and mid- and far-infrared imaging technologies, which are used to identify the characteristics of fire sources and obstacles. These image information are processed through computer vision algorithms to analyze the changes in the fire scene conditions, assist in decision-making support, ensure the safety of firefighters and rescue efficiency. By integrating these technical means, the data acquisition module can comprehensively and real-time monitor the physiological, motion and environmental states of firefighters, greatly improving the safety and scientific nature in task execution;

[0050] After receiving the raw data obtained by the data acquisition module, the data preprocessing module first filters the data to remove noise and interference, ensuring the accuracy and reliability of the data. Specifically, a low-pass filter is used, and its transfer function is:

[0051]

[0052] By setting an appropriate cut-off frequency f c , high-frequency noise can be effectively filtered out, and the useful information in the signal can be retained. This process is particularly important for physiological signals (such as electrocardiograms and pulse waveforms) because these signals are usually affected by motion artifacts and environmental interference. The filtered data will significantly improve the accuracy of subsequent analysis;

[0053] Next, the module normalizes the data. Using the Z-score normalization formula, by calculating the mean and standard deviation of the raw data, the data from different sensors are converted into standardized values with the same dimension. This not only helps to eliminate the measurement differences between different devices, but also improves the effectiveness of feature extraction;

[0054] After completing filtering and normalization, the data preprocessing module aligns the data of different sensors according to the time stamps, using the linear interpolation method, and the formula is:

[0055]

[0056] By aligning timestamps, it is possible to ensure that data from different sensors is synchronized in time, which is crucial for analyzing the physiological state of firefighters and its relationship with environmental changes. Time alignment not only improves the timeliness of data but also provides a consistent time basis for subsequent feature extraction, ensuring the accuracy and reliability of analysis results. After this series of preprocessing, the data finally sent to the feature extraction module will be of high quality and comparable, laying a solid foundation for subsequent state assessment and decision support;

[0057] After receiving the preprocessed data, the feature extraction module uses a variety of advanced algorithms and technologies to extract key physiological and motion features for subsequent state assessment and decision support. First, the physiological heart rate variability (HRV) is calculated by analyzing the time intervals (RR intervals) between adjacent heartbeats in the electrocardiogram signal. This process can be achieved by calculating the standard deviation of the RR intervals:

[0058] HRV = σ(RR)

[0059] This feature can reflect the state of the firefighter's autonomic nervous system and further reveal their psychological burden and fatigue level in a high-pressure environment;

[0060] Secondly, the extraction of pulse waveform features involves peak detection of the pulse signal and calculating the time difference P feature = T peak - T trough to obtain the dynamic changes of the pulse waveform. This process can not only provide an indication of cardiovascular health but also help identify physiological responses in emergency situations;

[0061] In terms of motion features, the module performs differential operations on the data collected by the acceleration sensor to calculate the acceleration of the firefighter, thereby reflecting their motion state:

[0062]

[0063] Through this calculation, it is possible to monitor the motion intensity and changes of the firefighter during the execution of tasks, helping to evaluate their physical exertion and work efficiency. In addition, the motion posture change feature is obtained through a posture estimation model and is usually expressed as the angle change Δθ = θ current - θ previous This feature helps to understand the adaptability and flexibility of firefighters in complex environments;

[0064] In the assessment of the psychological state, the fixation point position feature is obtained through head-eye movement tracking technology, and the position of the average fixation point is calculated:

[0065]

[0066] This feature can reveal the attention allocation of firefighters, helping to judge their concentration during task execution. At the same time, the expression duration and facial emotion change features are extracted through facial expression recognition technology, usually represented as the probability distribution of different emotions:

[0067] Emotion feature = P(emotion1), P(emotion2),..., P(emotion n )

[0068] These facial features can not only reflect the emotional state of firefighters, but also provide important basis for team cooperation and psychological support;

[0069] By extracting features from the data that has been filtered, normalized and time-aligned, the feature extraction module can comprehensively and deeply analyze the physiological state, motion performance and psychological reactions of firefighters. This process ensures the high quality and consistency of the extracted features, thus providing a reliable basis for subsequent state assessment, helping the commanders to make scientific decisions in a timely manner, and ensuring the safety and health of firefighters in high-risk environments;

[0070] After receiving data from different modalities, the data fusion module uses an advanced weighted average algorithm to comprehensively process the extracted features. This process first normalizes each feature to eliminate the dimensional differences between different data sources and ensure the accuracy of fusion. Through weighted average, the module can assign different weights according to the importance and relevance of each feature, so as to fuse multi-dimensional features such as physiological state, motion performance and psychological reactions into a unified data set. The specific weighted average formula can be expressed as:

[0071]

[0072] Among them, F fused is the fused feature data, w i is the weight of feature F i , n is the number of features. In this way, the data fusion module can effectively integrate data from multiple sensors such as heart rate, pulse, acceleration, posture, gaze point and facial emotion, form a comprehensive overview of the firefighters' state, and send it to the state assessment module;

[0073] Based on the fused data, the status evaluation module uses machine learning algorithms and statistical analysis methods to generate alertness scores, attention allocation scores, and operation reaction times for firefighters. These scores are used to evaluate the potential safety risks of firefighters in real time by analyzing their physiological and psychological states. For example, the alertness score may be based on heart rate variability (HRV) and facial emotional change features to evaluate the psychological burden of firefighters, while the attention allocation score combines the gaze point position and motion state to judge the degree of attention concentration of firefighters. The operation reaction time is evaluated by analyzing their reaction speed during task execution;

[0074] Finally, the alarm feedback module sends real-time alarms to firefighters through a vibration device according to the output of the status evaluation module. When the evaluation results show that the alertness or attention of firefighters is lower than the preset threshold, the system will immediately trigger a vibration alarm to remind firefighters of potential safety risks. This real-time feedback mechanism not only enhances the safety awareness of firefighters but also effectively reduces the probability of accidents, ensuring their safety and health in high-risk environments. Through this series of modular designs, the entire system realizes the comprehensive monitoring and evaluation of firefighters' status, providing strong technical support for emergency response.

[0075] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A firefighter physiological safety monitoring system based on multimodal data, characterized by: It includes data acquisition module, data preprocessing module, feature extraction module, data fusion module, state evaluation module and alarm feedback module; The data acquisition module collects firefighters' physiological data, firefighters' motion data and firefighters' environmental data in real time through sensors and high-frame rate cameras when firefighters are performing tasks. Firefighters' physiological data include electrocardiogram, pulse, skin temperature and blood oxygen saturation. Firefighters' motion data record the firefighters' motion state through posture sensors, and obtain their psychological state through head and eye movement tracking and facial expression analysis. Firefighters' environmental data monitors changes in the fire scene environment through visible light, near infrared and mid- and far-infrared images. The data preprocessing module filters the data acquired by the data acquisition module to remove noise, and after data standardization, aligns the data of different sensors according to the timestamp and sends them to the feature extraction module; The feature extraction module extracts key features from the preprocessed data, including physiological heart rate change rate, pulse waveform features, acceleration, motion posture change features, gaze point position features, expression duration, and facial emotion change features; The data fusion module comprehensively processes the data from different modes, and uses weighted average to fuse the extracted features into a unified data set and send it to the state assessment module; The state assessment module generates a firefighter alertness score, a firefighter attention allocation score and a firefighter operation reaction time based on the fused data, which are used to assess the firefighter's physiological state and potential safety risks in real time; The alarm feedback module sends a real-time alarm to the firefighters through a vibration device according to the output of the status evaluation module.

2. A firefighter physiological safety monitoring system based on multimodal data according to claim 1, characterized in that: The formula for data filtering to remove noise is as follows: In the formula, H(f) represents the transfer function of the filter, f represents the frequency of the input signal, and f c represents the cut-off frequency, and j represents the imaginary unit.

3. A firefighter physiological safety monitoring system based on multimodal data according to claim 2, characterized in that: The formula for data alignment is as follows: In the formula, D aligned (t) represents the data value after alignment at time t, D1(t) represents the data value of the first sensor at time t, t1 and t2 are the timestamps of the first and second sensors respectively, and D2(t2) represents the data value of the second sensor at time t2.

4. The firefighter physiological safety monitoring system based on multimodal data according to claim 3, characterized in that: The formula for calculating the physiological heart rate change rate is as follows: HRV=σ(RR) In the formula, HRV represents heart rate variability, σ represents the standard deviation function, and RR represents the time interval between adjacent heartbeats in the electrocardiogram.

5. A firefighter physiological safety monitoring system based on multimodal data according to claim 4, characterized in that: The formula for extracting pulse waveform features is as follows: P feature =T peak -T trough In the formula, P feature Represents the pulse waveform characteristics, T peak Indicates the peak time of the pulse wave, T trough Indicates the trough time of the pulse wave.

6. A firefighter physiological safety monitoring system based on multimodal data according to claim 5, characterized in that: The formula for calculating acceleration is as follows: In the formula, a(t) represents the acceleration at time t, v(t) represents the velocity at time t, and Δt represents the time interval.

7. A firefighter physiological safety monitoring system based on multimodal data according to claim 6, characterized in that: The formula for extracting the motion posture change feature is as follows: Δθ=θ current -θ previous In the formula, Δθ represents the angle change between the current and previous postures, θ current Indicates the current posture angle, θ previous Indicates the previous posture angle.

8. The firefighter physiological safety monitoring system based on multimodal data according to claim 7, characterized in that: The formula for extracting the position feature of the gaze point is as follows: In the formula, Gaze avg represents the average gaze point position, N represents the number of gazes, Gaze i represents the position of the i-th fixation point, and i represents the counting subscript.

9. A firefighter physiological safety monitoring system based on multimodal data according to claim 8, characterized in that: The formula for calculating duration is as follows: Duration=T end -T start In the formula, Duration represents the duration, T end Indicates the end time, T start Indicates the start time.

10. A firefighter physiological safety monitoring system based on multimodal data according to claim 9, characterized in that: The formula for extracting facial emotion change features is as follows: Emotion feature =P(emotion1),P(emotion2),...,P(emotion n ) In the formula, Emotion feature Represents the emotional feature vector, P(emotion i ) represents the probability of the i-th emotion.

Images