Firefighter alertness detection system and method based on multi-modal physiological signals

Abstract

The invention provides a fireman alertness detection system and method based on multi-modal physiological signals, and belongs to the technical field of fireman alertness detection. In order to solve the technical problems that real-time rapid alertness detection based on short-time and few multi-modal physiological signals is lacked at present, and a multi-dimensional alertness state and grade detailed division system is not established. According to the technical scheme, an EEG collecting dry electrode and a PPG sensor which are used for collecting multi-mode physiological signals are arranged on the inner side of the physiological signal collecting fire-fighting cap through a fixing band, and an alertness detection terminal is arranged on the inner side of the rear end of the physiological signal collecting fire-fighting cap. The EEG acquisition dry electrode and the PPG sensor are connected with the signal input end of the alertness detection terminal through wires; the alertness detection terminal is wirelessly connected with the fire-fighting command center terminal through the wireless communication module; an early warning unit is further arranged on one side of the physiological signal acquisition fire hat; the device and the method are applied to fireman alertness detection.

Description

Technical Field

[0001] The present invention provides a firefighter alertness detection system and method based on multimodal physiological signals, belonging to the technical field of firefighter alertness detection. Background Art

[0002] In the process of performing firefighting tasks, firefighters often face extremely dangerous environments such as high temperature, thick smoke, toxic gases and structural collapse. As the operation time increases, high-intensity physical exertion and brain cognitive load increase, which will lead to a decrease in the individual alertness of firefighters. Low alertness will cause misjudgment of the fire, equipment operation errors, and reduced information processing efficiency, thereby posing a threat to the firefighters' own safety and rescue efficiency. Therefore, in the process of performing tasks, it is very important to conduct real-time detection and early warning of firefighters' alertness. It is the key to ensuring the individual life safety of firefighters and improving rescue efficiency.

[0003] Alertness refers to an individual's ability to maintain attention and concentration on a target task for a long time, while responding to low-frequency and critical emergencies; alertness does not exist in a single dimension, and alertness is divided into awakening alertness and execution alertness; awakening alertness refers to an individual's level of consciousness and physiological activation state, reflecting the perception and sensitivity to external stimuli; execution alertness refers to an individual's attention allocation and information processing ability during task execution, reflecting the execution ability and decision-making ability when facing emergencies or emergency tasks.

[0004] At present, the detection methods of alertness are mainly divided into subjective methods and objective methods. Although the subjective method has the advantages of being simple, fast and easy to operate, the evaluation results may not be objective and stable due to the influence of personal subjective factors; the objective method estimates alertness based on physiological characteristic parameters, including EEG signals, photoplethysmography signals, PPG signals, EOG signals, etc.; EEG, as a direct manifestation of brain state, is considered to be one of the most effective methods for assessing alertness levels; however, the commonly used EEG acquisition equipment is mostly wet electrode acquisition systems based on conductive paste or saline, but because of its complicated preparation work, it is not convenient to wear quickly; at the same time, PPG, as a non-invasive real-time physiological signal measurement method, can also be used to assess alertness levels, which is reflected in the fact that the heart rate decreases as alertness continues to decrease.

[0005] In addition, due to the particularity of the firefighting environment, the alertness data obtained based on objective detection needs to be evaluated quickly and accurately. Currently, deep learning can be used to train evaluation models and quickly evaluate the input data. Deep learning can effectively improve detection performance by directly learning the feature representation and mapping relationship of the data set, but it still has certain limitations in practical applications. First, most existing methods focus on offline analysis and lack neural network structures that use a small amount of physiological signals to achieve rapid alertness detection, which makes it difficult to meet the actual application needs in firefighting scenarios. Second, most current neural network models focus on binary or three-category alertness level assessment. There is no effective distinction and comprehensive assessment of multi-dimensional alertness states and alertness levels based on actual work conditions, which affects the accuracy of firefighter alertness detection in actual firefighting work environments. Third, existing alertness detection systems focus on pilots, air traffic controllers, etc., and there is a lack of application for high-risk groups of firefighters. Due to the extremely complex environment that firefighting work inevitably involves, the model is required to be lightweight while having stronger generalization ability and robustness.

[0006] Based on the above problems, there is an urgent need to develop a firefighter alertness detection solution based on multimodal physiological signals, integrating functions such as firefighter alertness detection and early warning, taking into account both lightweight and high precision, so as to meet the technical requirements of real-time, stable, low data volume and high reliability in the extremely complex environment of firefighting. Summary of the Invention

[0007] The present invention aims to address the current lack of real-time and rapid alertness detection based on short-term, small-volume multimodal physiological signals, the lack of a detailed classification system for multi-dimensional alertness states and levels for actual operational conditions, and the lack of technical problems for alertness detection for firefighters in high-risk and complex task scenarios. The technical solution adopted is as follows: providing a firefighter alertness detection system based on multimodal physiological signals, including a physiological signal collection fire hat, wherein the inner side of the physiological signal collection fire hat is provided with EEG collection dry electrodes and PPG sensors for collecting multimodal physiological signals through a fixing belt, and an alertness detection terminal is provided on the inner side of the rear end of the physiological signal collection fire hat, and the EEG collection dry electrodes and PPG sensor are both connected to the signal input end of the alertness detection terminal via wires;

[0008] The embedded processor integrated in the alertness detection terminal has a built-in data pre-processing module and an alertness assessment model;

[0009] The alert detection terminal is wirelessly connected to the fire command center terminal via a wireless communication module;

[0010] An early warning unit for voice reminder and vibration feedback is also provided on one side of the physiological signal collection fire hat, and the signal input end of the early warning unit is connected to the alert detection terminal through a wire.

[0011] The specific number of EEG acquisition dry electrodes is 8. The electrode positions are set according to the international 10-20 system rules. The acquisition dry electrodes F3, F4, P3, and P4 are equidistantly set on the left and right frontal lobes and parietal lobes of the human head, the acquisition dry electrodes T3 and T4 are symmetrically set on the left and right temporal lobes of the human head, and the acquisition dry electrodes O1 and O2 are set on the left and right occipital lobes of the human head.

[0012] The PPG sensor is placed on the forehead vein of a person in a non-invasive manner.

[0013] The alert detection terminal is specifically encapsulated inside a shielding layer composed of a metal shell and conductive foam.

[0014] The conductor is specifically a shielded conductor consisting of an inner copper braid and an outer aluminum foil.

[0015] The data preprocessing module specifically processes the collected multimodal physiological signals, including:

[0016] Bandpass filtering and band-stop filtering were used to remove redundant signals, and independent component analysis was used to remove artifacts and noise from the EEG signal to obtain the corresponding data set, which was then subjected to short-time Fourier transform.

[0017] The alertness assessment model is an MP-Net neural network model that integrates a frequency-domain-channel residual attention module and includes criteria for assessing alertness states and alertness levels, where:

[0018] In the dimension of alertness state, it is divided into resting state, arousal alertness state, and executive alertness state;

[0019] The alertness level is divided into 1-9 levels, namely:

[0020] 1 Extremely alert, 2 Very alert, 3 Alert, 4 Somewhat alert, 5 Neither alert nor sleepy, 6 Some signs of sleepiness, 7 Sleepy, 8 Sleepy, 9 Extremely sleepy.

[0021] A method for detecting firefighter alertness based on multimodal physiological signals includes the following detection steps:

[0022] S1: Firefighters wear a fire cap to collect physiological signals and activate the detection system. The alert detection terminal collects EEG and PPG signals from the firefighters' heads in real time during operation.

[0023] S2: The alertness detection terminal pre-processes the received physiological signals and inputs them into the trained MP-Net neural network model to assess the firefighter's alertness state and alertness level;

[0024] S3: The alertness detection terminal makes a judgment based on the alertness level information output by the MP-Net neural network model. When it detects that the alertness level has dropped to a threshold, it automatically triggers an early warning prompt and sends a control signal to the early warning unit. The early warning unit then sends a voice reminder and vibration feedback to the firefighter, and feeds back the detection information to the fire command center terminal to alert the firefighter himself and the on-site commander.

[0025] The specific method for evaluating the alertness state and alertness level of firefighters in step S2 is:

[0026] S21: Preprocessing of physiological signals collected in real time, including:

[0027] Band-pass filtering was used to extract EEG signals between 0.5 Hz and 60 Hz;

[0028] Use 50Hz band-stop filtering to remove frequency noise caused by AC power;

[0029] Independent component analysis is used to remove irrelevant artifacts such as blinks and eye drifts from EEG signals and separate noise from mixed signals;

[0030] Perform short-time Fourier transform on the data set to convert the time-domain physiological signals into frequency-domain physiological signals. The calculation formula of the short-time Fourier transform is:

[0031] ;

[0032] in, is the original signal; is a sliding window function used to extract local time segments; t is the time position; is the angular frequency; is the Fourier kernel in complex exponential form;

[0033] S22: The preprocessed data is fed into the depthwise separable convolution module in the MP-Net neural network model to decouple spatial feature extraction from channel feature mapping, and then fed into the residual module for processing;

[0034] S23: The residual module constructs four stages in series and fuses multi-level features through skip connections. The network jointly introduces three complementary loss terms: focal loss, center loss, and triplet loss. Finally, a classifier consisting of adaptive average pooling and fully connected layers outputs denoised multimodal physiological signals.

[0035] S24: Cut the denoised multimodal physiological signals into data sets of different sizes according to different time lengths for training, input the training data sets into the MP-Net neural network model, and obtain the optimal MP-Net neural network model by adjusting the batch size and screening the optimal physiological signal slice length, and perform evaluation based on the optimal model.

[0036] The specific method of reminding the firefighters and the on-site commander in step S3 is:

[0037] According to the alert level results of 1 to 9 output by the optimal MP-Net neural network model, when the alert level reaches level 6 or above, the warning unit is controlled to issue a voice and vibration warning. At the same time, a warning signal is sent to the fire command center terminal through the alert detection terminal. The alert status and alert level information of each firefighter performing the task are sent to the fire command center terminal and dynamically displayed on the operation interface of the fire command center terminal in the form of a color level icon.

[0038] The beneficial effects of the present invention over the prior art are as follows: the firefighter alertness detection scheme provided by the present invention, compared with the traditional alertness detection scheme, mainly performs a two-dimensional division of human alertness state and alertness level, and constructs a multi-level alertness classification system. This multi-dimensional division is more in line with the staged physiological changes of firefighters in actual task execution, and is conducive to the accurate modeling and dynamic detection of alertness in different operation stages (such as standby, receiving alarms, charging, command execution, etc.), and the detection effect is significantly better than the traditional two-classification or three-classification method; the present invention constructs a lightweight neural network model based on short-term, small-scale multimodal physiological signals to achieve rapid and accurate assessment of firefighter alertness, meeting the dual requirements of real-time and accuracy in extreme application environments, and at the same time adopts voice reminders, vibration feedback and remote terminal interconnection, etc., which can automatically trigger intervention reminders in the early stage of firefighter alertness decline, effectively reducing the risk of human operational errors and safety accidents caused by fatigue or decreased attention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present invention will be further described below with reference to the accompanying drawings:

[0040] Figure 1 Schematic diagram of the structure of the firefighter alertness detection system of the present invention;

[0041] Figure 2 This is a diagram showing the layout of dry electrodes for EEG acquisition in an embodiment of the present invention;

[0042] Figure 3 This is a flow chart of the steps of the firefighter alertness detection method of the present invention;

[0043] Figure 4Schematic diagram of the process of using MP-Net neural network training in an embodiment of the present invention;

[0044] Figure 5 Schematic diagram of the structure of the alertness detection neural network used in an embodiment of the present invention;

[0045] Figure 6 This is a diagram showing the effect of the short-time Fourier transform used in an embodiment of the present invention;

[0046] Figure 7 This is a diagram showing the experimental effect of alertness detection in an embodiment of the present invention;

[0047] In the figure, 1 is a fire helmet for collecting physiological signals, 2 is a terminal of the fire command center, 101 is an EEG collection dry electrode, 102 is a PPG sensor, 103 is an alert detection terminal, and 104 is an early warning unit. DETAILED DESCRIPTION

[0048] like Figures 1 to 7 As shown, the present invention is to solve the technical problem that firefighters are unable to perform real-time and accurate multi-level alertness detection and early warning when working in extremely complex environments. It provides a firefighter alertness detection solution based on multimodal physiological signals, wherein the detection system provided includes a physiological signal acquisition fire hat, an alertness assessment model and a state recognition and early warning unit. The multimodal physiological signal sensor is integrated in the physiological signal acquisition fire hat, which can collect EEG and PPG signals in real time without affecting the normal operation of firefighters in complex environments, solving the defects of physiological signal detection equipment being difficult to carry and unstable signal quality; the alertness assessment model adopts an MP-Net neural network with an optimized structural design, which can update the firefighter alertness in real time at a frequency of 1Hz. The firefighters' alertness state and alertness level assessment results specifically integrate frequency attention and channel attention mechanisms, and combine deep separable convolution and residual structure to perform multi-level assessment from the dual dimensions of alertness state and alertness level, realizing dynamic monitoring with high timeliness and high precision. The model can realize rapid and accurate identification and graded warning of firefighters' alertness by identifying short-term, small amounts of multimodal physiological signals; after the state recognition and warning unit receives the neural network evaluation results, it automatically triggers the warning mechanism when it detects that the alertness level has dropped to the warning line, and can send reminder signals through voice, vibration or remote platform to prompt the on-site commander to carry out personnel rotation, task adjustment or intervention in time to avoid operational errors or safety accidents caused by decreased alertness.

[0049] like Figure 1 and Figure 2As shown, the present application provides a firefighter alertness detection system suitable for extremely complex firefighting operating environments, with high timeliness and high precision. The system integrates multimodal physiological signal acquisition equipment and a lightweight neural network model. Through joint modeling of frequency domain and channel features, it can achieve rapid identification and dynamic early warning of firefighter alertness in complex environments. The detection system mainly includes a physiological signal acquisition fire hat, the inner side of which is provided with EEG acquisition dry electrodes and PPG sensors for collecting multimodal physiological signals through a fixing belt. An alertness detection terminal is provided on the inner side of the rear end of the physiological signal acquisition fire hat. The EEG acquisition dry electrodes and PPG sensors are both connected to the signal input end of the alertness detection terminal through wires.

[0050] The embedded processor integrated in the alertness detection terminal has a built-in data pre-processing module and an alertness assessment model;

[0051] The alert detection terminal is wirelessly connected to the fire command center terminal via a wireless communication module;

[0052] An early warning unit for voice reminder and vibration feedback is also provided on one side of the physiological signal collection fire hat, and the signal input end of the early warning unit is connected to the alert detection terminal through a wire.

[0053] The specific number of EEG acquisition dry electrodes is 8. The electrode positions are set according to the international 10-20 system rules. The acquisition dry electrodes F3, F4, P3, and P4 are equidistantly set on the left and right frontal lobes and parietal lobes of the human head, the acquisition dry electrodes T3 and T4 are symmetrically set on the left and right temporal lobes of the human head, and the acquisition dry electrodes O1 and O2 are set on the left and right occipital lobes of the human head.

[0054] like Figures 3 to 6 As shown, based on the detection system, the present invention also provides a firefighter alertness detection method based on multimodal physiological signals, comprising the following steps:

[0055] S1: Firefighters wear fire caps to collect physiological signals, and collect EEG and PPG signals in real time while they are working;

[0056] S2: pre-processing the physiological signal and inputting it into the trained MP-Net neural network, wherein the model is a neural network for assessing the alertness state and alertness level of firefighters based on multimodal physiological signals;

[0057] S3: Based on the alert level output by the MP-Net neural network, the early warning unit that plays voice, vibrates, and is remotely connected is linked. When a decrease in alertness is detected, an early warning prompt is automatically triggered and an early warning is initiated.

[0058] In an embodiment of the present invention, the physiological signal collection fire hat used in step S1 can be used for alertness detection. The hat body is the basic load-bearing structure of the fire hat, which is made of high-temperature resistant and impact-resistant materials and has good fireproof, waterproof, electrical and pressure-resistant properties; 8-channel EEG collection dry electrodes and 1 PPG sensor are provided inside the hat body for physiological signal collection; the signal line connecting the electrodes / sensors inside the fire hat and the alertness detection terminal uses a double-layer shielded wire (inner layer copper braid and aluminum foil); the alertness detection terminal is located at the rear of the fire hat and is encapsulated in a metal shell and a conductive foam shielding layer.

[0059] Before conducting alertness testing, it is necessary to first conduct alertness induction experiments under different alertness states and alertness levels using a fire helmet and collect multimodal physiological signals. The task paradigm used in this invention simulates the different states of firefighters' daily work. The alertness experiment includes three subtasks:

[0060] 1) Resting state task: in the standby state before the alarm, without setting up any emergency events, the resting physiological signals of firefighters are collected during the operation. This state is the reference stage for evaluating the individual baseline state;

[0061] 2) Arousal alertness task: a visual selection experiment was conducted using a behavioral function test trainer. When the red indicator light representing the fire alarm lights up, the firefighter is required to quickly press the button to respond;

[0062] 3) Performing alert tasks involves a discriminative response experiment using a behavioral function test trainer. Firefighters are required to make decisions and coordinate multiple tasks based on sudden dangerous situations. When the green indicator light comes on, firefighters must simultaneously perform multiple firefighting tasks, focusing on the direction of the water cannon spray, controlling the spread of the fire, and operating ventilation equipment, all while maintaining uninterrupted radio communication with the command center.

[0063] Before the formal experiment begins, firefighters need to enter the experimental platform for training in advance. The training content includes making the firefighters clear about the experimental task process, practicing each part of the experimental task individually and completing the proficiency test. After the training, the firefighters will rest for five minutes and then fill in their personal basic information. During the experiment, firefighters need to wear standard fire-proof clothing and fire hats to simulate real-life operation scenarios.

[0064] The vigilance experiment included three tasks: (a) resting-state task, (b) arousal vigilance task, and (c) executive vigilance task.

[0065] The above three tasks appear randomly, and are considered as a group if completed in sequence. Each firefighter needs to complete three sets of experiments at a level of 9 alertness. Before the start of each set of experiments, they need to fill out the Karolinska Sleepiness Scale to ensure that the alertness level meets the experimental needs. After each set of experiments, they need to continue working for 2-3 hours until the last set of experiments is completed, in order to ensure fatigue. In addition, they are not allowed to sleep before doing experiments in a fatigued state, and are not allowed to drink coffee or any beverages that help to wake up. Physiological signals are always collected during the execution of the task, and each set of tasks follows the same process.

[0066] Each tester needs to complete three alertness states and nine alertness level experiments. The alertness state level is divided into resting state, awake alertness state, and executive alertness state. The alertness level level is divided into 1-9 levels, namely: 1 extremely alert (energized, excellent state), 2 very alert, 3 alert, 4 a little alert (but not in the best state), 5 neither alert nor sleepy, 6 some signs of sleepiness, 7 sleepy (but can stay awake without effort), 8 sleepy (requires some effort to stay awake), 9 extremely sleepy (difficult to stay awake, with a strong desire to fall asleep).

[0067] In an embodiment of the present invention, the S2 specifically includes:

[0068] Preprocessing of the collected multimodal physiological signals includes:

[0069] 1) Use bandpass filtering to extract EEG signals between 0.5Hz and 60Hz;

[0070] 2) Use a 50Hz band-stop filter to remove the frequency noise caused by the AC power;

[0071] 3) Use independent component analysis to remove irrelevant artifacts such as blinks and eye drifts from the EEG signal and separate the noise from the mixed signal;

[0072] 4) Perform short-time Fourier transform (STFT) on the dataset to convert the time-domain physiological signals into frequency-domain physiological signals to capture more discriminative feature information. The STFT formula is as follows:

[0073] ;

[0074] in, Refers to the original signal; Refers to the sliding window function, which is used to extract local time segments; t refers to the time position; is the angular frequency; Refers to the Fourier kernel in the form of complex exponential;

[0075] The MP-Net neural network model adopted in this paper is a multi-layer residual neural network structure that combines the attention mechanism with deep separable convolution. By introducing a module that fused channels and spectral attention mechanisms, the model significantly enhances its adaptive modeling capability of the frequency domain and channel features of multimodal physiological signals, thereby improving the sensitivity and recognition accuracy to changes in alertness under complex task states.

[0076] The model is equipped with a frequency domain-channel residual attention module. By establishing a residual module that fuses the channel and spectral attention modules, it can adaptively model the inter-channel dependency and the key feature distribution in the frequency dimension, improve the ability to extract local features, and effectively enhance the adaptive modeling capability of frequency domain and channel features. To further optimize parameter efficiency and reduce computational complexity, the module introduces depthwise separable convolution, which effectively reduces the number of model parameters and improves computational efficiency by decoupling spatial feature extraction from channel feature mapping. In the overall architecture, multiple residual modules are constructed in series in four stages, and multi-level features are fused through jump connections, which enhances the model's expressiveness while maintaining high computational efficiency. Finally, the classifier composed of adaptive average pooling and fully connected layers achieves accurate assessment of alertness status and alertness level.

[0077] The depthwise separable convolution module in the model effectively reduces the number of model parameters and improves computational efficiency by decoupling spatial feature extraction from channel feature mapping.

[0078] The residual module set in the model constructs four stages in series and fuses multi-level features through skip connections to enhance the model's expressiveness while maintaining high computational efficiency. The model is based on the complementary loss function back-propagation collaborative mechanism, and the network jointly introduces three complementary loss terms: focal loss, center loss, and triplet loss, where:

[0079] Focus loss effectively alleviates the sample imbalance problem by suppressing the impact of a large number of easy-to-classify samples on the total loss and enhancing the model's attention to difficult-to-classify samples;

[0080] The center loss constrains samples of the same type to be close to the center of their corresponding category in the feature space, enhancing intra-class aggregation and improving inter-class separability.

[0081] The triplet loss constructs an anchor-positive sample-negative sample triple to shorten the distance between similar samples and increase the distance between heterogeneous samples, thereby enhancing the contrast of the global structure in the discriminant space.

[0082] In order to flexibly adjust the contribution of each loss term to model training, the model introduces hyperparameters to control the weights of the center loss and triplet loss respectively, thereby achieving balanced optimization between different constraint objectives.

[0083] To further improve the model's training efficiency and generalization ability, the model also introduces a dynamic learning rate strategy. When the average loss fails to decrease significantly in five consecutive training cycles, the learning rate will automatically decay to half of its original value. This strategy can effectively prevent the model from falling into local optimality and help achieve a more robust convergence process.

[0084] Finally, a classifier consisting of adaptive average pooling and fully connected layers was used to achieve feature dimensionality reduction and accurate classification of alertness status;

[0085] Establish a training dataset. Cut the denoised multimodal physiological signals into training datasets of different sizes according to different time lengths (1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, and 5 seconds).

[0086] During the training process, the training dataset was input into the MP-Net model, and the batch sizes were 64, 128, 192, 256, 320, 384, 448, 512, 576, 640, 704, 786, 832, 896, and 960, respectively. At the same time, physiological signals of different sizes were input into the model for testing to determine the optimal parameters;

[0087] The alertness assessment model constructed by the present invention obtains the optimal MP-Net neural network model through two steps of batch size adjustment and optimal physiological signal slice length screening, thereby improving the classification accuracy and robustness of the model.

[0088] In an embodiment of the present invention, the S3 specifically includes:

[0089] Based on the alertness level output from the MP-Net neural network model, which ranges from 1 to 9, the system automatically triggers voice and vibration warning mechanisms when the alertness level reaches level 6 or above. The voice announcement system integrated into the fire helmet broadcasts the current alertness level in real time, accompanied by a slight vibration stimulus, such as "Current state is slightly sleepy, please be alert." This physiologically assists firefighters in regaining their attention, effectively raising the wearer's alertness level and enabling timely, non-invasive state intervention.

[0090] The MP-Net neural network model results are connected to the fire center command platform via wireless communication to achieve remote terminal interconnection. The alertness status and alertness level of each firefighter performing a task are uploaded to the command terminal in real time and dynamically displayed on the operation interface in the form of color-grade icons. This mechanism helps commanders fully understand the alertness status of on-site workers, facilitates scientific scheduling, timely rotation, and task optimization, thereby improving the level of personnel safety and task execution efficiency in high-risk working environments.

[0091] The effectiveness of the present invention is verified by experiments below:

[0092] In order to verify the effectiveness and optimal parameter configuration of the MP-Net neural network model proposed in this invention, physiological signal slices of different time lengths were input into the model and the batch size was adjusted. Figure 7 The figure shows the overall classification performance of the model under different slice lengths and batch sizes. Slice lengths between 1 and 2 seconds show higher surface heights, indicating better model performance at this slice length. The surface reaches its highest point when the batch size is between 800 and 900, and the model exhibits higher stability and accuracy with this parameter combination. The results demonstrate that the proposed model has high accuracy and sensitivity in alertness detection. When the batch size is 896, the model can effectively detect alertness using 1-second physiological signal slices, making this parameter combination the optimal MP-Net model. While maintaining accuracy and robustness, the model achieves both high efficiency and lightweightness, achieving an inference time of 5.34 milliseconds per sample and using 12.22MB of GPU memory.

[0093] In summary, the present invention proposes for the first time a firefighter alertness detection scheme based on multimodal physiological signals from the perspective of awakening-execution alertness. It has a multi-level alertness detection and early warning method with high accuracy, high real-time performance and low data requirements, which provides strong technical support for the transformation of firefighting operations towards refinement and intelligence, and provides efficient protection for the safety of firefighters in extremely complex environments. The detection system provided by the present invention can use multimodal physiological signals to achieve real-time updates of alertness status and alertness level assessment results at a frequency of 1Hz, which is suitable for firefighters to achieve real-time alertness detection and early warning in extreme fire environments such as high temperature, electromagnetic interference, and low visibility.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A firefighter alertness detection system based on multimodal physiological signals, comprising a physiological signal acquisition fire helmet (1), characterized in that: An EEG acquisition dry electrode (101) and a PPG sensor (102) for acquiring multimodal physiological signals are provided on the inner side of the physiological signal acquisition fire cap (1) via a fixing belt, and an alertness detection terminal (103) is provided on the inner side of the rear end of the physiological signal acquisition fire cap (1). Both the EEG acquisition dry electrode (101) and the PPG sensor (102) are connected to a signal input end of the alertness detection terminal (103) via a wire. The embedded processor integrated in the alertness detection terminal (103) has a built-in data pre-processing module and an alertness evaluation model; The alert detection terminal (103) is wirelessly connected to the fire command center terminal (2) via a wireless communication module; An early warning unit (104) for providing voice reminders and vibration feedback is also provided on one side of the physiological signal collection fire cap (1), and a signal input end of the early warning unit (104) is connected to the alert detection terminal (103) via a wire.

2. The firefighter alertness detection system based on multimodal physiological signals according to claim 1, characterized in that: The specific number of the EEG acquisition dry electrodes (101) is 8. The electrode positions are set according to the international 10-20 system rules. The acquisition dry electrodes F3, F4, P3, and P4 are equidistantly set on the left and right frontal lobes and parietal lobes of the human head, the acquisition dry electrodes T3 and T4 are symmetrically set on the left and right temporal lobes of the human head, and the acquisition dry electrodes O1 and O2 are set on the left and right occipital lobes of the human head.

3. The firefighter alertness detection system based on multimodal physiological signals according to claim 2, characterized in that: The PPG sensor (102) is arranged at the forehead vein of a person in a non-invasive manner.

4. The firefighter alertness detection system based on multimodal physiological signals according to claim 3, characterized in that: The alert detection terminal (103) is specifically encapsulated inside a shielding layer composed of a metal shell and conductive foam.

5. The firefighter alertness detection system based on multimodal physiological signals according to claim 4, characterized in that: The conductor is specifically a shielded conductor consisting of an inner copper braid and an outer aluminum foil.

6. The firefighter alertness detection system based on multimodal physiological signals according to claim 1, characterized in that: The data preprocessing module specifically processes the collected multimodal physiological signals, including: Bandpass filtering and band-stop filtering were used to remove redundant signals, and independent component analysis was used to remove artifacts and noise from the EEG signal to obtain the corresponding data set, which was then subjected to short-time Fourier transform.

7. The firefighter alertness detection system based on multimodal physiological signals according to claim 6, characterized in that: The alertness assessment model is an MP-Net neural network model that integrates a frequency-domain-channel residual attention module and includes criteria for assessing alertness states and alertness levels, where: In the dimension of alertness state, it is divided into resting state, arousal alertness state, and executive alertness state; The alertness level is divided into 1-9 levels, namely: 1 Extremely alert, 2 Very alert, 3 Alert, 4 Somewhat alert, 5 Neither alert nor sleepy, 6 Some signs of sleepiness, 7 Sleepy, 8 Sleepy, 9 Extremely sleepy.

8. The detection method of a firefighter alertness detection system based on multimodal physiological signals according to claim 7, characterized in that: The detection steps include the following: S1: The firefighter wears a physiological signal collection fire hat (1) and starts the detection system, and the alert detection terminal (103) collects EEG signals and PPG signals fed back from the firefighter's head during operation in real time; S2: The alertness detection terminal (103) pre-processes the received physiological signals and inputs them into the trained MP-Net neural network model to evaluate the alertness state and alertness level of the firefighters; S3: The alert detection terminal (103) makes a judgment based on the alert level information output by the MP-Net neural network model. When it detects that the alert level has dropped to a threshold, it automatically triggers an early warning prompt and sends a control signal to the early warning unit (104). The early warning unit (104) sends a voice reminder and vibration feedback to the firefighter, and feeds back the detection information to the fire command center terminal (2) to remind the firefighter himself and the on-site commander.

9. The method for detecting firefighter alertness based on multimodal physiological signals according to claim 8, characterized in that: The specific method for evaluating the alertness state and alertness level of firefighters in step S2 is: S21: Preprocessing of physiological signals collected in real time, including: Band-pass filtering was used to extract EEG signals between 0.5 Hz and 60 Hz; Use 50Hz band-stop filtering to remove frequency noise caused by AC power; Independent component analysis is used to remove irrelevant artifacts such as blinks and eye drifts from EEG signals and separate noise from mixed signals; Perform short-time Fourier transform on the data set to convert the time-domain physiological signals into frequency-domain physiological signals. The calculation formula of the short-time Fourier transform is: ； in, is the original signal; is a sliding window function used to extract local time segments; t is the time position; is the angular frequency; is the Fourier kernel in complex exponential form; S22: The preprocessed data is fed into the depthwise separable convolution module in the MP-Net neural network model to decouple spatial feature extraction from channel feature mapping, and then fed into the residual module for processing; S23: The residual module constructs four stages in series and fuses multi-level features through skip connections. The network jointly introduces three complementary loss terms: focal loss, center loss, and triplet loss. Finally, a classifier consisting of adaptive average pooling and fully connected layers outputs denoised multimodal physiological signals. S24: Cut the denoised multimodal physiological signals into data sets of different sizes according to different time lengths for training, input the training data sets into the MP-Net neural network model, and obtain the optimal MP-Net neural network model by adjusting the batch size and screening the optimal physiological signal slice length, and perform evaluation based on the optimal model.

10. The method for detecting firefighter alertness based on multimodal physiological signals according to claim 9, characterized in that: The specific method of reminding the firefighters and the on-site commander in step S3 is: According to the alert level results of 1 to 9 output by the optimal MP-Net neural network model, when the alert level reaches level 6 or above, the warning unit (104) is controlled to issue a voice and vibration warning, and at the same time, a warning signal is sent to the fire command center terminal (2) through the alert detection terminal (103), and the alert status and alert level information of each firefighter performing the task are sent to the fire command center terminal (2) and dynamically displayed in the form of a color level icon on the operation interface of the fire command center terminal (2).

Images