Miner sign evaluation device and method based on multi-modal information fusion
Through the multimodal information fusion miner sign assessment device, multiple physiological signals and video data are collected and fused, and the mood and fatigue status of coal miners are evaluated, which solves the problem that existing equipment cannot fully evaluate miner signs, and improves safety and Reliability of assessment.
Patent Information
- Application Number
- CN202411075835.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing mine sign monitoring equipment only measures a single physiological signal, cannot provide comprehensive and accurate sign information, and is not suitable for multi-person signal collection, and cannot effectively evaluate the sign status of coal miners in complex environments, which increases safety risks.
Design a miner's sign evaluation device based on multimodal information fusion, and collects electrocardiogram signals, skin electrical signals, body temperature signals, blood pressure signals, blood oxygen saturation signals, EEG signals and facial video data through multimodal data collectors , and use the physiological signal feature extraction module, facial data feature extraction module and multi-feature fusion module in the data processor to fuse multiple signal characteristics to evaluate the emotional state, fatigue state and comprehensive sign status of miners.
It realizes a comprehensive and accurate assessment of miners' multimodal data, can promptly detect abnormal situations, improve safety, and is suitable for multi-person sign monitoring at the wellhead of coal mines, reducing safety hazards.
Smart Images

Figure CN119007076A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of vital sign assessment, and in particular to a miner vital sign assessment device and method based on multimodal information fusion. Background Art
[0002] Coal mine safety issues are related to the stable operation of the mine and the life safety of miners, and have a far-reaching impact on the coal mining industry and its stakeholders. In existing mine safety accidents, human factors have been confirmed to be the main cause. In human-caused mine accidents, unsafe behaviors of miners are particularly prominent and have become one of the main causes of mine accidents. The abnormal physical signs of miners under complex working environments and heavy labor pressure, such as fatigue, bad mood, and abnormal perception, are often the direct causes of these unsafe behaviors, which undoubtedly increases the safety hazards of mine work.
[0003] In the prior art, many technology companies have developed vital sign monitoring devices such as smart watches and bracelets. However, when using these vital sign monitoring devices for vital sign monitoring, the existing devices on the market have the following problems:
[0004] 1) Measuring only a single physiological signal cannot provide users with comprehensive and accurate vital sign information;
[0005] 2) The equipment is only suitable for collecting personal vital signs signals. It is time-consuming and labor-intensive to collect signals from multiple people, and is not suitable for the special working environment of coal mines.
[0006] However, there is currently no corresponding product for the monitoring and assessment of vital signs of miners at the mine entrance. The coal mining industry is in urgent need of such a product to fill the gap. Summary of the invention
[0007] The purpose of this application is to provide a miner's vital signs assessment device and method based on multimodal information fusion, which can measure the miner's multimodal data at the shaft head of a coal mine and accurately assess the miner's various vital signs.
[0008] To achieve the above objectives, this application provides the following solutions:
[0009] In a first aspect, the present application provides a miner vital sign assessment device based on multimodal information fusion, comprising: a multimodal data collector, a data processor and a display.
[0010] The multimodal data collector is used to collect the multimodal data of the miners and transmit the multimodal data to the data processor; the multimodal data includes electrocardiogram signals, skin electricity signals, body temperature signals, blood pressure signals, blood oxygen saturation signals, EEG signals and facial videos.
[0011] The data processor is used to: use the physiological signal feature extraction module to respectively extract the physiological signal features of the electrocardiogram signal, the skin electrode signal, the body temperature signal, the blood pressure signal, the blood oxygen saturation signal and the electroencephalogram signal; use the facial data feature extraction module to extract the dynamic features and static features of the facial video; use the multi-feature fusion module to fuse the physiological signal features of the electrocardiogram signal, the skin electrode signal, the body temperature signal, the blood pressure signal, the blood oxygen saturation signal, the electroencephalogram signal, the dynamic features of the facial video and the static features of the facial video to obtain multimodal fusion features; use the physical sign status assessment module to determine the miner's emotional state, fatigue state and comprehensive physical sign state according to the multimodal fusion features.
[0012] The display is used to show the miners the emotional state, fatigue state and comprehensive physical sign status.
[0013] Optionally, the multimodal data collector specifically includes: an ECG sensor, an galvanic skin sensor, a temperature sensor, a blood pressure and blood oxygen sensor, an EEG cap and a camera; the ECG sensor, the galvanic skin sensor, the temperature sensor, the blood pressure and blood oxygen sensor and the camera are integrated on the device housing; the ECG sensor, the galvanic skin sensor, the temperature sensor, the blood pressure and blood oxygen sensor, the camera and the EEG cap are respectively connected to the data processor.
[0014] Optionally, the EEG cap includes: collection electrodes, EEG signal collection chip circuit, main control chip circuit, communication circuit, power management circuit and Bluetooth circuit; the EEG signal collection chip circuit contacts the miner's scalp through the collection electrodes to obtain the miner's EEG signals; the main control chip circuit communicates with the power management circuit, EEG signal collection chip circuit and Bluetooth circuit through the communication circuit, and is used to control the EEG signal collection chip circuit, power management circuit and Bluetooth circuit; the Bluetooth circuit is used to send the EEG signals obtained by the EEG signal collection chip circuit to the data processor.
[0015] Optionally, the data processor is specifically used to extract time domain features and frequency domain features of electroencephalogram signals, electrocardiogram signals, skin electrical signals, body temperature signals, blood oxygen saturation signals, and blood pressure signals.
[0016] Optionally, extracting dynamic features and static features of the facial video using a facial data feature extraction module specifically includes the following steps:
[0017] The static features of the facial video are extracted using the facial data static feature extraction submodule; the facial data static feature extraction submodule is a convolutional neural network.
[0018] The dynamic features of the facial video are extracted using the facial data dynamic feature extraction submodule; the facial data dynamic feature extraction submodule is a recurrent neural network.
[0019] Optionally, the convolutional neural network includes the following key parts: an input layer, a convolution layer, a pooling layer and a fully connected layer; the input layer is used to input image information; the convolution layer includes three convolution kernels with a kernel size of 7×7 and one convolution kernel of 1×1 to reduce the feature dimension; a pooling layer is added after the convolution layer to prevent overfitting, and finally the static features of the facial video are connected through the fully connected layer.
[0020] Optionally, the recurrent neural network takes sequence data as input, effectively captures the relationship characteristics between sequences through the internal structural design of the network, and also outputs in the form of sequences; the loop mechanism of the recurrent neural network enables the result generated by the model layer at the previous time to be used as part of the input of the current time step, affecting the output of the current time step, and at the same time increasing the expressive power of the model by using nonlinear activation functions.
[0021] Optionally, the multi-feature fusion module is a Transformer model; the Transformer model includes an encoder and a decoder; the encoder is responsible for extracting and transforming features from the embedded sequence and mapping it to a high-dimensional representation space; through a multi-head self-attention mechanism and a feedforward fully connected structure, the encoder can capture key information, as well as the correlation and complementarity between embedded sequence features; the decoder generates a target sequence based on the high-dimensional representation and context information output by the encoder. In addition to the basic structure, the Transformer model may also include residual connections and normalization to improve training effects; the residual connections effectively alleviate the gradient vanishing problem and promote information flow; the normalization helps to reduce internal covariate shifts and improve generalization capabilities.
[0022] Optionally, the display is also used to display electrocardiogram signals, skin electrical signals, body temperature signals, blood pressure signals, blood oxygen saturation signals, brain wave signals and facial videos to the miners in the form of images.
[0023] In a second aspect, the present application provides a method for assessing miners' vital signs based on multimodal information fusion, comprising the following steps:
[0024] Collect multimodal data of miners and transmit the multimodal data to the data processor; the multimodal data includes electrocardiogram signals, skin electrical signals, body temperature signals, blood pressure signals, blood oxygen saturation signals, EEG signals and facial videos.
[0025] The physiological signal feature extraction module is used to extract the physiological signal features of electrocardiogram signals, skin electrical signals, body temperature signals, blood pressure signals, blood oxygen saturation signals and electroencephalogram signals.
[0026] The facial data feature extraction module is used to extract dynamic and static features of facial videos.
[0027] A multi-feature fusion module is used to fuse the physiological signal features of ECG signals, skin electrical signals, body temperature signals, blood pressure signals, blood oxygen saturation signals, EEG signals, dynamic features of facial videos, and static features of facial videos to obtain multimodal fusion features.
[0028] The physical sign status assessment module is used to determine the miners' emotional state, fatigue state and comprehensive physical sign state based on multimodal fusion features.
[0029] Multimodal data, emotional status, fatigue status and comprehensive physical status are presented to miners through displays.
[0030] According to the specific embodiments provided in this application, this application discloses the following technical effects:
[0031] The present application provides a miner's vital signs assessment device and method based on multimodal information fusion, and the device includes: a multimodal data collector, a data processor and a display. The multimodal data of the miner is collected by the multimodal data collector, and the multimodal data is transmitted to the data processor; the physiological signal feature extraction module is used to extract the physiological signal features of the electrocardiogram signal, the skin electrical signal, the body temperature signal, the blood pressure signal, the blood oxygen saturation signal and the brain electrical signal respectively; and the dynamic features and static features of the facial video are extracted by the facial data feature extraction module; then the various physiological signal features, the dynamic features of the facial video and the static features of the facial video are fused by the multi-feature fusion module to obtain the multimodal fusion features; the emotional state, fatigue state and comprehensive vital signs state of the miner are determined by the vital signs state assessment module according to the multimodal fusion features, and are intuitively displayed to the miner. This application evaluates the miners' vital signs through multimodal data. The evaluation results are more comprehensive and reliable. Abnormal conditions of miners can be discovered in time, making it easier to take corresponding measures to ensure safe operations. The device can be installed at the entrance of a coal mine. Before going down into the mine, it can accurately evaluate the miners' various vital signs by measuring the miners' multimodal data. It can be competent for the physical sign evaluation of miners at the mine entrance before going down to work, thereby ensuring the safety of miners. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0033] Figure 1 This is a functional module diagram of a miner's vital signs assessment device based on multimodal information fusion in one embodiment of the present application.
[0034] Figure 2 It is a detailed structural diagram of a miner's vital signs assessment device based on multimodal information fusion in one embodiment of the present application.
[0035] Figure 3 This is a schematic diagram of an EEG cap in a miner's vital sign assessment device based on multimodal information fusion in one embodiment of the present application.
[0036] Figure 4 It is a schematic diagram of a multimodal data collection process in a miner's vital signs assessment device based on multimodal information fusion in one embodiment of the present application.
[0037] Figure 5 This is a schematic diagram of a collection and evaluation interface displayed on a display in a miner's vital signs evaluation device based on multimodal information fusion in one embodiment of the present application.
[0038] Figure 6 A flowchart of a miner's vital signs assessment method based on multimodal information fusion provided in one embodiment of the present application. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0040] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.
[0041] In an exemplary embodiment, Figure 1 As shown, a miner's vital signs assessment device based on multimodal information fusion is provided, including: a multimodal data collector, a data processor 3 and a display 1.
[0042] The multimodal data collector is used to collect the multimodal data of the miners and transmit the multimodal data to the data processor; the multimodal data includes electrocardiogram signals, skin electricity signals, body temperature signals, blood pressure signals, blood oxygen saturation signals, EEG signals and facial videos.
[0043] The above physiological signals and facial videos can be measured by different sensors. In this embodiment, Figure 2-3As shown, in addition to the display 1 and the data processor 3, the multimodal data collector of the miner's vital signs assessment device based on multimodal information fusion specifically includes: camera 2, temperature sensor 6, skin galvanic sensor 7, ECG sensor 8, blood pressure and blood oxygen sensor 9 and EEG cap 10; the ECG sensor, skin galvanic sensor, temperature sensor, blood pressure and blood oxygen sensor and camera are integrated on the device housing; the ECG sensor, skin galvanic sensor, temperature sensor, blood pressure and blood oxygen sensor, camera and EEG cap are connected to the data processor respectively. A charging port 4 is reserved on the side of the device for convenient charging, and an inspection door 5 is reserved on the front of the device for maintenance personnel to detect faults. The device housing is made of alloy, and four pulleys are installed underneath to facilitate the movement of the device.
[0044] Specifically in this embodiment, the EEG cap 10 includes: acquisition electrodes, EEG signal acquisition chip circuits, main control chip circuits, communication circuits, power management circuits and Bluetooth circuits; the EEG signal acquisition chip circuit contacts the miner's scalp through the acquisition electrodes to acquire the miner's EEG signals; the main control chip circuit communicates with the power management circuit, the EEG signal acquisition chip circuit and the Bluetooth circuit through the communication circuit, and is used to control the EEG signal acquisition chip circuit, the power management circuit and the Bluetooth circuit; the Bluetooth circuit is used to send the EEG signals acquired by the EEG signal acquisition chip circuit to the data processor 3.
[0045] In this embodiment, the data processor 3 is used to: use the physiological signal feature extraction module to respectively extract the physiological signal features of the electrocardiogram signal, the skin electrode signal, the body temperature signal, the blood pressure signal, the blood oxygen saturation signal and the electroencephalogram signal; use the facial data feature extraction module to extract the dynamic features and static features of the facial video; use the multi-feature fusion module to fuse the physiological signal features of the electrocardiogram signal, the skin electrode signal, the body temperature signal, the blood pressure signal, the blood oxygen saturation signal, the electroencephalogram signal, the dynamic features of the facial video and the static features of the facial video to obtain multimodal fusion features; use the physical sign status assessment module to determine the miner's emotional state, fatigue state and comprehensive physical sign state according to the multimodal fusion features.
[0046] The multimodal data collection process is as follows: Figure 4As shown, the miner wears an EEG cap 10, and sensors 5 to 9 adopt the finger contact measurement method. By placing the finger on the corresponding sensor, the data of the corresponding modality can be collected. Specifically, the ECG sensor 8 is used to collect the ECG signal of the miner, the skin electrode sensor 7 is used to measure the skin conductance of the miner, the temperature sensor 6 is used to measure the body temperature of the miner, and the blood pressure and blood oxygen sensor 9 is used to monitor the blood pressure and blood oxygen saturation of the miner. The camera 2 monitors the posture, facial expression and other information of the miner through image recognition technology, and saves the captured video in the form of image frames. The collected 7 kinds of vital sign data are first preprocessed; then the data processor 3 uses the physiological signal feature extraction (PSFE) model and the CNN+RNN network structure (BodySign Recognition Based on Multimodal Fusion, BSRM-MF) learning framework for fusion analysis and comprehensive evaluation.
[0047] Specifically, the time domain features and frequency domain features of EEG signals, ECG signals, skin electrical signals, body temperature signals, blood oxygen saturation signals, and blood pressure signals are extracted.
[0048] The PSFE model not only combines the time domain and frequency domain features of each modal data, but also combines the feature information of each modal before extracting the time domain and frequency domain features. Therefore, the use of the PSFE model can prevent information loss and fully capture the global information of the signal, thereby improving the performance of the model.
[0049] The facial data feature extraction module is used to extract dynamic features and static features of the facial video, which specifically includes the following steps:
[0050] The static features of the facial video are extracted using the facial data static feature extraction submodule; the facial data static feature extraction submodule is a convolutional neural network.
[0051] Convolutional Neural Networks (CNN) is a deep learning model that performs well in image recognition, video analysis, and natural language processing. The structure of CNN mainly includes the following key parts: input layer, convolution layer, pooling layer, and fully connected layer. The input layer mainly inputs image information; the convolution layer includes three convolution kernels with a kernel size of 7×7 and a 1×1 convolution kernel to reduce the feature dimension; a pooling layer is added after the convolution layer to prevent overfitting, and finally the facial static features are connected through the fully connected layer.
[0052] The dynamic features of the facial video are extracted using the facial data dynamic feature extraction submodule; the facial data dynamic feature extraction submodule is an RNN network.
[0053] Recurrent Neural Network (RNN) generally takes sequence data as input, effectively captures the relationship characteristics between sequences through the internal structure design of the network, and generally outputs in the form of sequences. The recurrence mechanism of RNN enables the results of the model layer at the previous time to be used as part of the input of the current time step (the input of the current time step includes the output of the layer of the previous step in addition to the normal input) to affect the output of the current time step, and at the same time increases the expressiveness of the model by using nonlinear activation functions.
[0054] The multi-feature fusion module is a Transformer model; the Transformer model includes an encoder and a decoder, which are key components working together in the Transformer model. The encoder is responsible for extracting and transforming features from the embedded sequence, mapping it into a high-dimensional representation space. Through the multi-head self-attention mechanism and the feedforward fully connected structure, the encoder can capture key information, as well as the correlation and complementarity between the embedded sequence features. The decoder generates the target sequence based on the high-dimensional representation and contextual information output by the encoder. In addition to the basic structure, the Transformer module also introduces residual connections and normalization to improve the training effect. Residual connections effectively alleviate the gradient vanishing problem and promote information flow, while normalization helps reduce internal covariate shifts and improve generalization capabilities.
[0055] The display 1 is used to display the emotional state, fatigue state and comprehensive physical state to the miners. In order to enable the miners to understand their own physiological information more intuitively, the display needs to display more information. For example, in one embodiment, the display is also used to display the electrocardiogram signal, skin electrical signal, body temperature signal, blood pressure signal, blood oxygen saturation signal, brain wave signal and facial video to the miners in the form of images, so that relevant personnel can understand the miners' physical state in time. Figure 5 shown.
[0056] Based on the same inventive concept, the embodiment of the present application also provides a method for assessing miners' vital signs based on multimodal information fusion, which is applied to the above-mentioned device for assessing miners' vital signs based on multimodal information fusion. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more embodiments of the method for assessing miners' vital signs based on multimodal information fusion provided below can be referred to the limitations of the device for assessing miners' vital signs based on multimodal information fusion above, and will not be repeated here.
[0057] In an exemplary embodiment, Figure 6 As shown, a miner's vital signs assessment method based on multimodal information fusion is provided, comprising the following steps:
[0058] S1. Collect multimodal data of miners and transmit the multimodal data to the data processor; the multimodal data includes electrocardiogram signals, skin electrical signals, body temperature signals, blood pressure signals, blood oxygen saturation signals, EEG signals and facial videos.
[0059] S2. Use the physiological signal feature extraction module to extract the physiological signal features of the electrocardiogram signal, skin electrical signal, body temperature signal, blood pressure signal, blood oxygen saturation signal and brain electrical signal respectively.
[0060] S3. Utilize the facial data feature extraction module to extract dynamic features and static features of the facial video.
[0061] S4. Use a multi-feature fusion module to fuse the physiological signal features, the dynamic features of the facial video, and the static features of the facial video to obtain a multimodal fusion feature.
[0062] S5. Use the physical sign status assessment module to determine the miners' emotional state, fatigue state and comprehensive physical sign state based on multimodal fusion features.
[0063] S6. Display the multimodal data, emotional state, fatigue state and comprehensive physical state to the miners through the display. In order to enable the miners to understand their own physiological information more intuitively, the display needs to display more information. For example, in one embodiment, the display is also used to display the electrocardiogram signal, skin electrical signal, body temperature signal, blood pressure signal, blood oxygen saturation signal, brain wave signal and facial video to the miners in the form of images, so that relevant personnel can understand the physical state of the miners in a timely manner.
[0064] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0065] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.
Claims
1. A miner's vital signs assessment device based on multimodal information fusion, characterized in that: include: Multimodal data collector, data processor and display; The multimodal data collector is used to collect multimodal data of the miners and transmit the multimodal data to the data processor; the multimodal data includes electrocardiogram signals, skin electrical signals, body temperature signals, blood pressure signals, blood oxygen saturation signals, brain electrical signals and facial videos; The data processor is used to: use a physiological signal feature extraction module to respectively extract the physiological signal features of the electrocardiogram signal, the skin electrode signal, the body temperature signal, the blood pressure signal, the blood oxygen saturation signal and the electroencephalogram signal; use a facial data feature extraction module to extract the dynamic features and static features of the facial video; use a multi-feature fusion module to fuse the physiological signal features of the electrocardiogram signal, the skin electrode signal, the body temperature signal, the blood pressure signal, the blood oxygen saturation signal, the electroencephalogram signal, the dynamic features of the facial video and the static features of the facial video to obtain a multimodal fusion feature; use a physical sign status assessment module to determine the miner's emotional state, fatigue state and comprehensive physical sign state according to the multimodal fusion feature; The display is used to show the emotional state, the fatigue state and the comprehensive physical sign state to the miner.
2. The miner's vital signs assessment device based on multimodal information fusion according to claim 1 is characterized in that: The multimodal data collector specifically includes: an electrocardiogram sensor, an electrical skin sensor, a temperature sensor, a blood pressure and blood oxygen sensor, an electroencephalogram cap and a camera; the electrocardiogram sensor, the electrical skin sensor, the temperature sensor, the blood pressure and blood oxygen sensor and the camera are integrated on the device housing; the electrocardiogram sensor, the electrical skin sensor, the temperature sensor, the blood pressure and blood oxygen sensor, the camera and the electroencephalogram cap are respectively connected to the data processor.
3. The miner's vital signs assessment device based on multimodal information fusion according to claim 2, characterized in that: The EEG cap includes: acquisition electrodes, EEG signal acquisition chip circuit, main control chip circuit, communication circuit, power management circuit and Bluetooth circuit; the EEG signal acquisition chip circuit contacts the miner's scalp through the acquisition electrodes to acquire the miner's EEG signals; the main control chip circuit communicates with the power management circuit, the EEG signal acquisition chip circuit and the Bluetooth circuit through the communication circuit to control the EEG signal acquisition chip circuit, the power management circuit and the Bluetooth circuit; the Bluetooth circuit is used to send the EEG signals acquired by the EEG signal acquisition chip circuit to the data processor.
4. The miner's vital signs assessment device based on multimodal information fusion according to claim 1, characterized in that: The data processor is specifically used to extract the time domain features and frequency domain features of the EEG signal, the ECG signal, the galvanic skin signal, the body temperature signal, the blood oxygen saturation signal, and the blood pressure signal.
5. The miner's vital signs assessment device based on multimodal information fusion according to claim 1, characterized in that: The facial data feature extraction module is used to extract the dynamic features and static features of the facial video, specifically including: The static features of the facial video are extracted using a facial data static feature extraction submodule; the facial data static feature extraction submodule is a convolutional neural network; The dynamic features of the facial video are extracted using a facial data dynamic feature extraction submodule; the facial data dynamic feature extraction submodule is a recurrent neural network.
6. The miner's vital signs assessment device based on multimodal information fusion according to claim 5, characterized in that: The convolutional neural network includes the following key parts: an input layer, a convolution layer, a pooling layer and a fully connected layer; the input layer is used to input image information; the convolution layer includes three convolution kernels with a kernel size of 7×7 and one convolution kernel of 1×1 to reduce the feature dimension; a pooling layer is added after the convolution layer to prevent overfitting, and finally the static features of the facial video are connected through the fully connected layer.
7. The miner's vital signs assessment device based on multimodal information fusion according to claim 5, characterized in that: The recurrent neural network takes sequence data as input, effectively captures the relationship characteristics between sequences through the internal structural design of the network, and also outputs in the form of sequences; the loop mechanism of the recurrent neural network enables the result generated by the model layer at the previous time to be used as part of the input of the current time step, affecting the output of the current time step, and at the same time increasing the expressive power of the model by using nonlinear activation functions.
8. The miner's vital signs assessment device based on multimodal information fusion according to claim 1, characterized in that: The multi-feature fusion module is a Transformer model; the Transformer model includes an encoder and a decoder; the encoder is responsible for extracting and transforming features from the embedded sequence and mapping it to a high-dimensional representation space; through a multi-head self-attention mechanism and a feedforward fully connected structure, the encoder is used to capture key information, as well as the correlation and complementarity between embedded sequence features; the decoder generates a target sequence based on the high-dimensional representation and context information output by the encoder. In addition to the basic structure, the Transformer model may also include residual connections and normalization to improve training effects; the residual connections effectively alleviate the gradient vanishing problem and promote information flow; the normalization helps to reduce internal covariate shifts and improve generalization capabilities.
9. The miner's vital signs assessment device based on multimodal information fusion according to claim 1, characterized in that: The display is also used to display the electrocardiogram signal, the skin electrical signal, the body temperature signal, the blood pressure signal, the blood oxygen saturation signal, the electroencephalogram signal and the facial video to the miner in the form of an image.
10. A method for assessing miners' vital signs based on multimodal information fusion, characterized in that: include: Collecting multimodal data of miners and transmitting the multimodal data to the data processor; The multimodal data includes electrocardiogram signals, skin electrical signals, body temperature signals, blood pressure signals, blood oxygen saturation signals, brain electrical signals and facial videos; Using a physiological signal feature extraction module to respectively extract physiological signal features of the electrocardiogram signal, the skin electrical signal, the body temperature signal, the blood pressure signal, the blood oxygen saturation signal, and the electroencephalogram signal; Utilizing a facial data feature extraction module to extract dynamic features and static features of the facial video; Using a multi-feature fusion module, the physiological signal features of the electrocardiogram signal, the physiological signal features of the electrical skin signal, the physiological signal features of the body temperature signal, the physiological signal features of the blood pressure signal, the physiological signal features of the blood oxygen saturation signal, the physiological signal features of the electroencephalogram signal, the dynamic features of the facial video, and the static features of the facial video are fused to obtain a multimodal fusion feature; Using the physical sign status assessment module to determine the miner's emotional state, fatigue state and comprehensive physical sign state according to the multimodal fusion features; The multimodal data, the emotional state, the fatigue state and the comprehensive physical sign state are presented to the miner via a display.
Citation Information
Patent Citations
Brain electrical signal and physiological signal fused fatigue detection system
CN106691474A
A face emotion recognition method based on a double-flow convolutional neural network
CN109815785A
Wearable multi-mode emotional state monitoring device
CN112120716A
Non-contact fatigue detection method and system
CN113420624A
Mental disease auxiliary diagnosis system based on multi-modal analysis
CN117133438A
Cited By
Miner behavior monitoring method and device
CN119337189A
Mine operator state identification method, equipment, medium and product
CN120495829A
Method and system for detecting physical state of operator based on multiple physiological signals
CN121621982A
