State monitoring method and system based on multi-modal fusion features of electroencephalogram and electrocardio

Abstract

The invention provides a state monitoring method and system based on multi-modal fusion features of electroencephalogram and electrocardio. The electroencephalogram and electrocardio-based multi-mode fusion feature state monitoring method comprises the following steps: acquiring a simulated data set of an electroencephalogram and an electrocardiogram of a testee and a real data set of an electroencephalogram and an electrocardiogram of a worker in an actual scene; determining a training data set according to the simulation data set and the real data set; adopting the training data set to train a preset deep learning neural network, and determining a physiological and psychological state monitoring model fusing electroencephalogram and electrocardio characteristics; and inputting the simulation data set of the electroencephalogram and the electrocardiogram of the to-be-tested person into the physiological and psychological state monitoring model fusing the electroencephalogram and electrocardiogram characteristics, and determining the physiological and psychological state of the to-be-tested person. According to the method and the device, the electroencephalogram and the electrocardiogram are adopted for multi-modal analysis, so that the physiological and psychological states of the user are quickly analyzed, the monitoring process is simplified, and the state monitoring accuracy is improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of biological signal processing and artificial intelligence technology, and in particular, to a state monitoring method and system based on multimodal fusion features of electroencephalogram (EEG) and electrocardiogram (ECG). Background Art

[0002] The physiological and psychological state of a person is a complex and ubiquitous physiological and psychological phenomenon, which is widely present in daily life, such as medical diagnosis, sports training, and high-pressure work. For example, in a high-pressure working environment, a person's physiological and psychological state will change significantly, which is a phenomenon of the human body adapting to the environment. Changes in physiological and psychological states can significantly affect a person's cognition, emotions, and behaviors, and play an important role in understanding human cognitive functions, emotional regulation, and other research. In order to fully understand the changes in physiological and psychological states, it is necessary to accurately measure a person's physiological and psychological state.

[0003] The measurement of human physiological and psychological states mainly relies on two detection methods: electroencephalogram (EEG) and electrocardiogram (ECG). EEG is a method of recording brain electrical activity. Electrodes are placed on the scalp to capture the electrical signals of the cerebral cortex, namely, electroencephalogram (EEG) signals. EEG signals reflect the activity patterns of the brain and can be used to study cognitive processes. However, the quality of EEG signals is extremely dependent on the placement of electrodes and skin resistance, and is easily interfered by noise. Moreover, the detection electrodes can only be used to measure the electrical activity of the cerebral cortex, and cannot measure the activity of deep brain regions. Despite this, EEG is still an important tool for studying human physiological and psychological states because it can provide millisecond-level time resolution, allowing researchers to study rapid changes in brain activity. Electrocardiogram (ECG) is a method of recording cardiac electrical activity. Electrodes are placed on the surface of the body to capture the electrical signals of the heart, namely, electrocardiogram (ECG) signals. ECG signals can be used to study the functional state of the heart, such as heart rate and the function of the cardiac conduction system. The quality of ECG signals is also extremely dependent on the placement of electrodes and skin resistance, and is easily interfered by noise. Moreover, ECG can only be used to measure the electrical activity of the heart, and cannot directly measure the mechanical activity of the heart, such as cardiac contraction and relaxation. Electroencephalogram (EEG) and electrocardiogram (ECG) are important tools for studying a person's physiological and psychological state, and each has its own limitations.

[0004] When measuring a person's physiological and psychological state, it is necessary to overcome the problems of environmental changes, rapid changes in physiological and psychological states, unclear signal acquisition, large data recognition and counting errors, and slow response speed. Conventional methods based on multiple sensors increase the complexity of the system and significantly increase the amount of data and the difficulty of feature extraction. Summary of the invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a state monitoring method and system based on multimodal fusion features of EEG and ECG.

[0006] To achieve the above object, according to a first aspect of the present disclosure, a state monitoring method based on multimodal fusion features of EEG and ECG is provided, comprising:

[0007] Obtain simulated data sets of EEG and ECG of test subjects and real data sets of EEG and ECG of staff in actual scenarios;

[0008] Determine a training data set according to the simulated data set and the real data set;

[0009] The training data set is used to train a preset deep learning neural network to determine a physiological and psychological state monitoring model that integrates EEG and ECG features;

[0010] The electroencephalogram (EEG) and electrocardiogram (ECG) data of the person to be tested are input into the physiological and psychological state monitoring model integrating EEG and ECG characteristics to determine the physiological and psychological state of the person to be tested.

[0011] Optionally, determining the training data set according to the simulated data set and the real data set includes:

[0012] Preprocessing the simulated data set to determine a preprocessed simulated data set;

[0013] Performing sliding window segmentation processing on the preprocessed simulation data set to determine a first data set;

[0014] Preprocessing the real data set to determine a preprocessed real data set;

[0015] Performing scene annotation processing on the preprocessed real data set to determine a second data set;

[0016] The second training data set is aligned with the first training data set to determine the training data set.

[0017] Optionally, the preprocessing includes any one or more combinations of filtering, denoising, regularization, and data transformation.

[0018] Optionally, the physiological and psychological state monitoring model integrating EEG and ECG features includes an EEG classifier, an ECG classifier, a dynamic multimodal fusion module and a multimodal feature classifier.

[0019] Optionally, the using the training data set to train a preset deep learning neural network to determine a physiological and psychological state monitoring model integrating EEG and ECG features includes:

[0020] Converting the electroencephalogram data of the training data set from one-dimensional disordered spatial data to two-dimensional ordered spatial matrix data to determine the two-dimensional ordered spatial matrix data of the electroencephalogram data of the training data set;

[0021] Inputting the two-dimensional ordered spatial matrix data of the electroencephalogram data of the training data set into the electroencephalogram classifier, and outputting the electroencephalogram features and the true class probability of the electroencephalogram modality;

[0022] Inputting the electrocardiogram data in the training data set into the electrocardiogram classifier, and outputting the electrocardiogram features and the true class probability of the electrocardiogram modality;

[0023] Input the EEG feature, the true class probability of the EEG modality, the ECG feature, and the true class probability of the ECG modality into the dynamic multimodal fusion module, and output the EEG-ECG multimodal feature;

[0024] The EEG and ECG multimodal features are input into the multimodal feature classifier, and the alertness or arousal degree is output.

[0025] Optionally, the step of inputting the EEG feature, the true class probability of the EEG modality, the ECG feature, and the true class probability of the ECG modality into the dynamic multimodal fusion module, and outputting the EEG / ECG multimodal feature, comprises:

[0026] Assigning feature weights to the EEG features and the ECG features, and determining the EEG key feature information and the ECG key feature information;

[0027] According to the true class probability of the EEG modality and the true class probability of the ECG modality, modality weights are assigned to the EEG key feature information and the ECG key feature information to determine the EEG-ECG multimodal features.

[0028] Optionally, inputting the electroencephalogram (EEG) and electrocardiogram (ECG) data of the person to be tested into the physiological and psychological state monitoring model integrating EEG and ECG features to determine the physiological and psychological state of the person to be tested includes:

[0029] Inputting the EEG and ECG data of the person to be tested into the physiological and psychological state monitoring model integrating EEG and ECG features, and outputting the alertness or arousal degree of the person to be tested;

[0030] Determine the physiological and psychological state of the person to be tested based on the alertness or arousal of the person to be tested

[0031] Optionally, determining the physiological and psychological state of the person to be tested according to the alertness or arousal of the person to be tested includes:

[0032] If the alertness or arousal of the person to be tested is a positive class, it is determined that the physiological and psychological state of the person to be tested is a tense state;

[0033] If the alertness or arousal of the person to be tested is a negative class, it is determined that the physiological and psychological state of the person to be tested is a relaxed state.

[0034] According to a second aspect of the present disclosure, a state monitoring system based on multimodal fusion features of EEG and ECG is provided, comprising:

[0035] A data acquisition module is used to obtain a simulated data set of the electroencephalogram and electrocardiogram of the person to be tested and a real data set of the electroencephalogram and electrocardiogram of the staff in the actual scene;

[0036] A data processing module, used to determine a training data set according to the simulated data set and the real data set;

[0037] A model training module, used to train a preset deep learning neural network using the training data set to determine a physiological and psychological state monitoring model that integrates EEG and ECG features;

[0038] The state monitoring module is used to input the simulated data sets of the electroencephalogram and electrocardiogram of the person to be tested into the physiological and psychological state monitoring model that integrates the electroencephalogram and electrocardiogram characteristics to determine the physiological and psychological state of the person to be tested.

[0039] According to a third aspect of the present disclosure, a state monitoring device based on multimodal fusion features of EEG and ECG is provided, comprising:

[0040] An EEG sensor module is used to collect EEG data of the person to be tested;

[0041] An ECG sensor module is used to collect ECG data of the person to be tested;

[0042] A data transmission module, used for transmitting the electroencephalogram data and the electrocardiogram data;

[0043] The terminal is used to store and display the electroencephalogram data and the electrocardiogram data.

[0044] According to a fourth aspect of the present disclosure, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of any one of the methods provided in the first aspect of the present disclosure are implemented.

[0045] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

[0046] Through the above technical scheme, simulated data sets of the EEG and ECG of the subjects and real data sets of the EEG and ECG of the staff in the actual scene are obtained, and multimodal analysis is performed on the EEG data and ECG data, thereby improving the accuracy of monitoring the physiological and psychological state of the user. In addition, the real data set and the simulated data set are used as training data sets to train and process the preset deep learning neural network, and a physiological and psychological state monitoring model that integrates EEG and ECG features is determined. The physiological and psychological state monitoring model that integrates EEG and ECG features is used to predict the physiological and psychological state of the person to be tested, and multiple modal data are integrated together to achieve rapid and automatic analysis of the user's physiological and psychological state, simplify the physiological and psychological state monitoring process, and improve the efficiency of physiological and psychological state monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0048] Figure 1 The present invention is a flowchart of a state monitoring method based on multimodal fusion features of EEG and ECG according to an exemplary embodiment.

[0049] Figure 2 It is a schematic diagram of the structure of a physiological and psychological state monitoring model integrating EEG and ECG features according to an exemplary embodiment.

[0050] Figure 3 It is a schematic diagram of an internationally used 10-20 electrode distribution system according to an exemplary embodiment.

[0051] Figure 4 It is a block diagram of a state monitoring device based on multimodal fusion features of EEG and heart points according to an exemplary embodiment.

[0052] Figure 5 It is a block diagram of a state monitoring system based on multimodal fusion features of EEG and ECG according to an exemplary embodiment. DETAILED DESCRIPTION

[0053] The present disclosure is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but are not intended to limit the present disclosure in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present disclosure. These all fall within the scope of protection of the present disclosure.

[0054] In this disclosure, the terms are explained as follows:

[0055] Alertness refers to an individual's sensitivity and preparedness to potential threats or important information in a specific environment.

[0056] Arousal refers to the intensity of an individual's experience of an emotion when it occurs.

[0057] The true class probability refers to the degree of certainty of the model about the classification result. The higher the true class probability, the higher the model classification credibility. The lower the true class probability, the lower the model classification credibility.

[0058] Figure 1 The present invention is a flowchart of a state monitoring method based on multimodal fusion features of EEG and ECG according to an exemplary embodiment.

[0059] like Figure 1 As shown, the present disclosure provides a state monitoring method based on multimodal fusion features of EEG and ECG, including S11 to S14.

[0060] S11, obtaining a simulated data set of the electroencephalogram (EEG) and electrocardiogram (ECG) of the test subjects and a real data set of the electroencephalogram (EEG) and electrocardiogram (ECG) of the staff in the actual scene.

[0061] The person to be tested can be shown audio and video with target emotions, and the electroencephalogram and electrocardiogram of the subject can be collected to determine a simulated data set of the electroencephalogram and electrocardiogram of the person to be tested.

[0062] The EEG and ECG of workers working in high-voltage scenes can be collected to determine a real data set of the EEG and ECG of workers in actual scenes.

[0063] S12, determining a training data set according to the simulated data set and the real data set.

[0064] The training data set includes a simulated data set and a real data set.

[0065] The simulated data set refers to a data set in which subjects are stimulated to watch different videos and the alertness / arousal scores are obtained by recording the subjects' electroencephalograms, electrocardiograms, and the physiological and psychological states assessed by the subjects themselves after watching the videos.

[0066] The real data set refers to a data set composed of alertness / arousal scores obtained by recording the subjects' electroencephalograms, electrocardiograms, and physiological and psychological states at work in actual work scenarios.

[0067] In order to improve the training effect and final accuracy of the model, in a possible embodiment, S12 may include S21 to S25.

[0068] S21, preprocessing the simulated data set to determine a preprocessed simulated data set.

[0069] The preprocessing includes any one or more combinations of filtering, denoising, regularization, and data transformation.

[0070] In this step, by preprocessing the simulated data set and the real data set, the data quality can be improved and the stability of subsequent model training can be enhanced.

[0071] S22, performing sliding window segmentation processing on the preprocessed simulation data set to determine a first data set.

[0072] In this step, the simulated data set is segmented using a sliding window to increase the amount of training data, increase the feature density, and reduce the risk of overfitting in model training.

[0073] S23, preprocessing the real data set to determine the preprocessed real data set.

[0074] The preprocessing includes any one or more combinations of filtering, denoising, regularization, and data transformation.

[0075] S24, performing scoring and labeling processing on the preprocessed real data set to determine a second data set.

[0076] Among them, the score labeling processing refers to manually labeling the alertness / arousal scores obtained from the physiological and psychological states of the subjects at work as labels in the real data set according to the actual work scenarios.

[0077] In this step, scoring and labeling the preprocessed real data set helps the model learn the relationship between EEG and ECG data and alertness / arousal.

[0078] S25, performing data alignment processing on the second training data set and the first training data set to determine the training data set.

[0079] In this step, the data alignment method is used to improve the data consistency between the simulated data set and the real data set, thereby improving the model training effect.

[0080] S13, using the training data set to train the preset deep learning neural network to determine the physiological and psychological state monitoring model that integrates EEG and ECG features.

[0081] Figure 2 It is a schematic diagram of the structure of a physiological and psychological state monitoring model integrating EEG and ECG features according to an exemplary embodiment.

[0082] like Figure 2As shown, the physiological and psychological state monitoring model integrating EEG and ECG features includes an EEG classifier, an ECG classifier, a dynamic multimodal fusion module, and a multimodal feature classifier.

[0083] The electroencephalogram classifier includes a first encoder and a first classifier; the electrocardiogram classifier includes a second encoder and a second classifier.

[0084] The EEG classifier, ECG classifier, dynamic multimodal fusion module and multimodal feature classifier are all optimized using the Adam optimizer.

[0085] The first encoder of the EEG classifier includes three layers of convolutional neural networks and two layers of long short-term memory networks. Each layer of the convolutional neural network is connected to a maximum pooling layer and an activation layer, wherein the activation layer can use the ReLU function as the activation function. Each layer of the neural network also includes a dropout layer. The first encoder is used to extract EEG features.

[0086] The maximum pooling layer and activation layer are used to improve computational efficiency and increase feature density. The elimination layer is used to randomly reset the weights of some neurons to zero to reduce the risk of overfitting. The long short-term memory network is used to extract the temporal information of the EEG signal from the feature vector extracted by the convolutional neural network.

[0087] The first classifier includes a fully connected layer, the fully connected layer of the first classifier is used to calculate the true class probability of the electroencephalogram modality, and the first classifier outputs the true class probability of the electroencephalogram modality for dynamic multimodal fusion of the electroencephalogram modality.

[0088] The second encoder of the electrocardiogram classifier includes three groups of convolutional neural networks, each group of convolutional neural networks includes two layers of convolutional neural networks, each layer of convolutional neural network is connected to a maximum pooling layer and an activation layer, wherein the activation layer can use the ReLU function as the activation function, and the second encoder is used to extract electrocardiogram features.

[0089] The second classifier includes a fully connected layer, the fully connected layer of the second classifier is used to calculate the true class probability of the electrocardiogram modality, and the second classifier outputs the true class probability of the electrocardiogram modality for dynamic multimodal fusion of the electrocardiogram modality.

[0090] The dynamic multimodal fusion layer is used to assign feature weights and modal weights. By assigning feature weights, key feature information is retained. The modal weights are assigned using the true class probability of the modality, key modal information is retained, and the impact of useless information on the prediction results is reduced.

[0091] The multimodal feature classifier includes a fully connected layer and a SoftMax layer. The fully connected layer of the multimodal feature classifier is used for feature learning, and the SoftMax layer is used for classification prediction. The multimodal feature classifier is used to predict the user's alertness or arousal, calculate the cross entropy error between the predicted alertness and the actual alertness or between the arousal and the actual arousal, and optimize the parameters of the neural network through the optimizer to achieve the optimal model prediction result.

[0092] S14, inputting the EEG and ECG data of the person to be tested into a physiological and psychological state monitoring model integrating EEG and ECG characteristics to determine the physiological and psychological state of the person to be tested.

[0093] A physiological and psychological state monitoring model that integrates EEG and ECG features is used to predict the alertness or arousal of the person under test based on the EEG and ECG data of the person under test, and to judge the physiological and psychological state of the person under test based on the alertness or arousal of the person under test.

[0094] Through the above technical scheme, simulated data sets of the EEG and ECG of the subjects and real data sets of the EEG and ECG of the staff in the actual scene are obtained, and multimodal analysis is performed on the EEG data and ECG data, thereby improving the accuracy of monitoring the physiological and psychological state of the user. In addition, the real data set and the simulated data set are used as training data sets to train and process the preset deep learning neural network, and a physiological and psychological state monitoring model that integrates EEG and ECG features is determined. The physiological and psychological state monitoring model that integrates EEG and ECG features is used to predict the physiological and psychological state of the person to be tested, and multiple modal data are integrated together to achieve rapid and automatic analysis of the user's physiological and psychological state, simplify the physiological and psychological state monitoring process, and improve the efficiency of physiological and psychological state monitoring.

[0095] In a possible embodiment, before using the training data set to train the preset deep learning neural network and determine the physiological and psychological state monitoring model integrating EEG and ECG features, it also includes:

[0096] The EEG data in the training data set are filtered to retain the data with a frequency of 4 Hz to 30 Hz and eliminate noise and artifacts in the EEG signal.

[0097] After filtering, the EEG data is subjected to baseline removal to highlight the dynamic changes of EEG signals and reduce the impact of differences between different experimental subjects on the experimental results.

[0098] After the baseline removal operation, the EEG data is normalized to adjust the values ​​of the EEG data to the same scale to improve the generalization ability of the model.

[0099] In another possible embodiment, before using the training data set to train the preset deep learning neural network and determine the physiological and psychological state monitoring model integrating EEG and ECG features, it also includes:

[0100] The ECG data in the training data set are filtered and the data with a frequency greater than 0.8 Hz are retained to eliminate noise, obtain pure and reliable ECG data, and improve the accuracy of the ECG data.

[0101] Through the above technical solution, the EEG data in the training data set are filtered, baseline removed and normalized in turn, and the ECG data are filtered, which can improve the robustness and accuracy of the subsequent training model.

[0102] In a possible embodiment, S13 may include S31 to S35.

[0103] S31, converting the electroencephalogram data of the training data set from one-dimensional disordered space data into two-dimensional ordered space matrix data, and determining the two-dimensional ordered space matrix data of the electroencephalogram data of the training data set.

[0104] Among them, the EEG data in the training data set can be converted from one-dimensional disordered spatial data to two-dimensional ordered spatial matrix data through spatial encoding.

[0105] Figure 3 It is a schematic diagram of an internationally used 10-20 electrode distribution system according to an exemplary embodiment.

[0106] like Figure 3 As shown, as an example, the internationally used 10-20 electrode distribution system can be used to spatially encode EEG data.

[0107] Specifically, the position of the scalp surface is defined according to the internationally accepted 10-20 electrode distribution system, and the EEG data at different positions are mapped to a 9*9 single-channel matrix, thereby converting the one-dimensional disordered spatial data of the EEG data into two-dimensional ordered spatial matrix data.

[0108] By performing spatial encoding processing on the EEG data, the one-dimensional disordered spatial data of the EEG is converted into two-dimensional ordered spatial matrix data, which facilitates a better understanding of the distribution of EEG signals on the scalp.

[0109] S32, inputting the two-dimensional ordered spatial matrix data of the EEG data of the training data set into the EEG classifier, and outputting the EEG features and the true class probability of the EEG modality.

[0110] In the first encoder of the EEG classifier, a convolutional neural network is used to extract the feature vector of spatial information of the EEG data after spatial encoding processing, and a long short-term memory network is used to extract the feature vector of temporal information of the feature vector of spatial information extracted by the convolutional neural network. The first encoder outputs the EEG features.

[0111] In the first classifier of the EEG classifier, a fully connected layer is used to calculate the true class probability of the EEG modality for the feature vectors of spatial information of the EEG data that has been spatially encoded and extracted by the convolutional neural network in the first encoder and the feature vectors of temporal information extracted by the long short-term memory network, and the first classifier outputs the true class probability of the EEG modality.

[0112] S33, inputting the electrocardiogram data in the training data set into the electrocardiogram classifier, and outputting the electrocardiogram features and the true class probability of the electrocardiogram modality.

[0113] In the second encoder of the electrocardiogram classifier, a convolutional neural network is used to extract the feature vector of the time information of the electrocardiogram data of the training data set, and the second encoder outputs the electrocardiogram features.

[0114] In the second classifier of the electrocardiogram classifier, a fully connected layer is used to calculate the true class probability of the electrocardiogram modality for the feature vector of the time information of the electrocardiogram data of the training data set extracted by the convolutional neural network in the second encoder, and the second classifier outputs the true class probability of the electrocardiogram modality.

[0115] S34, inputting the EEG features, the true class probability of the EEG modality, the ECG features, and the true class probability of the ECG modality into the dynamic multimodal fusion module, and outputting the EEG-ECG multimodal features.

[0116] In a possible embodiment, in the dynamic multimodal fusion module, feature weights are assigned to the EEG features and the ECG features to determine the EEG key feature information and the ECG key feature information; the true class probability of the EEG modality and the true class probability of the ECG modality are used to assign modal weights to the EEG features and the ECG features, respectively, to determine the EEG-ECG multimodal features.

[0117] By assigning feature weights to EEG features and ECG features, intra-modal analysis is performed to retain key feature information; by assigning modal weights to EEG features and ECG features, inter-modal analysis is performed to retain key modal information and reduce the impact of useless information on state prediction results.

[0118] S35, inputting the EEG and ECG multimodal features into a multimodal feature classifier, and outputting the alertness or arousal level.

[0119] Among them, alertness refers to an individual's sensitivity and preparedness to potential threats or important information in a specific environment.

[0120] Arousal refers to the intensity of an individual's experience of an emotion when it occurs.

[0121] By using a training data set that is a mixture of simulated data sets and real data sets to train a preset deep learning neural network, the neural network can learn the relationship between the electroencephalogram, electrocardiogram and alertness and arousal. The obtained physiological and psychological state monitoring model that integrates EEG and ECG characteristics can predict the current alertness or arousal of the person under test through the EEG and ECG data of the person under test.

[0122] In a possible embodiment, S14 may include S41 to S42.

[0123] S41, inputting the electroencephalogram signal and electrocardiogram signal of the person to be tested into a dynamic multimodal classifier, and outputting the alertness or arousal degree of the person to be tested.

[0124] As an example, the EEG and ECG data of the person to be tested are input into a physiological and psychological state monitoring model that integrates EEG and ECG features, and the EEG data is spatially encoded to convert the EEG data from one-dimensional disordered spatial data into two-dimensional ordered spatial matrix data; the two-dimensional ordered spatial matrix data of the EEG data of the training data set is input into the first encoder of the EEG classifier to output the EEG features; the EEG features are input into the first classifier to output the true class probability of the EEG modality; the ECG data is input into the second encoder to output the ECG features; the ECG features are input into the second classifier to output the true class probability of the ECG modality; the EEG features, the ECG features, the true class probability of the EEG modality, and the true class probability of the ECG modality are input into the dynamic multimodal fusion module to output the EEG-ECG multimodal features; the EEG-ECG multimodal features are input into the multimodal feature classifier to output the alertness or arousal.

[0125] S42, determining the physiological and psychological state of the person to be tested according to the alertness or arousal level of the person to be tested.

[0126] Among them, the physiological and psychological state of the person to be tested includes high / low alertness and high / low arousal. The alertness threshold / arousal threshold can be preset to distinguish the high / low alertness or high / low arousal of the person to be tested, that is, the positive and negative categories of alertness / arousal, so as to achieve the purpose of classifying the physiological and psychological state.

[0127] When the alertness score is not less than the preset alertness threshold, it means that the alertness of the person to be tested is a positive class. When the alertness score is less than the preset alertness threshold, it means that the alertness of the person to be tested is a negative class.

[0128] When the arousal score is not less than the preset arousal threshold, it indicates that the arousal of the person to be tested is a positive class. When the current arousal score is less than the preset arousal threshold, it indicates that the arousal of the person to be tested is a negative class.

[0129] As an example, if the alertness or arousal of the person to be tested is a positive class, it is determined that the physiological and psychological state of the person to be tested is a tense state.

[0130] When the alertness or arousal of the person under test is positive, the person under test is more sensitive to potential threats or important information in the current environment, is in a better state of preparedness, and can respond quickly under special circumstances. In addition, the person under test has a stronger experience of emotions and feels a strong state of tension.

[0131] As another example, if the alertness or arousal of the person to be tested is a negative class, it is determined that the physiological and psychological state of the person to be tested is a relaxed state.

[0132] When the alertness or arousal of the person under test is negative, the person under test is less sensitive to potential threats or important information in the current environment, is slightly less prepared, reacts more slowly in special circumstances, and experiences emotions more slowly, and feels relaxed.

[0133] In a possible embodiment, the state monitoring method based on multimodal fusion features of EEG and ECG proposed in the present disclosure can be used to monitor the physiological and psychological state of high-voltage workers. The EEG data and ECG data of the high-voltage workers are collected, and the physiological and psychological state monitoring model that integrates EEG and ECG features trained above is input to quickly output the predicted alertness or arousal of the high-voltage workers to determine whether the current physiological and psychological state of the high-voltage workers is in a tense state, and to make timely intervention plans to ensure the physical health of the high-voltage workers.

[0134] Figure 4 It is a block diagram of a state monitoring device based on multimodal fusion features of EEG and heart points according to an exemplary embodiment.

[0135] like Figure 4 As shown, in a possible embodiment, the present disclosure also provides a state monitoring device based on multimodal fusion features of EEG and ECG, including: an EEG sensor, an ECG sensor, a data transmission module and a terminal.

[0136] An EEG sensor module is used to collect EEG data of the person to be tested;

[0137] An ECG sensor module is used to collect ECG data of the person to be tested;

[0138] A data transmission module, used for transmitting EEG data and ECG data;

[0139] The terminal is used for storing and displaying EEG data and ECG data.

[0140] As an example, the EEG sensor module uses a headset equipped with an EEG sensor, and the ECG sensor module can use a wristband ECG sensor.

[0141] The EEG data of the person under test collected by the EEG sensor module includes EEG power spectrum, eSense concentration index, eSense relaxation index and blinking information.

[0142] A headset equipped with EEG sensors is worn on the head of the person to be tested, and electrodes are placed on the scalp to record the voltage fluctuations generated by the ionic current of brain neurons, measure the synchronous sum of postsynaptic potentials when pyramidal cells are excited, and collect EEG signals, i.e., EEG data.

[0143] The original EEG data is a discrete time series data consisting of multiple sample points. Each sample point data represents the amplitude of a brain wave, and its unit is microvolt (uV).

[0144] The ECG sensor in the ECG sensor module can adopt a photoelectric reflective analog sensor, which is worn on the human wrist. It measures the ECG signal, i.e., the ECG data, by the different light transmittances generated by human tissue when blood vessels fluctuate. The ECG sensor module filters and amplifies the photoelectric signal and outputs an analog voltage value in millivolts (mV).

[0145] In a possible embodiment, the EEG data collected by the EEG sensor module and the ECG data collected by the ECG sensor module are processed separately, and the data processing may adopt any one or more combinations of noise reduction, filtering and standardization processing.

[0146] The computing power required for data processing is determined by the required measurement speed. The higher the computing power of the data processing system, the higher the measurement speed and calculation speed that can be achieved.

[0147] The data transmission module can use 4G to transmit the processed EEG data and ECG data to the terminal via HTTP protocol or MQTT protocol.

[0148] HTTP protocol requests and responses have high transmission speed and low latency.

[0149] The MQTT protocol has a small transmission volume and has high reliability and low latency.

[0150] The terminal performs subsequent detection and evaluation tasks.

[0151] When implementing the state monitoring method based on the multimodal fusion features of EEG and ECG provided by the present disclosure, the state monitoring device based on the multimodal fusion features of EEG and ECG provided by the present disclosure can be used to collect EEG signals and ECG signals, that is, EEG data and ECG data, of the subjects, the staff in the actual scene, and the persons to be tested.

[0152] Figure 5 It is a block diagram of a state monitoring system based on multimodal fusion features of EEG and ECG according to an exemplary embodiment.

[0153] Based on the same concept, the present disclosure also provides a state monitoring system based on multi-modal fusion features of EEG and ECG, such as Figure 5 As shown, it includes a data acquisition module 110, a data processing module 120, a model training module 130, and a status monitoring module 140.

[0154] The data acquisition module 110 is used to obtain the simulated data sets of the electroencephalogram and electrocardiogram of the person to be tested and the real data sets of the electroencephalogram and electrocardiogram of the staff in the actual scene;

[0155] A data processing module 120, configured to determine a training data set according to the simulated data set and the real data set;

[0156] A model training module 130 is used to train a preset deep learning neural network using the training data set to determine a physiological and psychological state monitoring model that integrates EEG and ECG features;

[0157] The state monitoring module 140 is used to input the simulated data sets of the electroencephalogram and electrocardiogram of the person to be tested into the physiological and psychological state monitoring model integrating the electroencephalogram and electrocardiogram characteristics to determine the physiological and psychological state of the person to be tested.

[0158] Through the above technical scheme, simulated data sets of the EEG and ECG of the subjects and real data sets of the EEG and ECG of the staff in actual scenarios are obtained, and multimodal analysis is performed on the EEG data and ECG data, thereby improving the accuracy of monitoring the physiological and psychological state of the user. In addition, the real data set is used as the training data set to train the preset deep learning neural network, and a physiological and psychological state monitoring model that integrates EEG and ECG features is determined. The physiological and psychological state monitoring model that integrates EEG and ECG features is used to predict the physiological and psychological state of the person to be tested, thereby realizing rapid and automatic analysis of the user's physiological and psychological state, simplifying the physiological and psychological state monitoring process, and improving the efficiency of physiological and psychological state monitoring.

[0159] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0160] Based on the same concept as above, in another embodiment of the present disclosure, there is also provided an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor is used to execute a state monitoring method based on multimodal fusion features of EEG and ECG when executing the program.

[0161] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic RandomAccess Memory, abbreviated: DDRSDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0162] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0163] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0164] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0165] In an embodiment of the present disclosure, a non-temporary computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a state monitoring method based on multimodal fusion features of EEG and ECG in any of the above embodiments are implemented.

[0166] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0170] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.

[0171] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A state monitoring method based on multimodal fusion features of EEG and ECG, characterized in that: include, Obtain simulated data sets of EEG and ECG of test subjects and real data sets of EEG and ECG of staff in actual scenarios; Determine a training data set according to the simulated data set and the real data set; The training data set is used to train a preset deep learning neural network to determine a physiological and psychological state monitoring model that integrates EEG and ECG features; The electroencephalogram (EEG) and electrocardiogram (ECG) data of the person to be tested are input into the physiological and psychological state monitoring model integrating EEG and ECG characteristics to determine the physiological and psychological state of the person to be tested.

2. The method according to claim 1, characterized in that The step of determining a training data set according to the simulated data set and the real data set includes: Preprocessing the simulated data set to determine a preprocessed simulated data set; Performing sliding window segmentation processing on the preprocessed simulation data set to determine a first data set; Preprocessing the real data set to determine a preprocessed real data set; Performing scoring and annotation processing on the preprocessed real data set to determine the second data set; The second training data set is aligned with the first training data set to determine the training data set.

3. The method according to claim 2, characterized in that The preprocessing includes any one or more combinations of filtering, denoising, regularization, and data transformation.

4. The method according to claim 1, characterized in that The physiological and psychological state monitoring model integrating EEG and ECG features includes an EEG classifier, an ECG classifier, a dynamic multimodal fusion module and a multimodal feature classifier; The method of using the training data set to train a preset deep learning neural network to determine a physiological and psychological state monitoring model integrating EEG and ECG features includes: Converting the electroencephalogram data of the training data set from one-dimensional disordered spatial data to two-dimensional ordered spatial matrix data to determine the two-dimensional ordered spatial matrix data of the electroencephalogram data of the training data set; Inputting the two-dimensional ordered spatial matrix data of the electroencephalogram data of the training data set into the electroencephalogram classifier, and outputting the electroencephalogram features and the true class probability of the electroencephalogram modality; Inputting the electrocardiogram data in the training data set into the electrocardiogram classifier, and outputting the electrocardiogram features and the true class probability of the electrocardiogram modality; Input the EEG feature, the true class probability of the EEG modality, the ECG feature, and the true class probability of the ECG modality into the dynamic multimodal fusion module, and output the EEG-ECG multimodal feature; The EEG and ECG multimodal features are input into the multimodal feature classifier, and the alertness or arousal degree is output.

5. The method according to claim 4, characterized in that The step of inputting the EEG feature, the true class probability of the EEG modality, the ECG feature, and the true class probability of the ECG modality into the dynamic multimodal fusion module and outputting the EEG / ECG multimodal feature comprises: Assigning feature weights to the EEG features and the ECG features, and determining the EEG key feature information and the ECG key feature information; According to the true class probability of the EEG modality and the true class probability of the ECG modality, modality weights are assigned to the EEG key feature information and the ECG key feature information to determine the EEG-ECG multimodal features.

6. The method according to claim 1, characterized in that The step of inputting the electroencephalogram (EEG) and electrocardiogram (ECG) data of the person to be tested into the physiological and psychological state monitoring model integrating EEG and ECG characteristics to determine the physiological and psychological state of the person to be tested includes: Inputting the EEG and ECG data of the person to be tested into the physiological and psychological state monitoring model integrating EEG and ECG features, and outputting the alertness or arousal degree of the person to be tested; The physiological and psychological state of the person to be tested is determined according to the alertness or arousal level of the person to be tested.

7. The method according to claim 6, characterized in that The step of determining the physiological and psychological state of the person to be tested according to the alertness or arousal of the person to be tested includes: If the alertness or arousal of the person to be tested is a positive class, it is determined that the physiological and psychological state of the person to be tested is a tense state; If the alertness or arousal of the person to be tested is a negative class, it is determined that the physiological and psychological state of the person to be tested is a relaxed state.

8. A state monitoring system based on multimodal fusion features of EEG and ECG, characterized in that: include: A data acquisition module is used to obtain a simulated data set of the electroencephalogram and electrocardiogram of the person to be tested and a real data set of the electroencephalogram and electrocardiogram of the staff in the actual scene; A data processing module, used to determine a training data set according to the simulated data set and the real data set; A model training module, used to train a preset deep learning neural network using the training data set to determine a physiological and psychological state monitoring model that integrates EEG and ECG features; The state monitoring module is used to input the simulated data sets of the electroencephalogram and electrocardiogram of the person to be tested into the physiological and psychological state monitoring model that integrates the electroencephalogram and electrocardiogram characteristics to determine the physiological and psychological state of the person to be tested.

9. A state monitoring device based on multimodal fusion features of EEG and ECG, characterized in that: include: An EEG sensor module is used to collect EEG data of the person to be tested; An ECG sensor module is used to collect ECG data of the person to be tested; A data transmission module, used for transmitting the electroencephalogram data and the electrocardiogram data; The terminal is used to store and display the electroencephalogram data and the electrocardiogram data.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images