Personnel emotion recognition method and device based on EEG-fNIRS multi-mode signals, computer storage medium and electronic equipment

Abstract

The invention discloses a person emotion recognition method and device based on EEG-fNIRS multi-mode signals, a computer storage medium and electronic equipment, and the method comprises the steps: collecting physiological signals of a tested user, the physiological signals comprising an EEG signal and a functional near infrared spectral imaging fNIRS signal; processing the EEG signal and the fNIRS signal to obtain an EEG tensor and an fNIRS tensor; performing feature extraction on the EEG tensor to obtain a power spectral density feature, and performing feature extraction on the fNIRS tensor to obtain a local oxygenation level change feature; and based on a preset neural network model, performing feature extraction on the power spectral density feature and the local oxygenation level change feature to obtain a first spatial feature and a second spatial feature, and based on a fusion feature of the first spatial feature and the second spatial feature, identifying emotion classification of the tested user. Therefore, according to the method, the influence of low-spatial-resolution EEG and fNIRS response delay on emotion detection and recognition is solved through complementation, and the accuracy of emotion classification is improved.

Description

Technical Field

[0001] The present application relates to the technical field of emotion recognition, and in particular to a method for human emotion recognition based on EEG-fNIRS multimodal signals, a human emotion recognition device based on EEG-fNIRS multimodal signals, a computer-readable storage medium, and an electronic device. Background Art

[0002] Human factors engineering is a comprehensive and interdisciplinary subject. It uses a combination of physiology, psychology, anthropometry, biomechanics, computer science, system science and other multidisciplinary research methods and means to study the relationship and influence between people and other elements of the system, and ultimately make the system safe, efficient and pleasant. Human factors engineering provides a scientific and objective method for emotion detection by combining physiological signals with advanced data processing technologies such as brain-computer interfaces and deep learning.

[0003] In the related art, human factors engineering uses EEG (Electroencephalogram) signals to detect emotions. However, EEG signals are easily affected by noise, such as electromyographic interference and eye movement artifacts, and are difficult to meet the needs of accurate emotion classification. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first purpose of the present application is to propose a method for identifying human emotions based on EEG-fNIRS (Functional Near-Infrared Spectroscopy) multimodal signals, classify the emotions of the tested user based on EEG signals and fNIRS signals, and solve the influence of low spatial resolution EEG and fNIRS response delay on emotion detection and recognition through complementarity, thereby improving the accuracy of emotion classification.

[0005] The second objective of the present application is to propose a human emotion recognition device based on EEG-fNIRS multimodal signals.

[0006] The third object of the present application is to provide a computer-readable storage medium.

[0007] The fourth objective of the present application is to provide an electronic device.

[0008] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a method for identifying personal emotions based on EEG-fNIRS multimodal signals, the method comprising: collecting physiological signals of the user under test, the physiological signals comprising: electroencephalogram (EEG) signals and functional near-infrared spectral imaging (fNIRS) signals; processing the EEG signals and the fNIRS signals to obtain EEG tensors and fNIRS tensors; performing feature extraction on the EEG tensor to obtain power spectral density features, and performing feature extraction on the fNIRS tensor to obtain local oxygenation level change features; performing feature extraction on the power spectral density features and the local oxygenation level change features respectively based on a preset neural network model to obtain first spatial features and second spatial features, and identifying the emotion classification of the user under test based on the fusion features of the first spatial features and the second spatial features.

[0009] According to the personnel emotion recognition method based on EEG-fNIRS multimodal signals of the embodiment of the present application, first, the physiological signals of the user under test are collected, and the physiological signals include: electroencephalogram (EEG) signals and functional near-infrared spectral imaging (fNIRS) signals, and the EEG signals and fNIRS signals are processed to obtain EEG tensors and fNIRS tensors, and the EEG tensors are feature extracted to obtain power spectral density features, and the fNIRS tensors are feature extracted to obtain local oxygenation level change features, and then the power spectral density features and the local oxygenation level change features are feature extracted based on a preset neural network model to obtain first spatial features and second spatial features, and the emotion classification of the user under test is identified based on the fusion features of the first spatial features and the second spatial features. Thus, the method classifies the emotions of the user under test based on EEG signals and fNIRS signals, solves the influence of low spatial resolution EEG and fNIRS response delays on emotion detection and recognition through complementarity, and improves the accuracy of emotion classification.

[0010] In addition, the method for identifying human emotions based on EEG-fNIRS multimodal signals according to the above embodiment of the present application may also have the following additional technical features:

[0011] According to one embodiment of the present application, an EEG signal is processed to obtain an EEG tensor, including: performing bandpass filtering on the EEG signal to obtain an EEG filtered signal; and performing notch processing on the EEG filtered signal to obtain an EEG tensor.

[0012] According to one embodiment of the present application, the fNIRS tensor includes: changes in oxygenated hemoglobin concentration, changes in deoxygenated hemoglobin concentration, and total hemoglobin concentration. The fNIRS signal is processed to obtain the fNIRS tensor, including: performing optical density conversion on the fNIRS signal to obtain changes in oxygenated hemoglobin concentration, changes in deoxygenated hemoglobin concentration, and total hemoglobin concentration.

[0013] According to one embodiment of the present application, based on the fusion features of the first spatial features and the second spatial features, the emotion classification of the user under test is identified, including: identifying at least one target brain area based on the second spatial features, wherein the target brain area is used to represent the brain activity area related to a specific task; limiting different weight coefficients for the first spatial features corresponding to the target brain area and the remaining brain areas; outputting probability values ​​of different emotions according to the weight coefficients corresponding to the first spatial features based on a preset neural network model; and determining the emotion classification of the user under test based on the probability values ​​of different emotions.

[0014] According to an embodiment of the present application, determining an emotion classification of a measured user based on probability values ​​of different emotions includes: taking an emotion category corresponding to a maximum value among multiple probability values ​​as the emotion classification of the measured user.

[0015] According to one embodiment of the present application, the method for identifying human emotions based on EEG-fNIRS multimodal signals further includes: classifying the emotions of the tested user and outputting them in the form of graphics and / or numerical values.

[0016] According to one embodiment of the present application, the preset neural network model includes an EEG branch for obtaining a first spatial feature, an fNIRS branch for obtaining a second spatial feature, and a fusion branch for fusing features, and the fusion branch is used to output an emotion classification of the measured user.

[0017] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a personnel emotion recognition device based on EEG-fNIRS multimodal signals, the device comprising: an acquisition module, used to acquire physiological signals of the user under test, the physiological signals comprising: electroencephalogram (EEG) signals and functional near-infrared spectral imaging (fNIRS) signals; a processing module, used to process the EEG signals and the fNIRS signals to obtain EEG tensors and fNIRS tensors; an extraction module, used to perform feature extraction on the EEG tensor to obtain power spectral density features, and to perform feature extraction on the fNIRS tensor to obtain local oxygenation level change features; an identification module, used to perform feature extraction on the power spectral density features and the local oxygenation level change features respectively based on a preset neural network model to obtain first spatial features and second spatial features, and identify the emotion classification of the user under test based on the fusion features of the first spatial features and the second spatial features.

[0018] According to the personnel emotion recognition device based on EEG-fNIRS multimodal signals of the embodiment of the present application, the physiological signals of the measured user are collected by the acquisition module, and the physiological signals include: electroencephalogram (EEG) signals and functional near-infrared spectral imaging (fNIRS) signals. The EEG signals and fNIRS signals are processed by the processing module to obtain EEG tensors and fNIRS tensors. The EEG tensors are feature extracted by the extraction module to obtain power spectrum density features, and the fNIRS tensors are feature extracted to obtain local oxygenation level change features. The recognition module extracts the power spectrum density features and the local oxygenation level change features based on the preset neural network model to obtain the first spatial features and the second spatial features, and identifies the emotion classification of the measured user based on the fusion features of the first spatial features and the second spatial features. Thus, the device classifies the emotions of the measured user based on the EEG signals and the fNIRS signals, solves the influence of the low spatial resolution EEG and fNIRS response delays on emotion detection and recognition through complementarity, and improves the accuracy of emotion classification.

[0019] To achieve the above-mentioned objectives, the third aspect embodiment of the present application proposes a computer-readable storage medium, on which a personnel emotion recognition program based on EEG-fNIRS multimodal signals is stored. When the personnel emotion recognition program based on EEG-fNIRS multimodal signals is executed by a processor, the above-mentioned personnel emotion recognition method based on EEG-fNIRS multimodal signals is implemented.

[0020] According to the computer-readable storage medium of an embodiment of the present application, when the personnel emotion recognition program based on EEG-fNIRS multimodal signals is executed by a processor, the above-mentioned personnel emotion recognition method based on EEG-fNIRS multimodal signals is implemented. Based on the above-mentioned personnel emotion recognition method based on EEG-fNIRS multimodal signals, the emotions of the measured user are classified based on EEG signals and fNIRS signals, thereby improving the accuracy of emotion classification.

[0021] To achieve the above-mentioned objectives, the fourth aspect embodiment of the present application proposes an electronic device, comprising: a memory, a processor, and a personnel emotion recognition program based on EEG-fNIRS multimodal signals stored in the memory and executable on the processor. When the processor executes the personnel emotion recognition program based on EEG-fNIRS multimodal signals, the above-mentioned method of personnel emotion recognition based on EEG-fNIRS multimodal signals is implemented.

[0022] According to the electronic device of the embodiment of the present application, the processor executes the personnel emotion recognition program based on EEG-fNIRS multimodal signals to implement the above-mentioned personnel emotion recognition method based on EEG-fNIRS multimodal signals. Based on the above-mentioned personnel emotion recognition method based on EEG-fNIRS multimodal signals, the emotions of the measured user are classified based on EEG signals and fNIRS signals, thereby improving the accuracy of emotion classification.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Flow chart of a method for identifying human emotions based on EEG-fNIRS multimodal signals according to an embodiment of the present application;

[0025] Figure 2 Flow chart of a method for identifying human emotions based on EEG-fNIRS multimodal signals according to one embodiment of the present application;

[0026] Figure 3 Schematic diagram of the connection of a device for identifying human emotions based on EEG-fNIRS multimodal signals according to an embodiment of the present application;

[0027] Figure 4 Schematic diagram of a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] The following describes, with reference to the accompanying drawings, a method for identifying human emotions based on EEG-fNIRS multimodal signals, a device for identifying human emotions based on EEG-fNIRS multimodal signals, a computer-readable storage medium, and an electronic device proposed in an embodiment of the present application.

[0030] Figure 1 Flow chart of a method for human emotion recognition based on EEG-fNIRS multimodal signals according to an embodiment of the present application.

[0031] like Figure 1 As shown, the method for identifying human emotions based on EEG-fNIRS multimodal signals in the embodiment of the present application may include:

[0032] S1, collects physiological signals of the user under test, including: electroencephalogram (EEG) signals and functional near-infrared spectroscopy (fNIRS) signals;

[0033] S2, processing the EEG signal and the fNIRS signal to obtain an EEG tensor and an fNIRS tensor;

[0034] S3, feature extraction is performed on the EEG tensor to obtain power spectral density features, and feature extraction is performed on the fNIRS tensor to obtain local oxygenation level change features;

[0035] S4, based on the preset neural network model, feature extraction is performed on the power spectrum density feature and the local oxygenation level change feature to obtain the first spatial feature and the second spatial feature, and based on the fusion feature of the first spatial feature and the second spatial feature, the emotion classification of the measured user is identified.

[0036] Specifically, in the emotion induction task, the emotional response of the subject can be induced by external stimuli (such as videos, music, etc.), and then the EEG-fNIRS device is used to collect the corresponding physiological signals, including the EEG signal and fNIRS signal of the user being tested. Among them, the EEG signal is the electrical signal generated by the activity of brain neurons recorded by electrodes placed on the scalp, and the fNIRS signal is the signal measured by functional near-infrared spectroscopy technology, which reflects the changes in the blood oxygen level of the cerebral cortex.

[0037] The EEG signal is processed, such as filtering, re-referencing, downsampling, artifact removal and other processing operations, and then the processed EEG signal data is converted into a tensor form to obtain an EEG tensor. The EEG tensor is a characteristic with multiple dimensions, such as time, space (electrode position) and frequency, which can be represented by a tensor. For example, the EEG signal data consists of multiple channels and multiple sampling points. A two-dimensional tensor can be constructed, in which the first dimension is the channel and the second dimension is the sampling point. The EEG tensor is then feature extracted to obtain the power spectral density feature. Among them, the power spectral density is used to describe the statistic of the signal power changing with frequency, which represents the average power within a unit frequency. Specifically, the power distribution at different frequencies can be obtained by Fourier transforming the signal.

[0038] The fNIRS signal is processed, such as filtering, optical density conversion, etc., and then the processed fNIRS signal data is converted into an fNIRS tensor. The specific tensor conversion process can refer to the above-mentioned EEG tensor conversion process. Then, the fNIRS tensor is feature extracted to obtain the local oxygenation level change feature, where the local oxygenation level change feature refers to the change of the local oxygenation level of a tissue or organ under specific conditions.

[0039] The acquired power spectrum density and local oxygenation level change features are input into the preset neural network model, so as to classify the emotions of the tested user according to the power spectrum density and local oxygenation level change features through the preset neural network model. Specifically, the power spectrum density features are extracted by the convolutional neural network to obtain the first spatial features, and the local oxygenation level change features are extracted by the convolutional neural network to obtain the second spatial features. Then, the attention mechanism is introduced to fuse the first spatial features and the second spatial features, and the emotions such as happiness, sadness, anger, etc. are classified based on the fused features.

[0040] In this embodiment, the EEG signal and the fNIRS signal provide complementary information. The EEG signal has a high temporal resolution, while the fNIRS signal has a high spatial resolution. The combination of the two can more accurately reflect the brain's emotional cognitive process and improve the accuracy of emotion recognition.

[0041] In one embodiment of the present application, the EEG signal is processed to obtain an EEG tensor, including: performing bandpass filtering on the EEG signal to obtain an EEG filtered signal; and performing notch processing on the EEG filtered signal to obtain an EEG tensor.

[0042] For example, the EEG signal is first subjected to 1-90 Hz bandpass filtering, and then the filtered EEG signal is subjected to 50 Hz notch processing to perform noise reduction on the EEG signal, remove power supply interference, electromyography and other interference, and obtain an EEG tensor.

[0043] In one embodiment of the present application, the fNIRS tensor includes: changes in oxygenated hemoglobin concentration, changes in deoxygenated hemoglobin concentration, and total hemoglobin concentration. The fNIRS signal is processed to obtain the fNIRS tensor, including: performing optical density conversion on the fNIRS signal to obtain changes in oxygenated hemoglobin concentration, changes in deoxygenated hemoglobin concentration, and total hemoglobin concentration.

[0044] Specifically, the modified Lambert-Beer law is applied to perform optical density conversion on the fNIRS signal to obtain the concentration changes of oxygenated hemoglobin, deoxygenated hemoglobin and total hemoglobin concentration, thereby monitoring the neural activity of the brain.

[0045] In one embodiment of the present application, based on the fusion features of the first spatial features and the second spatial features, the emotion classification of the measured user is identified, including: identifying at least one target brain area according to the second spatial features, wherein the target brain area is used to represent the brain activity area related to a specific task; defining different weight coefficients for the first spatial features corresponding to the target brain area and the remaining brain areas; outputting probability values ​​of different emotions according to the weight coefficients corresponding to the first spatial features based on a preset neural network model; and determining the emotion classification of the measured user based on the probability values ​​of different emotions.

[0046] Specifically, in the feature fusion process, the attention mechanism is introduced, and the attention weight guided by fNIRS is used to emphasize the brain areas related to emotions in the EEG signal, that is, the more important brain areas are identified as target brain areas through the spatial information of the fNIRS signal, that is, the second spatial feature. Different weight coefficients are used for the EEG features of different brain areas (corresponding channels), that is, the first spatial features. Then, the probability values ​​of different emotions are calculated and output according to the weight coefficients corresponding to the first spatial features to achieve the effect of feature fusion, and the emotions of the tested users are classified according to the probability values ​​of different emotions.

[0047] In one embodiment of the present application, determining the emotion classification of the measured user based on the probability values ​​of different emotions includes: taking the emotion category corresponding to the maximum value among multiple probability values ​​as the emotion classification of the measured user.

[0048] That is to say, assuming that based on the preset neural network model, according to the weight coefficient output corresponding to the first spatial feature: the probability value of emotion A is 30%, and the probability value of emotion B is 70%, then it is determined that the emotion of the measured user is emotion B.

[0049] In one embodiment of the present application, the method for identifying human emotions based on EEG-fNIRS multimodal signals further includes: classifying the emotions of the measured user and outputting them in the form of graphics and / or numerical values.

[0050] That is, the emotion classification result is sent to the terminal interface, so as to be output to the user in the form of a graphic or a numerical value through the terminal interface, so that the user can intuitively view the emotion classification result.

[0051] In one embodiment of the present application, the preset neural network model includes an EEG branch for obtaining a first spatial feature, an fNIRS branch for obtaining a second spatial feature, and a fusion branch for fusing features, and the fusion branch is used to output an emotion classification of the measured user.

[0052] That is to say, in the EEG branch, a convolutional neural network is applied to extract spatial features of the EEG tensor to obtain the first spatial feature; in the fNIRS branch, a convolutional neural network is applied to extract spatial features of the fNIRS tensor to obtain the second spatial feature; in the fusion branch, an attention mechanism is introduced to fuse the first spatial feature and the second spatial feature, and the emotions of the tested users are classified based on the fused features, and the final emotion recognition results are output.

[0053] Furthermore, the preset neural network model is trained using labeled emotion data, and the model performance is optimized through cross-validation, hyperparameter tuning and other methods to improve the accuracy of the emotion classification model.

[0054] As a specific embodiment of the present application, Figure 2 As shown, the method for identifying human emotions based on EEG-fNIRS multimodal signals may include the following steps:

[0055] S101, collect EEG signals and fNIRS signals of the user under test, and execute steps S102 and S105 respectively.

[0056] S102, performing bandpass filtering on the EEG signal to obtain an EEG filtered signal.

[0057] S103, performing notch processing on the EEG filter signal to obtain an EEG tensor.

[0058] S104, extract features from the EEG tensor to obtain power spectrum density features. Execute step S107.

[0059] S105, performing optical density conversion on the fNIRS signal to obtain the change in oxygenated hemoglobin concentration, the change in deoxygenated hemoglobin concentration and the total hemoglobin concentration.

[0060] S106, extracting features of changes in oxygenated hemoglobin concentration, deoxygenated hemoglobin concentration, and total hemoglobin concentration to obtain features of changes in local oxygenation levels.

[0061] S107, based on a preset neural network model, feature extraction is performed on the power spectrum density feature and the local oxygenation level change feature respectively to obtain a first spatial feature and a second spatial feature.

[0062] S108, identifying at least one target brain region according to the second spatial feature, wherein the target brain region is used to represent a brain activity area related to a specific task.

[0063] S109, defining different weight coefficients for the first spatial features corresponding to the target brain region and the other brain regions.

[0064] S110, outputting probability values ​​of different emotions according to the weight coefficient corresponding to the first spatial feature based on a preset neural network model;

[0065] S111, taking the emotion category corresponding to the maximum value among the multiple probability values ​​as the emotion classification of the measured user.

[0066] S112, outputting the emotion classification of the tested user in the form of graphics and / or numerical values.

[0067] This embodiment achieves early signal fusion through deep learning, and uses the fNIRS-guided attention mechanism to emphasize the emotion-related brain areas in the EEG signal, thereby improving the accuracy and robustness of emotion recognition. In addition, the preset neural network model can be trained using labeled emotion data, and the model performance can be optimized through cross-validation, hyperparameter tuning and other methods, aiming to achieve efficient recognition of human emotional states through multimodal signal processing and advanced machine learning technology.

[0068] In the field of medical rehabilitation, the patient's emotional state is crucial to postoperative recovery and overall health. The human emotion recognition method based on EEG-fNIRS multimodal signals in this application can be applied in the following scenarios:

[0069] 1. Postoperative mood monitoring: This method is used to monitor the patient's mood changes during the postoperative recovery period. By analyzing the patient's emotional state in real time, the medical team can promptly identify anxiety, depression or other negative emotions and take appropriate intervention measures.

[0070] 2. Personalized rehabilitation plan: According to the patient's emotional state, a personalized rehabilitation plan can be formulated. For example, for patients with low mood, the frequency of psychological counseling and emotional support can be increased, while for patients with high anxiety levels, relaxation training and stress management techniques can be provided.

[0071] 3. Pain management: Emotional state is closely related to pain perception. By monitoring the patient's mood, the medical team can better manage the patient's pain treatment, because improved mood can reduce the perception of pain, thereby reducing dependence on pain medication.

[0072] 4. Mental health intervention: Long-term mood monitoring can help identify patients who need further mental health support. For those who show persistent negative emotions, more intensive psychological treatments such as cognitive behavioral therapy or medication can be provided.

[0073] 5. Clinical research: In clinical research, the system can be used to study the impact of emotional state on postoperative recovery, as well as the impact of different interventions on mood and recovery. This helps develop more effective rehabilitation strategies and treatment plans.

[0074] In summary, according to the method for identifying human emotions based on EEG-fNIRS multimodal signals of the embodiment of the present application, first, the physiological signals of the user under test are collected, and the physiological signals include: electroencephalogram (EEG) signals and functional near-infrared spectral imaging (fNIRS) signals, and the EEG signals and fNIRS signals are processed to obtain EEG tensors and fNIRS tensors, and the EEG tensors are feature extracted to obtain power spectral density features, and the fNIRS tensors are feature extracted to obtain local oxygenation level change features, and then the power spectral density features and the local oxygenation level change features are feature extracted based on the preset neural network model to obtain the first spatial features and the second spatial features, and the emotion classification of the user under test is identified based on the fusion features of the first spatial features and the second spatial features. Thus, the method classifies the emotions of the user under test based on EEG signals and fNIRS signals, solves the influence of low spatial resolution EEG and fNIRS response delays on emotion detection and recognition through complementarity, and improves the accuracy of emotion classification.

[0075] Corresponding to the above embodiments, the present application also proposes a human emotion recognition device based on EEG-fNIRS multimodal signals.

[0076] like Figure 3 As shown, the device for identifying human emotions based on EEG-fNIRS multimodal signals according to an embodiment of the present application may include: an acquisition module 10 , a processing module 20 , an extraction module 30 and an identification module 40 .

[0077] Among them, the acquisition module 10 is used to collect physiological signals of the user under test, and the physiological signals include: electroencephalogram (EEG) signals and functional near-infrared spectroscopy (fNIRS) signals. The processing module 20 is used to process the EEG signals and the fNIRS signals to obtain EEG tensors and fNIRS tensors. The extraction module 30 is used to perform feature extraction on the EEG tensor to obtain power spectral density features, and to perform feature extraction on the fNIRS tensor to obtain local oxygenation level change features. The recognition module 40 is used to perform feature extraction on the power spectral density features and the local oxygenation level change features based on a preset neural network model, respectively, to obtain the first spatial features and the second spatial features, and to identify the emotion classification of the user under test based on the fusion features of the first spatial features and the second spatial features.

[0078] According to an embodiment of the present application, the processing module 20 processes the EEG signal to obtain an EEG tensor, specifically for: performing bandpass filtering on the EEG signal to obtain an EEG filtered signal; performing notch processing on the EEG filtered signal to obtain an EEG tensor.

[0079] According to one embodiment of the present application, the fNIRS tensor includes: changes in oxygenated hemoglobin concentration, changes in deoxygenated hemoglobin concentration and total hemoglobin concentration. The processing module 20 processes the fNIRS signal to obtain the fNIRS tensor, which is specifically used to: perform optical density conversion on the fNIRS signal to obtain changes in oxygenated hemoglobin concentration, changes in deoxygenated hemoglobin concentration and total hemoglobin concentration.

[0080] According to one embodiment of the present application, the recognition module 40 identifies the emotion classification of the user under test based on the fusion characteristics of the first spatial feature and the second spatial feature, and is specifically used to: identify at least one target brain area based on the second spatial feature, wherein the target brain area is used to represent the brain activity area related to a specific task; define different weight coefficients for the first spatial features corresponding to the target brain area and the remaining brain areas; output probability values ​​of different emotions according to the weight coefficients corresponding to the first spatial features based on a preset neural network model; and determine the emotion classification of the user under test based on the probability values ​​of different emotions.

[0081] According to an embodiment of the present application, the recognition module 40 determines the emotion classification of the measured user based on the probability values ​​of different emotions, and is specifically configured to: use the emotion category corresponding to the maximum value among multiple probability values ​​as the emotion classification of the measured user.

[0082] According to an embodiment of the present application, the recognition module 40 is further used to: classify the emotion of the tested user and output it in the form of graphics and / or numerical values.

[0083] According to one embodiment of the present application, the preset neural network model includes an EEG branch for obtaining a first spatial feature, an fNIRS branch for obtaining a second spatial feature, and a fusion branch for fusing features, and the fusion branch is used to output an emotion classification of the measured user.

[0084] It should be noted that for details not disclosed in the device for identifying human emotions based on EEG-fNIRS multimodal signals in the embodiment of the present application, please refer to the details disclosed in the method for identifying human emotions based on EEG-fNIRS multimodal signals in the above embodiment of the present application, and the details will not be repeated here.

[0085] According to the personnel emotion recognition device based on EEG-fNIRS multimodal signals of the embodiment of the present application, the physiological signals of the measured user are collected by the acquisition module, and the physiological signals include: electroencephalogram (EEG) signals and functional near-infrared spectral imaging (fNIRS) signals. The EEG signals and fNIRS signals are processed by the processing module to obtain EEG tensors and fNIRS tensors. The EEG tensors are feature extracted by the extraction module to obtain power spectrum density features, and the fNIRS tensors are feature extracted to obtain local oxygenation level change features. The recognition module extracts the power spectrum density features and the local oxygenation level change features based on the preset neural network model to obtain the first spatial features and the second spatial features, and identifies the emotion classification of the measured user based on the fusion features of the first spatial features and the second spatial features. Thus, the device classifies the emotions of the measured user based on the EEG signals and the fNIRS signals, solves the influence of the low spatial resolution EEG and fNIRS response delays on emotion detection and recognition through complementarity, and improves the accuracy of emotion classification.

[0086] Corresponding to the above embodiments, the present application also proposes a computer-readable storage medium.

[0087] The computer-readable storage medium of an embodiment of the present application stores a personnel emotion recognition program based on EEG-fNIRS multimodal signals, and when the personnel emotion recognition program based on EEG-fNIRS multimodal signals is executed by a processor, the above-mentioned personnel emotion recognition method based on EEG-fNIRS multimodal signals is implemented.

[0088] According to the computer-readable storage medium of an embodiment of the present application, when the personnel emotion recognition program based on EEG-fNIRS multimodal signals is executed by a processor, the above-mentioned personnel emotion recognition method based on EEG-fNIRS multimodal signals is implemented. Based on the above-mentioned personnel emotion recognition method based on EEG-fNIRS multimodal signals, the emotions of the measured user are classified based on EEG signals and fNIRS signals, thereby improving the accuracy of emotion classification.

[0089] Corresponding to the above embodiment, the present application also proposes an electronic device.

[0090] like Figure 4 As shown, the electronic device 100 of the embodiment of the present application includes: a storage 110, a processor 120, and a personnel emotion recognition program based on EEG-fNIRS multimodal signals stored in the storage 110 and executable on the processor 120. When the processor 120 executes the personnel emotion recognition program based on EEG-fNIRS multimodal signals, the above-mentioned method of personnel emotion recognition based on EEG-fNIRS multimodal signals is implemented.

[0091] According to the electronic device of the embodiment of the present application, the processor executes the personnel emotion recognition program based on EEG-fNIRS multimodal signals to implement the above-mentioned personnel emotion recognition method based on EEG-fNIRS multimodal signals. Based on the above-mentioned personnel emotion recognition method based on EEG-fNIRS multimodal signals, the emotions of the measured user are classified based on EEG signals and fNIRS signals, thereby improving the accuracy of emotion classification.

[0092] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0093] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0094] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0095] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0096] In this application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0097] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for identifying human emotions based on EEG-fNIRS multimodal signals, characterized in that: The method comprises: Collecting physiological signals of the user under test, wherein the physiological signals include: electroencephalogram (EEG) signals and functional near-infrared spectroscopy (fNIRS) signals; Processing the EEG signal and the fNIRS signal to obtain an EEG tensor and an fNIRS tensor; performing feature extraction on the EEG tensor to obtain power spectral density features, and performing feature extraction on the fNIRS tensor to obtain local oxygenation level change features; Based on a preset neural network model, feature extraction is performed on the power spectrum density feature and the local oxygenation level change feature respectively to obtain a first spatial feature and a second spatial feature, and based on the fusion feature of the first spatial feature and the second spatial feature, the emotion classification of the measured user is identified.

2. The method for identifying human emotions based on EEG-fNIRS multimodal signals according to claim 1, characterized in that: The EEG signal is processed to obtain an EEG tensor, including: Performing bandpass filtering on the EEG signal to obtain an EEG filtered signal; The EEG filter signal is notched to obtain the EEG tensor.

3. The method for identifying human emotions based on EEG-fNIRS multimodal signals according to claim 1, characterized in that: The fNIRS tensor includes: oxygenated hemoglobin concentration change, deoxygenated hemoglobin concentration change and total hemoglobin concentration. The fNIRS signal is processed to obtain the fNIRS tensor, including: The fNIRS signal is subjected to optical density conversion to obtain the change in the oxygenated hemoglobin concentration, the change in the deoxygenated hemoglobin concentration, and the total hemoglobin concentration.

4. The method for identifying human emotions based on EEG-fNIRS multimodal signals according to claim 1, characterized in that: Identifying the emotion classification of the measured user based on the fusion feature of the first spatial feature and the second spatial feature includes: identifying at least one target brain region according to the second spatial feature, wherein the target brain region is used to represent a brain activity area related to a specific task; defining different weight coefficients for the first spatial features corresponding to the target brain region and other brain regions; Outputting probability values ​​of different emotions according to the weight coefficients corresponding to the first spatial features based on the preset neural network model; The emotion classification of the measured user is determined based on the probability values ​​of different emotions.

5. The method for identifying human emotions based on EEG-fNIRS multimodal signals according to claim 4, characterized in that: Determining the emotion classification of the measured user based on the probability values ​​of different emotions includes: The emotion category corresponding to the maximum value among the multiple probability values ​​is used as the emotion classification of the measured user.

6. The method for identifying human emotions based on EEG-fNIRS multimodal signals according to claim 4, characterized in that: The method further comprises: The emotion classification of the measured user is output in the form of a graph and / or a numerical value.

7. The method for identifying human emotions based on EEG-fNIRS multimodal signals according to claim 1, characterized in that: The preset neural network model includes an EEG branch for acquiring the first spatial feature, an fNIRS branch for acquiring the second spatial feature, and a fusion branch for fusing features, wherein the fusion branch is used to output the emotion classification of the measured user.

8. A device for identifying human emotions based on EEG-fNIRS multimodal signals, characterized in that: The device comprises: An acquisition module is used to acquire physiological signals of the user under test, wherein the physiological signals include: electroencephalogram (EEG) signals and functional near-infrared spectroscopy (fNIRS) signals; A processing module, used for processing the EEG signal and the fNIRS signal to obtain an EEG tensor and an fNIRS tensor; An extraction module, used for performing feature extraction on the EEG tensor to obtain a power spectral density feature, and performing feature extraction on the fNIRS tensor to obtain a local oxygenation level change feature; The recognition module is used to extract the power spectrum density feature and the local oxygenation level change feature based on a preset neural network model to obtain a first spatial feature and a second spatial feature, and identify the emotion classification of the measured user based on the fusion feature of the first spatial feature and the second spatial feature.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the method for human emotion recognition based on EEG-fNIRS multimodal signals according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: A memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for human emotion recognition based on EEG-fNIRS multimodal signals according to any one of claims 1 to 7 is implemented.

Images