Vision-electroencephalogram-physiology multi-mode signal acquisition and real-time emotion evaluation method and system

Abstract

The invention discloses a vision-electroencephalogram-physiology multi-mode signal acquisition and real-time emotion evaluation method and system. The method comprises the following steps: (1) designing an experimental normal form for stimulating emotions based on picture and video materials; (2) in the experiment process, synchronously receiving expression modal, eye movement modal, physiological signal modal and electroencephalogram modal data through sensor equipment; (3) extracting corresponding secondary features from each piece of modal data; (4) performing feature selection and dimension reduction on the secondary features, and training two classifiers in each mode; results output by all classifiers and confidence coefficients are spliced to serve as new features, and a final multi-modal classification model is trained to complete integrated learning of a decision-making layer set; and (5) in an application process, inputting each modal signal into the multi-modal classification model to obtain an emotion classification result. According to the method, expression, eye movement, electroencephalogram and physiological multi-modal signals are accurately and synchronously collected, and multi-modal information is effectively utilized, so that the classification accuracy of emotion evaluation is improved.

Description

Technical Field

[0001] The present invention relates to the field of brain-electroencephalogram (EEG) interfaces, and in particular to a method and system for collecting visual-electroencephalogram (EEG)-physiological multimodal signals and for real-time emotion assessment. Background Art

[0002] The recognition and prediction of emotions are of great significance in the fields of human-computer interaction, brain-computer interface, and clinical diagnosis and treatment of brain-related diseases. The analysis of changes in visual, electroencephalographic, physiological and other signals caused by psychological changes of experimental subjects under certain stimulation conditions is the basis for predicting or even intervening in the emotions of subjects. Different modal signals contribute to describing human emotions as a side, which improves the accuracy of emotion recognition compared to single modality. Therefore, it is necessary to effectively utilize multi-modal signals.

[0003] For example, a Chinese patent document with publication number CN112535479A discloses a method for determining an emotion processing tendency and related products, comprising: presenting a visual stimulus, wherein the visual stimulus is a stimulus that includes both a positive emotion stimulus and a negative emotion stimulus; obtaining the gaze point of the eyes of a test target within a first time threshold after receiving the visual stimulus, and the gaze duration of the gaze point; and determining the emotion processing tendency of the test target based on the gaze point and the gaze duration.

[0004] A Chinese patent document with publication number CN109640811A discloses a device for evaluating the psychophysiological reactivity of an object, comprising: a stimulation unit for providing electrical stimulation to the skin of the object via a first electrode and a second electrode; a sensing unit for collecting an impedance signal indicating the impedance between the first electrode and the second electrode in response to the electrical stimulation; and an analysis unit suitable for identifying a double electric layer based on the impedance signal collected in response to the electrical stimulation and determining the psychophysiological reactivity of the object based on the identified double electric layer.

[0005] However, existing technologies lack a system that can accurately and synchronously capture various modalities while stimulating the emotions of experimental subjects. Summary of the invention

[0006] The present invention provides a visual-electroencephalogram-physiological multimodal signal acquisition and real-time emotion assessment method and system, which accurately and synchronously acquires facial expressions, eye movements, electroencephalograms, and physiological multimodal signals, effectively utilizes multimodal information, and thus improves the classification accuracy of emotion assessment.

[0007] A method for collecting visual-electroencephalogram-physiological multimodal signals and real-time emotion assessment comprises the following steps:

[0008] (1) Design an experimental paradigm based on pictures and videos to stimulate emotions;

[0009] (2) During the experiment, the facial expression modality, eye movement modality, physiological signal modality, and EEG modality data are received through sensor equipment, and all data signals are synchronized;

[0010] (3) Extract corresponding secondary features from each modal data;

[0011] (4) Perform feature selection and dimensionality reduction on the secondary features. Train two classifiers for each modality and output classification results and confidence probabilities. Combine the results and confidence levels of all classifiers as new features and train a final multimodal classification model to complete the integrated learning of the decision layer set.

[0012] (5) During the application process, each modal signal is input into the multimodal classification model to obtain the emotion classification result.

[0013] The experimental paradigm of step (1) is as follows:

[0014] At the beginning of the first experiment, a mental health assessment questionnaire was displayed on the screen to assess the subjects' depression and anxiety levels;

[0015] Each experiment shows 20 to 25 groups of pictures or 8 to 12 videos, which form a large group; three large groups of pictures-videos-pictures or videos-pictures-videos alternately constitute a complete experiment;

[0016] In the experimental stage, a group of 5 pictures, each lasting 3 to 4 seconds, or a 2 to 3 minute video were shown, and the subjects were asked to drag and score 0-100 on the two visual analogue scales of valence and arousal. If it was picture material, after scoring, the behavioral test was started, showing a picture and asking the subjects to choose whether it was among the 5 pictures in the group, in order to monitor whether they looked at the picture carefully. If it was video material, after scoring, the subjects were asked to choose one of the 9 emotions to feel when watching the video. Before the next group of materials were shown, a 10-second countdown was displayed in the center of the screen, during which the subjects rested and adjusted their emotions.

[0017] Every time the screen changes or the user clicks a button, the event time is obtained in the code immediately, and then an entry including the event type, event information, and event time point is generated to record the event, and finally exported as an experiment record table.

[0018] Compared with the existing experimental paradigm, this paradigm uses a group of 5 pictures of the same type as a scoring point to strengthen the emotional stimulation effect and make up for the shortcoming that the picture stimulation is not strong enough; it uses continuous valence-arousal for scoring, which can measure the subjects' emotions more accurately and scientifically than several discrete emotion scoring methods; it uses picture scoring consistency tests and behavioral tests to ensure that the subjects are serious in the experiment and the data is scientific and valid.

[0019] In step (2), when synchronizing all data signals, various methods such as timestamp, UDP communication, and serial port marking are used to achieve accurate synchronization between the data collected by different sensors in various modes and experimental records, solving the synchronization problem of these data.

[0020] In step (3), before extracting secondary features from the facial expression modality and eye movement modality data, the following processing is performed:

[0021] For expression videos, extract the xyz three-dimensional space coordinates of the head and the values ​​of each frame of each facial action unit as the expression data after preliminary processing; for each facial action unit, record two AU values, which are a continuous value of 0-1 and a discrete Boolean value of True / False;

[0022] For the eye movement data, the pupil sizes of the left and right eyes, the screen coordinates of the gaze point, and the spatial coordinates of the pupil positions of both eyes were recorded respectively.

[0023] The corresponding secondary features are extracted from the facial expression modality, eye movement modality, and physiological signal modality data, specifically:

[0024] First, three columns are added next to each first-level feature: rate of change, acceleration, and jerk; then, for each dimension, the covariance between all these features is calculated, as well as the time domain features within the duration of each data point; the covariance and time domain features constitute the generated second-level features.

[0025] For EEG modality data, the Band-wise Differential Entropy of the Gamma frequency band is calculated as the secondary feature.

[0026] In step (4), feature selection and dimensionality reduction are performed on the secondary features. The specific process is as follows:

[0027] First, PCA principal component analysis is performed to generate PCA1-PCA10 ten-dimensional PCA principal components for eye movement and expression modalities; and 3-dimensional PCA principal components are generated for physiological signal modalities; these generated PCA dimensions are spliced ​​after the original secondary features;

[0028] Based on this, statistical analysis is performed to calculate the mutual information value, and the dimensions with mutual information values ​​lower than the set threshold are discarded, so that the feature dimension is reduced to 100-200 dimensions; for these selected main features, the PCA1-PCA3 three-dimensional PCA principal components are calculated and placed in parallel with these main features to achieve a filtering effect.

[0029] Preferably, in step (4), two classifiers are trained for each modality, namely, RF random forest and SVM support vector machine.

[0030] A visual-electroencephalogram-physiological multimodal signal acquisition and real-time emotion assessment system comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned visual-electroencephalogram-physiological multimodal signal acquisition and real-time emotion assessment method.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention solves the problem in the prior art that there is a lack of the ability to accurately and synchronously collect facial expressions, eye movements, electroencephalograms, and physiological multimodal signals while conducting real-time emotional assessment of subjects. At the same time, an integrated learning architecture that integrates multimodality at the decision-making layer is proposed. Experiments have shown that the architecture effectively utilizes multimodal information and effectively improves classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of a method for visual-electroencephalography-physiological multimodal signal acquisition and real-time emotion assessment of the present invention;

[0034] Figure 2 A flowchart of the experimental paradigm for stimulating emotions based on pictures and video materials;

[0035] Figure 3 Schematic diagram of each sensor device. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0037] like Figure 1 As shown, a visual-electroencephalogram-physiological multimodal signal acquisition and real-time emotion assessment method includes the following steps:

[0038] S1. Real-time emotion assessment experimental paradigm design

[0039] S11. Design an experimental paradigm based on pictures and video materials to stimulate emotions. The stimulus materials are widely distributed on the two-dimensional plane of valence-arousal. Five pictures or about 2 minutes of video are used as a data point.

[0040] S12. Use Matlab Psychtoolbox to design the experimental task interface, including questionnaire interface, stimulation interface, scoring interface, classification interface, behavioral test interface, and rest interface.

[0041] S13, such as Figure 2As shown in the figure, before the experiment, a 5-point eye movement calibration was displayed on the screen. At the beginning of the first experiment, questionnaires such as PHQ-9, GAD-7, and HAMD-6 were displayed on the screen to assess the subjects' depression and anxiety levels. At the beginning of the first experiment, a tutorial was displayed on the screen and the subjects were asked to conduct a set of teaching experiments to fully understand the scoring rules.

[0042] S14. Each experiment shows 20 groups of pictures or about 10 videos, which form a large group. Three large groups of pictures-video-picture or video-picture-video alternately constitute a complete experiment. In the experimental stage, a group of 5 pictures, each for 3 to 4 seconds, or a video of about 2 minutes are shown, and then the subjects are asked to drag and score 0-100 on the two visual analog score bars of valence and arousal. If it is picture material, after scoring, enter the behavioral test, show a picture and let the subject choose whether it is among the 5 pictures in this group to monitor whether he looks at the picture carefully. If it is video material, after scoring, let the subject choose one of the 9 emotions to feel when watching the video. Before the next group of materials is shown, a 10-second countdown is displayed in the center of the screen. During this period, the subjects rest and adjust their emotions.

[0043] S15. At the beginning of the second experiment, three groups of pictures that the subjects had seen in a previous experiment were randomly inserted into the picture group above the 20 groups to evaluate the consistency of their scoring.

[0044] S16. Every time the screen changes or the user clicks a button, the event time is obtained in the code at the first time, and then an entry including the event type, event information, and event time point is generated to record the event, and finally exported as an experiment record table.

[0045] S2. Experimental signal synchronization method

[0046] S21, such as Figure 3 As shown, expression mode: the high-definition camera above the screen records 30 frames of facial video per second. The USB is connected to the experimental equipment and the local loopback UDP signal is controlled.

[0047] The camera connected to USB is controlled by Python OpenCV library, and an independent Python script is run to monitor UDP signals. At the beginning of each experiment, the event is recorded and a UDP signal is sent to the local machine to notify the script to start video recording. At the end of each experiment, a UDP signal is also sent to notify the script to end recording and save the video. In this way, the time point when the video starts is the timestamp of the experiment start event.

[0048] S22, Eye Movement Modality: The eye tracker below the screen records eye movement data at 120 frames per second. The network cable is connected and the matlab sdk sends signals for control.

[0049] Use the Tobii eye tracker and its MATLAB SDK to calibrate and record in MATLAB. The eye tracker is connected to the experimental computer via an Ethernet cable. The recorded data is transferred to CSV format, and each data is timestamped by the Get System Timestamp function provided by the SDK. The same timestamp function is used to record the time of each event entry recorded in the experimental data table. In this way, the data exported by the eye tracker and the data in the table can accurately correspond.

[0050] S23, EEG modality and ECG modality: Record intracranial SEEG and extracranial EEG signals through electrodes implanted in the skull and placed on the scalp surface. The USB serial port to parallel port marker is connected to the hospital EEG system, and MATLAB sends serial communication to mark.

[0051] There is a DC-IN channel in the EEG system, which receives the electrical signal input from the parallel port for marking. The present invention connects a USB to parallel port device through USB serial port communication and accesses the EEG system. When each event occurs and a table entry is generated, a marking signal is output through the serial port. When the EEG signal is extracted and processed later, the time of the EEG signal and the experimental record table is manually aligned according to the shape of the marking signal of the marking channel. After that, the EEG data required for the experiment can be intercepted. The ECG system and the EEG system belong to the same hospital optoelectronic system, and the segmentation and extraction operations are the same.

[0052] S24, GSR skin electrical modality: Use Bluetooth to operate the GSR recorder, and record the skin resistance signal GSR through the skin electrical sensors worn on the index and middle fingers. At the beginning of the experiment, calibrate the built-in time of the recorder and synchronize it with the timestamp of the experimental computer system. After the experiment, collect the recorder and intercept the skin electrical data according to the timestamp of the experimental record form.

[0053] S3. Extract corresponding secondary features for each modal feature.

[0054] S31, extracting the xyz three-dimensional space coordinates of the head and the values ​​of each frame of each facial action unit Action Units from the expression video as the expression data after preliminary processing. For each facial action unit, two AU values ​​are recorded, which are a continuous value of 0-1 and a discrete Boolean value of True / False.

[0055] S32. For the eye movement data, the pupil size, the screen coordinates of the gaze point and the spatial coordinates of the pupil positions of both eyes are recorded for the left and right eyes respectively.

[0056] S33. Generate secondary features for the three modalities of facial data, eye movement data and skin electricity data. Generation method: First, add three columns next to each primary feature: rate of change, acceleration, and jerkiness. Then, for each dimension, calculate the covariance between all these features, as well as the time domain features within the duration of each data point. Time domain features include: 10, 20, 30, 50, 70, 90, 95 statistical quantiles, kurt, skew, etc. Covariance and time domain features constitute the generated secondary features.

[0057] S34. For EEG features, calculate the Band-wise Differential Entropy of the Gamma frequency band as the secondary feature.

[0058] S4. Decode and classify multimodal features

[0059] S41. For the secondary features generated in S33, feature selection and dimensionality reduction are performed before training. The method is as follows: First, PCA principal component analysis is performed. For eye movement and expression modalities, PCA1-PCA10 ten-dimensional PCA principal components are generated, and for physiological signal modalities, three-dimensional PCA principal components are generated. These generated PCA dimensions are spliced ​​after the original secondary features. Statistical analysis is performed on this, and the mutual-information value is calculated. Dimensions with mutual information values ​​below a certain threshold are discarded, so that the feature dimension is reduced to about 100 dimensions. For these selected main features, PCA1-PCA3 three-dimensional PCA principal components are calculated and placed in parallel with them to achieve a filtering effect.

[0060] S42. Divide the aligned data into a training set and a test set according to ten-fold cross validation, and classify the data.

[0061] S43. Two classifiers are trained for each modality: RF random forest and SVM support vector machine, and the classification results and confidence probabilities are output.

[0062] S44. Concatenate the output results and confidence levels of all classifiers as new features, and train a final classifier to complete the integrated learning of the decision layer set.

[0063] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A visual-electroencephalogram-physiological multimodal signal acquisition and real-time emotion assessment method, characterized in that: The following steps are involved: (1) Design an experimental paradigm based on pictures and videos to stimulate emotions; (2) During the experiment, the facial expression modality, eye movement modality, physiological signal modality, and EEG modality data are received through sensor equipment, and all data signals are synchronized; (3) Extract corresponding secondary features from each modal data; (4) Perform feature selection and dimensionality reduction on the secondary features. Train two classifiers for each modality and output classification results and confidence probabilities. Combine the results and confidence levels of all classifiers as new features and train a final multimodal classification model to complete the integrated learning of the decision layer set. (5) During the application process, each modal signal is input into the multimodal classification model to obtain the emotion classification result.

2. The method for visual-electroencephalography-physiological multimodal signal acquisition and real-time emotion assessment according to claim 1, characterized in that: The experimental paradigm of step (1) is as follows: At the beginning of the first experiment, a mental health assessment questionnaire was displayed on the screen to assess the subjects' depression and anxiety levels; Each experiment shows 20 to 25 groups of pictures or 8 to 12 videos, which form a large group; three large groups of pictures-videos-pictures or videos-pictures-videos are alternated to form a complete experiment; In the experimental stage, a group of 5 pictures, each lasting 3 to 4 seconds, or a 2 to 3 minute video were shown, and the subjects were asked to drag and score 0-100 on the two visual analogue scales of valence and arousal. If it was picture material, after scoring, the behavioral test was started, showing a picture and asking the subjects to choose whether it was among the 5 pictures in the group, in order to monitor whether they looked at the picture carefully. If it was video material, after scoring, the subjects were asked to choose one of the 9 emotions to feel when watching the video. Before the next group of materials were shown, a 10-second countdown was displayed in the center of the screen, during which the subjects rested and adjusted their emotions. Every time the screen changes or the user clicks a button, the event time is obtained in the code immediately, and then an entry including the event type, event information, and event time point is generated to record the event, and finally exported as an experiment record table.

3. The method for visual-electroencephalography-physiological multimodal signal acquisition and real-time emotion assessment according to claim 1, characterized in that: In step (3), before extracting secondary features from the facial expression modality and eye movement modality data, the following processing is performed: For expression videos, extract the xyz three-dimensional space coordinates of the head and the values ​​of each frame of each facial action unit as the expression data after preliminary processing; for each facial action unit, record two AU values, which are a continuous value of 0-1 and a discrete Boolean value of True / False; For the eye movement data, the pupil sizes of the left and right eyes, the screen coordinates of the gaze point, and the spatial coordinates of the pupil positions of both eyes were recorded respectively.

4. The method for visual-electroencephalography-physiological multimodal signal acquisition and real-time emotion assessment according to claim 3, characterized in that: In step (3), the corresponding secondary features are extracted from the facial expression modality, eye movement modality, and physiological signal modality data, specifically: First, three columns are added next to each first-level feature: rate of change, acceleration, and jerk; then, for each dimension, the covariance between all these features is calculated, as well as the time domain features within the duration of each data point; the covariance and time domain features constitute the generated second-level features.

5. The method for visual-electroencephalography-physiological multimodal signal acquisition and real-time emotion assessment according to claim 3, characterized in that: In step (3), for the EEG modal data, the Band-wise DifferentialEntropy of the Gamma frequency band is calculated as the secondary feature.

6. The method for visual-electroencephalography-physiological multimodal signal acquisition and real-time emotion assessment according to claim 1, characterized in that: In step (4), feature selection and dimensionality reduction are performed on the secondary features. The specific process is as follows: First, PCA principal component analysis is performed to generate PCA1-PCA10 ten-dimensional PCA principal components for eye movement and expression modalities; and 3-dimensional PCA principal components are generated for physiological signal modalities; these generated PCA dimensions are spliced ​​after the original secondary features; Based on this, statistical analysis is performed to calculate the mutual information value, and the dimensions with mutual information values ​​lower than the set threshold are discarded, so that the feature dimension is reduced to 100-200 dimensions; for these selected main features, the PCA1-PCA3 three-dimensional PCA principal components are calculated and selected in parallel with these main features to achieve a filtering effect.

7. The method for visual-electroencephalography-physiological multimodal signal acquisition and real-time emotion assessment according to claim 1, characterized in that: In step (4), two classifiers are trained for each modality, namely RF random forest and SVM support vector machine.

8. A visual-electroencephalogram-physiological multimodal signal acquisition and real-time emotion assessment system, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the visual-electroencephalogram-physiological multimodal signal acquisition and real-time emotion assessment method described in any one of claims 1 to 7.

Images