Multi-modal emotion recognition evaluation method based on multi-feature two-dimensional convolution

Abstract

The invention discloses a multi-modal emotion recognition and evaluation method based on multi-feature two-dimensional convolution, and aims to solve the problem that the efficiency and accuracy of multi-classification are difficult to stably guarantee due to lack of a systematic and accurate classification method for brain emotions at present. The multi-modal emotion recognition and evaluation method comprises the following steps: performing elimination and denoising preprocessing on electroencephalogram signals obtained based on an audio stimulation emotion test to form pure electroencephalogram information, and performing feature extraction on the pure electroencephalogram information to construct a multi-modal data set; classifying, identifying and evaluating the multi-modal data set in a convolutional neural network mode; a network feature sequence of each node is obtained by calculating the node degree and the local efficiency of a network, the network feature sequences are combined with continuous wavelets, so that the process of constructing a multi-modal data set is completed, and a convolutional network model is obtained by constructing the multi-modal data set.

Description

Technical Field

[0001] The present invention specifically relates to a multi-modal emotion recognition and evaluation method based on multi-feature two-dimensional convolution. Background Art

[0002] Facial emotion recognition and expression generation have important application value in the fields of computer vision and artificial intelligence. Facial emotion recognition and expression generation technology based on convolutional neural networks has made significant progress and has been widely used in many fields.

[0003] Facial emotion recognition refers to accurately identifying a person's emotional state, such as happiness, sadness, anger, etc., by analyzing facial images or videos. Expression generation refers to generating corresponding expression images or animations based on given emotion labels or descriptions. The combination of these two technologies can play an important role in virtual character design, human-computer interaction, autonomous driving and other fields.

[0004] The principle of the convolution method is to use the unit impulse response h(t) of the circuit to solve the system response based on the properties of linear steady-state circuits (homogeneity, superposition, time invariance, integrality, etc.). The excitation of the system can generally be expressed as the convolution of the impulse function and the excitation function. The amplitude of the impulse function is determined by the area of ​​each rectangular element. In general, convolution is to express the excitation function with the impulse function, and then solve the zero-state response of the system based on the impulse response. Convolution is essentially filtering the signal.

[0005] Facial emotion recognition based on Convolutional Neural Network (CNN) has achieved good results. CNN is a neural network model specifically used for image processing. By stacking multiple convolutional layers, pooling layers, and fully connected layers, it can extract high-level features from images. In facial emotion recognition, we can use the CNN model to classify facial images and divide them into different emotion categories.

[0006] We need to build a dataset suitable for facial emotion recognition. This dataset should contain multiple people's face images, and each image should have a corresponding emotion label. We can obtain this data by manually annotating or searching for publicly available datasets on the Internet. Then, we can divide these images into training sets, validation sets, and test sets according to a certain ratio for model training and evaluation.

[0007] Next, we can use the CNN-based facial emotion recognition algorithm to train the training set. During the training process, we will use the back propagation algorithm to update the parameters in the network so that the model can gradually learn the emotional characteristics in the image. During the training process, we can use some optimization techniques, such as learning rate adjustment, regularization, etc., to improve the performance and generalization ability of the model.

[0008] After the model training is completed, we can use the validation set to evaluate the performance of the model and perform tuning. By adjusting the network structure, hyperparameters, etc., we can further improve the accuracy of the model on the validation set. Once we have a satisfactory model, we can use the test set to evaluate the performance of the model on unseen data.

[0009] In addition to facial emotion recognition, CNN-based technology can also be used for expression generation. Expression generation can generate facial images corresponding to the emotion by taking the emotion label as input. This generative model is usually implemented using models such as Generative Adversarial Network (GAN) or Variational Autoencoder (VAE). These models can learn the distribution characteristics of facial images based on training data and can generate new images with similar emotions.

[0010] In expression generation, we can use the trained generative model to generate the corresponding expression image through random noise vector according to the given emotion label or description. In order to improve the quality and diversity of generated images, we can use some techniques, such as conditional generation, inverse spatiotemporal convolution, etc. At the same time, we can also use some evaluation indicators, such as diversity measurement, authenticity evaluation, etc. to measure the quality of generated images.

[0011] Facial emotion recognition and expression generation technology based on convolutional neural networks has broad application prospects in the fields of computer vision and artificial intelligence. By training deep learning models, we can effectively identify emotions in facial images and generate new images with corresponding emotions. This technology is expected to achieve breakthroughs and innovations in the fields of virtual character design, human-computer interaction, etc. However, further research and practice are still needed to improve the performance and reliability of the algorithm to meet the needs of practical applications.

[0012] With the rapid development of wearable computing devices and the demand for immersive human-computer interaction environments, emotion recognition has become a global research hotspot, but multimodal recognition methods based on EEG signals need to be expanded. Currently, the commonly used deep learning methods in the field of EEG emotion recognition include convolutional neural networks (CNN), recurrent neural networks (RNN), deep belief networks (DBN) or other neural networks. Deep neural networks can perform end-to-end automatic learning of EEG signal preprocessing, feature map extraction and classification. EEG emotion recognition based on deep learning algorithms has higher feature dimensions than manually designed ones and takes less time to recognize.

[0013] However, there is currently no detailed and unified research process on the accuracy of multimodal deep learning recognition and classification of emotional EEG, and there is a lack of accurate classification of brain emotions. The efficiency and accuracy of multi-classification need to be improved. Summary of the invention

[0014] The present invention provides a multi-modal emotion recognition and evaluation method based on multi-feature two-dimensional convolution to solve the above problems.

[0015] The present invention provides the following technical solutions:

[0016] A multimodal emotion recognition and evaluation method based on multi-feature two-dimensional convolution, characterized in that: the multimodal emotion recognition and evaluation method is to perform a pre-processing of removing noise from the electroencephalogram (EEG) signal obtained based on an audio stimulation emotion test to form pure EEG information, then perform feature extraction on the pure EEG information to construct a multimodal data set, and classify, recognize and evaluate the multimodal data set through a convolutional neural network.

[0017] As a preferred solution: the process of obtaining EEG signals in the audio stimulation emotion test is:

[0018] Test preparation: A predetermined number of people who meet the relevant requirements are selected as test subjects. The test subjects are placed in a test environment where the room temperature is controlled between 25 and 26 degrees Celsius and the indoor noise is less than 30db. The EEG signals of the test subjects under audio stimulation are collected after the head has undergone impedance reduction operation.

[0019] Preparation for EEG signal collection: determine the number of channels in the scalp area, divide the degree of each emotion into 0 to 10 points, record the emotions generated by each subject after a test and record the corresponding emotion score when the emotion is generated, and use non-invasive wet electrode connection for each person to collect EEG signals through the international standard 10-20 system electrode connection method;

[0020] Audio stimulation process: ensure that each subject completes at least five groups of tests, each group of tests includes at least six trials, forming M test processes. During each trial, ensure that the subject starts brain wave collection with eyes closed at 0 seconds, turn on the audio at 10 seconds, ensure that the audio continues to play for 70 seconds, then turn off the audio and end the brain wave collection, ensuring that each trial contains a time sequence of no less than 60 seconds. After turning off the audio, the subject is required to verbally report the subjective score and complete a rest within a fixed time to complete a trial. According to the above process, the next trial is completed by analogy;

[0021] After the audio stimulation process, primary EEG signals were obtained.

[0022] As a preferred solution, the process of preprocessing the EEG signals obtained from the audio stimulation emotion test is as follows:

[0023] Step 1: Import the collected EDF data file into the EEG data preprocessing software eeglab, perform electrode positioning according to the international standard 10-20 system electrode connection method, determine the number of channels N, time duration S, sampling rate SR, and acquisition information data of sampling points SP, and select the time series EEG waves stimulated by audio in the primary EEG signals obtained in each group of experiments, that is, the primary EEG signals of the last 60 seconds of the total length of 70 seconds for data preprocessing;

[0024] Step 2: Sort the five frequency bands of the human brain state corresponding to different EEG signal frequency bands, and filter out other clutter in the frequency domain through a digital filter. Clutter filtering includes filtering out the primary EEG signals obtained from each group of experiments by low-pass filtering with a frequency of 30Hz to 100Hz, band-stop filtering with a frequency of 48 to 52Hz, and high-pass filtering with a frequency of 0.1Hz to 1Hz to form a first-level purified EEG data;

[0025] Step 3: The first-level purified EEG data is sampled to form the second-level purified EEG data according to the Nyquist sampling theorem under the limitation of acquisition conditions: According to the Nyquist sampling theorem, the first-level purified EEG data is sampled to form the second-level purified EEG data. The sampling conditions are that the sampling rate is more than 4 times higher than the frequency band of interest and less than 6 times the frequency band of interest, and the sampling rate is between 400 and 600 Hz.

[0026] Step 4: Perform ICA to remove artifacts: Decompose the secondary purified EEG data through ICA and remove the relevant frequency bands marked as artifacts in the secondary purified EEG data. After removal, reassemble the remaining EEG data to form the third-level purified EEG data, which is the real EEG data;

[0027] Step 5: Deviation screening: Perform a final round of screening to remove bad segments from the three-level purified EEG data, observe the specific waveform of each group of data, and when the three-level purified EEG data shows an abnormal situation where the amplitude collectively deviates from the normal value, remove the deviated EEG data to form the fourth-level purified EEG data, where the normal value range is 0μV-150μV;

[0028] Step 6: Export the four-level purified EEG data to form an EDF data file and save it as a set file. Encapsulate the data set of each subject into a three-dimensional matrix of M×N×S×SR, where M is the number of trials, N is the number of channels, and S×SR is the product of duration and sampling rate.

[0029] Step 7: Collect and record the subjective emotion scores of the subjects in a three-dimensional matrix of M×N×S×SR, normalize the recorded data first and then binarize it, and select effective emotional stimulation trials to form an M×P data format as the ultimate EEG data, where P is the number of emotion categories.

[0030] As a preferred solution: the ultimate EEG data formed after data preprocessing is used to establish a multimodal data set. The multimodal data set is a multimodal data set for constructing EEG signal features and brain network features. The specific establishment process is as follows:

[0031] The ultimate EEG data is analyzed using feature extraction methods. The feature extraction methods are continuous wavelet transform and brain network features to extract the ultimate EEG data, thereby obtaining the time-frequency scale diagram of different emotions in the ultimate EEG data and the PIL brain network feature values;

[0032] Continuous wavelet transform process:

[0033] The position of the moving base wavelet in the ultimate EEG data and the signal segments of different frequencies are constrained by base wavelet time window functions of different widths, so as to analyze the time-frequency domain characteristics in the ultimate EEG data to form a wavelet transform process. There are two basic types of wavelet transform: continuous wavelet transform and discrete wavelet transform. The conversion process of continuous wavelet transform and discrete wavelet transform is formed by taking different values ​​of parameters a and τ. The definition of wavelet transform is:

[0034]

[0035] In the above formula, a is the scaling factor. When |a|<1, the base wavelet is compressed and corresponds to high frequency. On the contrary, when |a|>1, it corresponds to low frequency. τ is the translation parameter, which is used to move the position of the base wavelet. Signal segments of different frequencies are constrained by base wavelet time window functions of different widths. The time-frequency domain features can be analyzed accurately and efficiently. f(t) is the input signal. It is the wavelet coefficient formed by the expansion or expansion of the basic wavelet at different scales and then translation to different positions. When the parameters a and τ take continuous values, it is called continuous wavelet transform, and when the parameters a and τ take discrete values, it is called discrete wavelet transform;

[0036] The process of extracting PIL brain network feature values ​​from the ultimate EEG data:

[0037] Extract the PIL brain network eigenvalues ​​of node degree and local efficiency in the ultimate EEG data. The extraction formula of the PIL brain network eigenvalues ​​is:

[0038]

[0039] In the above formula: sign is the sign function, Δφ rel (t) is the phase difference of the signal at time t; N is the number of nodes in the EEG data, n is 1, 2, 3...N, when the function's independent variable is greater than 0, the value of sign is equal to 1, when the independent variable is negative, its output is -1, and when the independent variable is equal to 0, the value of sign is equal to 0; the value range of PLI is [0,1], when the value of PLI is equal to 0, it means that the phase difference distribution of the two EEG signals is completely symmetrical, otherwise, the larger the value of PLI, the more asymmetrical the phase difference distribution of the corresponding two sets of EEG signals;

[0040] Finally, PLI is calculated using formula 2 to obtain the PLI connection matrix, which is used to quantify the connection strength between nodes. The threshold T is set for the PLI connection matrix, and binarization is performed to obtain a binarized matrix, which is then mapped to the PLI brain network. After the PLI brain functional network is constructed, the network feature sequence of each node is obtained by calculating the node degree and local efficiency of the network. The network feature sequence is combined with continuous wavelets to complete the process of constructing a multimodal data set, and a convolutional network model is obtained through the construction of a multimodal data set.

[0041] As a preferred solution: the convolutional network model is a multimodal emotion network model, and the constructed convolutional network model is evaluated by classification accuracy. When the classification accuracy reaches more than 90%, there is no need to modify the convolutional network model. When the classification accuracy is lower than 90%, the convolutional network model needs to be modified and adjusted until the classification accuracy reaches 90%.

[0042] As a preferred solution: the sampling rate is 400-600 Hz, which is four to six times the maximum frequency band value among the five frequency bands.

[0043] Compared with the prior art, the present invention provides a multimodal emotion recognition and evaluation method based on multi-feature two-dimensional convolution, which has the following beneficial effects:

[0044] The present invention is an experimental model based on audio stimulation of brain emotions. By collecting emotional EEG signals under different audio stimulations and observing the phase amplitude changes of emotional EEG signals in resting and task states, effective and standardized audio stimulation is achieved. After the EEG data is subjected to multiple preprocessing such as noise removal, filtering, bad segment removal, bad interpolation, and independent component analysis, a multimodal data set is established for the EEG data, a multimodal data set of EEG signal features and brain network features is constructed, and a multimodal deep learning network model is used to achieve accurate classification of brain emotions, thereby effectively improving the efficiency and accuracy of multi-classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the process of the present invention;

[0046] Figure 2 Schematic diagram of the audio stimulation experiment timeline;

[0047] Figure 3 This is a schematic diagram of the model training process;

[0048] Figure 4 It is a schematic diagram of the time-frequency scale generated by continuous wavelet transform;

[0049] Figure 5 It is a schematic diagram of the three-category accuracy data;

[0050] Figure 6 This is a schematic diagram of the four-category accuracy data. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Specific implementation method 1: Combination Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 This embodiment is described. The multimodal emotion recognition evaluation method based on multi-feature two-dimensional convolution in this embodiment is to construct a multimodal data set of EEG signal features and brain network features. The traditional EEG signal features are combined with brain function network features to establish multimodal data analysis and build the latest deep learning recognition model, thereby improving the accuracy of emotion recognition classification.

[0053] The specific process of the present invention is divided into four parts:

[0054] Firstly, an emotional EEG experimental paradigm under audio stimulation was conducted and a subjective evaluation method of the experiment was established;

[0055] Secondly, three platforms were built, namely, the experimental platform, the audio stimulation emotional EEG signal experimental platform and the microwave micro-brain stimulation experimental platform;

[0056] Then, a multimodal dataset of EEG signal features and brain network features is collected and constructed;

[0057] Finally, the multimodal data set that constructs EEG signal features and brain network features is subjected to signal processing, feature extraction, and multimodal deep learning classification and recognition processes.

[0058] First, we conduct multimodal recognition and classification of emotional EEG, conduct experiments based on audio stimulation emotions, collect EEG signals of subjects under audio stimulation, and record the subjective emotional arousal degree and emotional state of subjects. In order to obtain pure EEG information, we preprocess the data to remove the noise of electrooculogram, electrocardiogram, and electromyography physiological signals; perform frequency band filtering for non-physiological signals of power frequency, high frequency, and low frequency; remove bad segments and interpolation bad conductors; and perform independent component analysis to remove artifacts.

[0059] In order to verify the therapeutic application of microwave stimulation in brain dysfunction diseases, an audio stimulation emotion experiment under microwave stimulation was conducted. The subjects were placed in a microwave environment. Audio stimulation was also used. The microwave environment of the subjects was adjusted, and the EEG signals under microwave brain stimulation of different frequency bands were recorded, as well as the subjects' subjective emotional arousal state and degree.

[0060] Secondly, a multimodal dataset of EEG signal features and brain network features was constructed. EEG signal features are mainly composed of time domain features, frequency domain features, time-frequency features and nonlinear features. Time-frequency features were selected to analyze EEG signals. After comparing multiple methods, continuous wavelet transform was selected as the feature extraction method. At present, brain network research based on EEG signals usually constructs brain networks from the perspectives of dependence, synchronization or asynchrony, causality, information transmission, etc. between multi-lead signals. The phase hysteresis index brain functional network was selected to calculate the node degree and local efficiency of the network, and the network feature sequence of each node was obtained. After the wavelet transform time-frequency graph data, brain network feature sequence, category label and other data were encapsulated separately, multimodal fusion was performed. Multimodal fusion will try feature-level fusion and decision-level fusion respectively. Feature-level fusion is to directly splice, add, multiply and other operations of the time-frequency graph and other modal data features to obtain a new feature vector. Decision-level fusion is to use CNN to train the time-frequency graph and other modal data separately, and then fuse their classification results, such as by voting, weighted summation and other methods.

[0061] For model training of multimodal datasets, the EEGNet model is selected. This is a compact EEG feature extraction convolutional neural network based on EEG signals. It is the latest generalized application model for ERP and oscillation-based BCIs. The EEGNet structure uses deep convolution and deep separable convolution for convolution operations, which reduces the number of training parameters. It then uses the exponential linear unit ELU activation function, and finally the features are directly passed to the softmax output of the classification, reducing the number of free parameters in the model. Finally, the model is optimized: considering that the CNN convolutional neural network has obvious advantages in feature extraction, and the LSTM recurrent neural network has great advantages for time series, it is considered to combine the advantages of the two neural networks to improve the entire model and achieve more accurate recognition of emotional EEG signals.

[0062] Specific implementation method two: This implementation method is a further limitation of specific implementation method one. In this implementation method, the multimodal emotion recognition and evaluation method is to form multi-level EEG data after pre-processing the EEG signals obtained based on the audio stimulation emotion test to eliminate noise, and finally form pure EEG information, and then perform feature extraction on the pure EEG information to construct a multimodal data set, and classify, identify and evaluate the multimodal data set through a convolutional neural network.

[0063] Specific implementation method three: This implementation method is a further limitation of specific implementation method one or two. In this implementation method, the process of obtaining EEG signals in the audio stimulation emotion test is as follows:

[0064] Test preparation: A predetermined number of people who meet the relevant requirements are selected as test subjects. The test subjects are placed in a test environment where the room temperature is controlled between 25 and 26 degrees Celsius and the indoor noise is less than 30db. The EEG signals of the test subjects under audio stimulation are collected after the head has undergone impedance reduction operation.

[0065] Preparation for EEG signal collection: determine the number of channels in the scalp area, divide the degree of each emotion into 0 to 10 points, record the emotions generated by each subject after a test and record the corresponding emotion score when the emotion is generated, and use non-invasive wet electrode connection for each person to collect EEG signals through the international standard 10-20 system electrode connection method;

[0066] Audio stimulation process: ensure that each subject completes at least five groups of tests, each group of tests includes at least six trials, forming M test processes. During each trial, ensure that the subject starts brain wave collection with eyes closed at 0 seconds, turn on the audio at 10 seconds, ensure that the audio continues to play for 70 seconds, then turn off the audio and end the brain wave collection, ensuring that each trial contains a time sequence of no less than 60 seconds. After turning off the audio, the subject is required to verbally report the subjective score and complete a rest within a fixed time to complete a trial. According to the above process, the next trial is completed by analogy;

[0067] After the audio stimulation process, primary EEG signals were obtained.

[0068] In this embodiment, the international standard 10-20 system electrode connection method is: according to the recommendations of the International Electroencephalography Society, scalp EEG recording routinely uses the 10%-20% system to determine the placement of electrodes, referred to as the international 10-20 system. It includes 19 recording electrodes (excluding FPz and Oz) and 2 reference electrodes.

[0069] First, determine two baselines on the scalp surface:

[0070] One is the front-to-back line from the nasal root to the external occipital protuberance, which is 100%;

[0071] One is the left-right line between the two preauricular concavities, which is 100%;

[0072] The intersection of the two on the top of the head is the location of the Cz electrode.

[0073] FPz (frontal pole midline) is 10% back from the root of the nose. Every 20% back from FPz is the position of an electrode, namely Fz (frontal midline), Cz (central midline), Pz (parietal midline), and Oz (occipital midline). The distance between Oz and the external occipital protuberance is 10%.

[0074] The T3 (left middle temporal) electrode position is 10% away from the left preauricular fossa along the line connecting the two preauricular fossa, and an electrode is placed every 20% to the right, namely C3 (left central), Cz (central midline), C4 (right central), T4 (right middle temporal), and the distance between T4 and the right preauricular fossa is 10%.

[0075] The line from FPz through T3 to Oz is the left temporal line, 10% to the left from FPz is FP1 (left frontal pole), and an electrode is placed every 20% from FP1 along the left lateral side to the back, in order F7 (left anterior temporal), T3 (left middle temporal), T5 (left posterior temporal), and O1 (left occipital), where T3 is the intersection of this line and the line connecting the bilateral preauricular fossa, and O1 is 10% away from Oz.

[0076] FP2 corresponds to FP1 along the right lateral to posterior line, and an electrode is placed every 20% from front to back in the following order: FP2 (right frontal pole), F8 (right anterior temporal), T4 (right middle temporal), T6 (right posterior temporal), O2 (right occipital);

[0077] Draw a line from FP1 to O1, which is the left parasagittal line. From FP1 straight line back, every 20% is an electrode site. The left side is: F3 (left frontal), C3 (left central), P3 (left parietal), O1 (left occipital);

[0078] Draw a line from FP2 to O2, which is the right parasagittal line. From the FP2 straight line, every 20% backward is an electrode site. The right side is: F4 (right frontal), C4 (right central), P4 (right parietal), O2 (right occipital).

[0079] Specific implementation method 4: This implementation method is a further limitation of specific implementation methods 1, 2 or 3. The process of preprocessing the EEG signals obtained from the audio stimulation emotion test is as follows:

[0080] Step 1: Import the collected EDF data file into the EEG data preprocessing software eeglab, perform electrode positioning according to the international standard 10-20 system electrode connection method, determine the number of channels N, time duration S, sampling rate SR, and acquisition information data of sampling points SP, and select the time series EEG waves stimulated by audio in the primary EEG signals obtained in each group of experiments, that is, the primary EEG signals of the last 60 seconds of the total length of 70 seconds for data preprocessing;

[0081] In the above step, the single audio stimulation test time is ensured to be 70 seconds, and the cut-off time is consistent with the length of time the subjects receive the stimulating music. The test room temperature is controlled at 25℃, the EEG emotion induction method adopts music, and the indoor noise is less than 30db. The non-invasive wet electrode connection method is adopted, following the international standard 10-20 system electrode connection rule. In order to ensure that the emotions induced by music have good controllability and generalization, the corresponding stimulation music is "Rain Sound", "Kikujiro's Summer Piano Music", "The Grotto", "Stratus", "Just Blue", "Lucky" six pure music as random music materials. Stereo dual speakers are used to play each pure music, and the duration is 1min. The subjects are allowed to reach the required emotional state and maintain it to a certain extent, without causing auditory fatigue. A discrete emotion scoring table is designed, and the degree of mobilization of each emotion is divided into 0-10 points. Record the emotions and corresponding emotion scores generated by each subject after one test.

[0082] Step 2: Sort the five frequency bands of the human brain state corresponding to different EEG signal frequency bands, and filter out other clutter in the frequency domain through a digital filter. Clutter filtering includes filtering out the primary EEG signals obtained from each group of experiments by low-pass filtering with a frequency of 30Hz to 100Hz, band-stop filtering with a frequency of 48 to 52Hz, and high-pass filtering with a frequency of 0.1Hz to 1Hz to form a first-level purified EEG data;

[0083] Step 3: The first-level purified EEG data is converted into second-level purified EEG data through the Nyquist sampling theorem under the limitation of acquisition conditions: According to the Nyquist sampling theorem, the first-level purified EEG data is sampled to form second-level purified EEG data, and the sampling conditions are that the sampling rate is more than 4 times higher than the frequency band of interest and at the same time lower than 6 times the frequency band of interest, and the sampling rate ranges from 400 to 600 Hz; wherein, the sampling rate range of 400 to 600 Hz is obtained by the maximum frequency band value among the five frequency bands, that is, four to six times the maximum frequency band among the five frequency bands.

[0084] Step 4: Perform ICA to remove artifacts: Decompose the secondary purified EEG data through ICA and remove the relevant frequency bands marked as artifacts in the secondary purified EEG data. After removal, reassemble the remaining EEG data to form the third-level purified EEG data, which is the real EEG data;

[0085] Step 5: Deviation screening: Perform a final round of screening to remove bad segments from the three-level purified EEG data, observe the specific waveform of each group of data, and when the three-level purified EEG data shows an abnormal situation where the amplitude collectively deviates from the normal value, remove the deviated EEG data to form the fourth-level purified EEG data, where the normal value range is 0μV-150μV;

[0086] Step 6: Export the four-level purified EEG data to form an EDF data file and save it as a set file. Encapsulate the data set of each subject into a three-dimensional matrix of M×N×S×SR, where M is the number of trials, N is the number of channels, and S×SR is the product of duration and sampling rate.

[0087] Step 7: Collect and record the subjective emotion scores of the subjects in a three-dimensional matrix of M×N×S×SR, normalize the recorded data first and then binarize it, and select effective emotional stimulation trials to form an M×P data format as the ultimate EEG data, where P is the number of emotion categories.

[0088] The specific contents of the five frequency bands in this implementation are as follows in Table 1:

[0089] Table 1: Information of different frequency bands of EEG signals

[0090]

[0091] In this embodiment, the filter selected is the existing technology, namely the digital filter, which has outstanding advantages such as high stability, high precision, and great flexibility. The FIR finite impulse response filter can be selected to supplement the neurophysiological signals such as sensory stimulation or motor activity at different time and frequency domain scales. The main function is to attenuate certain frequencies or frequency bands in the original signal. The specific selection of the 150-order Hamming window FIR filter has the following two main functions:

[0092] First: Use band-stop filtering to filter out 50Hz mains interference;

[0093] Second: digitally filter the time series using a bandpass filter with a cutoff frequency of 0.03 Hz and 80 Hz.

[0094] The clutter filtering means in this embodiment includes filtering out the primary EEG signals obtained from each group of experiments by low-pass filtering to remove frequencies above 80 Hz, by band-stop filtering to remove frequencies between 48 and 52 Hz, and by high-pass filtering to remove waves below 0.5 Hz to form a first-level purified EEG data.

[0095] The range of normal values ​​in the three-level purified EEG data in this embodiment is obtained by manually observing the EEG of each collected trial. In the EEG corresponding to the three-level purified EEG data, it can be divided into several levels according to the height of the amplitude, specifically:

[0096] For adult EEG: less than 25μv is called low amplitude, between 25 and 75μv is called medium amplitude, between 75 and 150μv is called high amplitude, and greater than 150μV is called extremely high amplitude. The range of normal values ​​in the three-level purified EEG data is obtained after multiple trials.

[0097] Specific implementation method five: This implementation method is a further limitation of specific implementation methods one, two, three or four. The ultimate EEG data formed after data preprocessing is used to establish a multimodal data set. The multimodal data set is a multimodal data set for constructing EEG signal features and brain network features. The specific establishment process is as follows:

[0098] The ultimate EEG data is analyzed using feature extraction methods. The feature extraction methods are continuous wavelet transform and brain network features to extract the ultimate EEG data, thereby obtaining the time-frequency scale diagram of different emotions in the ultimate EEG data and the PIL brain network feature values;

[0099] Continuous wavelet transform process:

[0100] The position of the moving base wavelet in the ultimate EEG data and the signal segments of different frequencies are constrained by base wavelet time window functions of different widths, so as to analyze the time-frequency domain characteristics in the ultimate EEG data to form a wavelet transform process. There are two basic types of wavelet transform: continuous wavelet transform and discrete wavelet transform. The conversion process of continuous wavelet transform and discrete wavelet transform is formed by taking different values ​​of parameters a and τ. The definition of wavelet transform is:

[0101]

[0102] In the above formula, a is the scaling factor. When |a|<1, the base wavelet is compressed and corresponds to high frequency. On the contrary, when |a|>1, it corresponds to low frequency. τ is the translation parameter, which is used to move the position of the base wavelet. Signal segments of different frequencies are constrained by base wavelet time window functions of different widths. The time-frequency domain features can be analyzed accurately and efficiently. f(t) is the input signal. It is the wavelet coefficient formed by the expansion or expansion of the basic wavelet at different scales and then translation to different positions. When the parameters a and τ take continuous values, it is called continuous wavelet transform, and when the parameters a and τ take discrete values, it is called discrete wavelet transform;

[0103] The process of extracting PIL brain network feature values ​​from the ultimate EEG data:

[0104] Extract the PIL brain network eigenvalues ​​of node degree and local efficiency in the ultimate EEG data. The extraction formula of the PIL brain network eigenvalues ​​is:

[0105]

[0106] In the above formula: sign is the sign function, Δφ rel (t) is the phase difference of the signal at time t; N is the number of nodes in the EEG data, n is 1, 2, 3...N, when the function's independent variable is greater than 0, the value of sign is equal to 1, when the independent variable is negative, its output is -1, and when the independent variable is equal to 0, the value of sign is equal to 0; the value range of PLI is [0,1], when the value of PLI is equal to 0, it means that the phase difference distribution of the two EEG signals is completely symmetrical, otherwise, the larger the value of PLI, the more asymmetrical the phase difference distribution of the corresponding two sets of EEG signals;

[0107] Finally, PLI is calculated using formula 2 to obtain the PLI connection matrix, which is used to quantify the connection strength between nodes. The threshold T is set for the PLI connection matrix, and binarization is performed to obtain a binarized matrix, which is then mapped to the PLI brain network. After the PLI brain functional network is constructed, the network feature sequence of each node is obtained by calculating the node degree and local efficiency of the network. The network feature sequence is combined with continuous wavelets to complete the process of constructing a multimodal data set, and a convolutional network model is obtained through the construction of a multimodal data set.

[0108] Specific implementation method 6: This implementation method is a further limitation of specific implementation methods 1, 2, 3, 4 or 5. In this implementation method, the multimodal emotion recognition evaluation process based on multi-feature two-dimensional convolution includes an evaluation process of a convolutional network model, and the convolutional network model is a multimodal emotion network model. The specific evaluation process is:

[0109] The constructed convolutional network model is evaluated by classification accuracy to evaluate the effectiveness and efficiency of the network model. When the classification accuracy is high, the time is short, and the classification accuracy reaches more than 90%, there is no need to modify the convolutional network model. When the classification accuracy is low, the time is long, and the classification accuracy is lower than 90%, the convolutional network model needs to be modified and adjusted until the classification accuracy reaches 90%. The adjustment method is to adjust the model structure by continuously arranging and combining the multimodal data sets to form a trimming process. The convolutional layer parameter data involved in the present invention are:

[0110]

[0111]

[0112] Specific implementation method 7: This implementation method is a further limitation of specific implementation methods 1, 2, 3, 4, 5 or 6. In this implementation method, the multimodal emotion recognition evaluation process based on multi-feature two-dimensional convolution also includes an evaluation process for a multimodal data set, specifically:

[0113] The process of collecting, identifying and verifying multi-feature two-dimensional convolution multimodal emotion recognition EEG signals. After the emotional EEG signal collection experiment is completed, the process of designing and building an audio stimulation emotional EEG signal collection test platform is carried out. The resting state EEG signals and audio stimulation state EEG signals are collected, and the EEG data in the audio stimulation experiment are judged. The EEG signal data is preprocessed, filtered, independently component analyzed, and normalized to construct a multimodal data set. Specifically, it includes constructing time-frequency domain features and PIL brain network features, and then performing emotional EEG recognition and classification based on multimodal deep learning. The CNN model is used to complete the process of network model analysis and adjustment result analysis, and corresponding evaluation conclusions are given.

[0114] The present invention uses a deep learning model to achieve multimodal emotion classification. Through audio stimulation emotion experiments, six multimodal emotions are obtained, namely: typical emotions of excitement, neutrality, sadness, fear and tension. The convolutional neural network model of the present invention classifies the six emotions, and the accuracy rates of three-classification and four-classification can reach 98.39% and 92% respectively.

Claims

1. A multimodal emotion recognition and evaluation method based on multi-feature two-dimensional convolution, characterized by: The multimodal emotion recognition evaluation method is to pre-process the EEG signals obtained based on audio stimulation emotion experiments to remove noise to form pure EEG information, then extract features from the pure EEG information to construct a multimodal dataset, and classify, recognize and evaluate the multimodal dataset through a convolutional neural network.

2. The multimodal emotion recognition and evaluation method based on multi-feature two-dimensional convolution according to claim 1, characterized in that: The process of obtaining EEG signals in the audio stimulation emotion test is as follows: Test preparation: A predetermined number of people who meet the relevant requirements are selected as test subjects. The test subjects are placed in a test environment where the room temperature is controlled between 25 and 26 degrees Celsius and the indoor noise is less than 30db. The EEG signals of the test subjects under audio stimulation are collected after the head has undergone impedance reduction operation. Preparation for EEG signal collection: determine the number of channels in the scalp area, divide the degree of each emotion into 0 to 10 points, record the emotions generated by each subject after a test and record the corresponding emotion score when the emotion is generated, and use non-invasive wet electrode connection for each person to collect EEG signals through the international standard 10-20 system electrode connection method; Audio stimulation process: ensure that each subject completes at least five groups of tests, each group of tests includes at least six trials, forming M test processes. During each trial, ensure that the subject starts brain wave collection with eyes closed at 0 seconds, turn on the audio at 10 seconds, ensure that the audio continues to play for 70 seconds, then turn off the audio and end the brain wave collection, ensuring that each trial contains a time sequence of no less than 60 seconds. After turning off the audio, the subject is required to verbally report the subjective score and complete a rest within a fixed time to complete a trial. According to the above process, the next trial is completed by analogy; After the audio stimulation process, primary EEG signals were obtained.

3. The multimodal emotion recognition and evaluation method based on multi-feature two-dimensional convolution according to claim 2, characterized in that: The process of preprocessing the EEG signals obtained from the audio stimulation emotion test is as follows: Step 1: Import the collected EDF data file into the EEG data preprocessing software eeglab, perform electrode positioning according to the international standard 10-20 system electrode connection method, determine the number of channels N, time duration S, sampling rate SR, and acquisition information data of sampling points SP, and select the time series EEG waves stimulated by audio in the primary EEG signals obtained in each group of experiments, that is, the primary EEG signals of the last 60 seconds of the total length of 70 seconds for data preprocessing; Step 2: Sort the five frequency bands of the human brain state corresponding to different EEG signal frequency bands, and filter out other clutter in the frequency domain through a digital filter. Clutter filtering includes filtering out the primary EEG signals obtained from each group of experiments by low-pass filtering with a frequency of 30Hz to 100Hz, band-stop filtering with a frequency of 48 to 52Hz, and high-pass filtering with a frequency of 0.1Hz to 1Hz to form a first-level purified EEG data; Step 3: The first-level purified EEG data is sampled to form the second-level purified EEG data according to the Nyquist sampling theorem under the limitation of acquisition conditions: According to the Nyquist sampling theorem, the first-level purified EEG data is sampled to form the second-level purified EEG data. The sampling conditions are that the sampling rate is more than 4 times higher than the frequency band of interest and less than 6 times the frequency band of interest, and the sampling rate is between 400 and 600 Hz. Step 4: Perform ICA to remove artifacts: Decompose the secondary purified EEG data through ICA and remove the relevant frequency bands marked as artifacts in the secondary purified EEG data. After removal, reassemble the remaining EEG data to form the third-level purified EEG data, which is the real EEG data; Step 5: Deviation screening: Perform a final round of screening to remove bad segments from the three-level purified EEG data, observe the specific waveform of each group of data, and when the three-level purified EEG data shows an abnormal situation where the amplitude collectively deviates from the normal value, remove the deviated EEG data to form the fourth-level purified EEG data, where the normal value range is 0μV-150μV; Step 6: Export the four-level purified EEG data to form an EDF data file and save it as a set file. Encapsulate the data set of each subject into a three-dimensional matrix of M×N×S×SR, where M is the number of trials, N is the number of channels, and S×SR is the product of duration and sampling rate. Step 7: Collect and record the subjective emotion scores of the subjects in a three-dimensional matrix of M×N×S×SR, normalize the recorded data first and then binarize it, and select effective emotional stimulation trials to form an M×P data format as the ultimate EEG data, where P is the number of emotion categories.

4. The multimodal emotion recognition and evaluation method based on multi-feature two-dimensional convolution according to claim 3 is characterized in that: The final EEG data formed after data preprocessing is used to establish a multimodal data set. The multimodal data set is a multimodal data set that constructs EEG signal features and brain network features. The specific establishment process is as follows: The ultimate EEG data is analyzed using feature extraction methods. The feature extraction methods are continuous wavelet transform and brain network features to extract the ultimate EEG data, thereby obtaining the time-frequency scale diagram of different emotions in the ultimate EEG data and the PIL brain network feature values; Continuous wavelet transform process: The position of the moving base wavelet in the ultimate EEG data and the signal segments of different frequencies are constrained by base wavelet time window functions of different widths, so as to analyze the time-frequency domain characteristics in the ultimate EEG data to form a wavelet transform process. There are two basic types of wavelet transform: continuous wavelet transform and discrete wavelet transform. The conversion process of continuous wavelet transform and discrete wavelet transform is formed by taking different values ​​of parameters a and τ. The definition of wavelet transform is: In the above formula, a is the scaling factor. When |a|<1, the base wavelet is compressed and corresponds to high frequency. On the contrary, when |a|>1, it corresponds to low frequency. τ is the translation parameter, which is used to move the position of the base wavelet. Signal segments of different frequencies are constrained by base wavelet time window functions of different widths. The time-frequency domain features can be analyzed accurately and efficiently. f(t) is the input signal. It is the wavelet coefficient formed by the expansion or expansion of the basic wavelet at different scales and then translation to different positions. When the parameters a and τ take continuous values, it is called continuous wavelet transform, and when the parameters a and τ take discrete values, it is called discrete wavelet transform; The process of extracting PIL brain network feature values ​​from the ultimate EEG data: Extract the PIL brain network eigenvalues ​​of node degree and local efficiency in the ultimate EEG data. The extraction formula of the PIL brain network eigenvalues ​​is: In the above formula: sign is the sign function, Δφ rel (t) is the phase difference of the signal at time t; N is the number of nodes in the EEG data, n is 1, 2, 3...N, when the function's independent variable is greater than 0, the value of sign is equal to 1, when the independent variable is negative, its output is -1, and when the independent variable is equal to 0, the value of sign is equal to 0; the value range of PLI is [0,1], when the value of PLI is equal to 0, it means that the phase difference distribution of the two EEG signals is completely symmetrical, otherwise, the larger the value of PLI, the more asymmetrical the phase difference distribution of the corresponding two sets of EEG signals; Finally, PLI is calculated by formula 2 to obtain the PLI connection matrix, which is used to quantify the connection strength between nodes; A threshold T is set for the PLI connection matrix, and binarization is performed to obtain a binarized matrix, which is then mapped to the PLI brain network. After the PLI brain functional network is constructed, the network feature sequence of each node is obtained by calculating the node degree and local efficiency of the network. The network feature sequence is combined with continuous wavelets to complete the process of constructing a multimodal data set, and a convolutional network model is obtained through the construction of the multimodal data set.

5. The multimodal emotion recognition and evaluation method based on multi-feature two-dimensional convolution according to claim 4, characterized in that: The convolutional network model is a multimodal emotion network model. The constructed convolutional network model is evaluated by classification accuracy. When the classification accuracy reaches more than 90%, there is no need to modify the convolutional network model. When the classification accuracy is lower than 90%, the convolutional network model needs to be modified and adjusted until the classification accuracy reaches 90%.

6. The multimodal emotion recognition and evaluation method based on multi-feature two-dimensional convolution according to claim 3 is characterized in that: The sampling rate of 400-600 Hz is four to six times the maximum frequency band value of the five frequency bands.

Images