Facial nerve decoding method based on electroencephalogram signals

Abstract

Facial nerve decoding method based on electroencephalogram signal relates to the field of neural decoding and computer vision. It includes the following steps: 1) extracting facial expression information from human brain neural data to establish a realistic face generation model; 2) locking the corresponding brain area and neural band of human brain neural data by using DEAP data set; 3) connecting electroencephalogram and facial data through LSTM model, and quantifying the emotional parameters of subjects as the initialization parameters of LSTM by using CNN model; 4) editing the emotional parameters on the basis of the original face by using StyleGan network to obtain generated pictures very close to the original facial expression. Realistic faces can be generated, which is a step forward in the visualization level. It realizes an innovative step of directly decoding facial features from neural signals. The whole process from preprocessing to application is given, which makes good use of the advantages of the cross between brain science and computer science and optimizes the signal-to-noise ratio of the data end.

Description

Technical Field

[0001] This invention relates to the fields of neural decoding and computer vision, and in particular to a neural decoding method for facial expressions based on electroencephalogram (EEG) signals. Background Technology

[0002] StyleGan, as one of the milestones in computer graphics in recent years, has very powerful face editing capabilities. It can edit the original facial expression into the target expression with only quantified emotion parameters. In the field of neural decoding, emotion decoding is already very advanced. When the two are organically combined, there is hope for solving many of the social problems mentioned above. However, at present, no one has combined the two. This is the original intention of this invention - to generate facial expressions using EEG. Summary of the Invention

[0003] The purpose of this invention is to address the problem of unrealistic facial expression generation in existing technologies by providing a neural decoding method for facial expressions based on electroencephalogram (EEG) signals.

[0004] This invention includes the following steps:

[0005] 1) Extract facial expression information from human brain neural data and establish a realistic face generation model;

[0006] 2) Using the DEAP dataset, the corresponding brain regions and neural bands in human brain neural data were located;

[0007] 3) Connect EEG and facial data through an LSTM model, and use a CNN model to quantify the subject's emotional parameters as the initialization parameters for the LSTM;

[0008] 4) The StyleGAN network is used to edit the emotion parameters on the basis of the original face to obtain a generated image that is very close to the original face expression.

[0009] In step 1), the specific steps for extracting facial expression information from human brain neural data and establishing a realistic face generation model can be as follows:

[0010] (1) Decompose the neural signal into multiple frequency bands, including the δ band (1-3 Hz), the θ band (4-7 Hz), the α band (8-13 Hz), the β band (14-30 Hz) and the γ band (31-50 Hz).

[0011] (2) Extract features from each frequency band, where the extracted features are differential asymmetry.

[0012] (3) At the same time, time and frequency domain information are used to reflect the properties of neural signals, and the feature extraction method is wavelet transform.

[0013] In step 2), the sub-step of locking the corresponding brain region and neural waveband of the human brain neural data is as follows:

[0014] (1) Identify the brain regions and wavelengths that trigger human emotions based on relevant papers;

[0015] (2) Visualize brain region activity in the DEAP dataset using the MNE tool;

[0016] (3) The intersection of the actual brain regions with the brain regions identified in the paper is used to locate the brain regions.

[0017] In step 3), the sub-step of quantifying the subject's emotional parameters is as follows:

[0018] (1) Take the matching facial expression recognition CNN model of FER2013, remove the last classification head, and take its direct output;

[0019] (2) Use the output as the standard parameter for quantifying emotion, and use this number to initialize the intermediate value of the two LSTM layers.

[0020] In step 4), the sub-step of editing the emotion parameters based on the original face using the StyleGan network is as follows:

[0021] (1) Based on the face similarity loss function, the StyleGAN model is called to generate images similar to real faces.

[0022] (2) Record the specific parameters of stylegan when generating similar images, and use them as the final output samples of the LSTM model.

[0023] (3) Using EEG as input and stylegan parameters as output, perform one round of LSTM training to output parameters that control stylegan to generate real faces, and finally obtain real faces.

[0024] Compared with the prior art, the present invention has the following advantages and technical results:

[0025] 1. Currently, whether it is speech, eye movement or electromyography, these time-series signals can only generate a "face" composed of 144 coordinate points, which is far from a real face. However, this invention can generate a realistic face that is indistinguishable from a real face, taking a step forward in the visualization level and achieving the purpose of this invention.

[0026] 2. Currently, in the field of neural decoding, there are only methods that use neural signals for emotion classification. In recent years, with the development of computer vision, the bridge between emotion and expression has been gradually built. This method achieves an innovative step by directly decoding facial expression features from neural signals.

[0027] 3. This method provides a complete process from preprocessing to application, making good use of the advantages of the intersection of brain science and computer science, and optimizing the signal-to-noise ratio of the data. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating an embodiment of the present invention.

[0029] Figure 2 Perform rotation correction on the face.

[0030] Figure 3 This is a quantitative result of the emotion assessment. A represents the quantified emotion curve, and B represents the image showing the transition from happiness to anger.

[0031] Figure 4 This is an effect generated using Mona Lisa as a sample.

[0032] Figure 5 This is a comparison between the original video quantization result and the video quantization result generated by this invention.

[0033] Figure 6 A preprocessing procedure for collecting EEG signals based on the brain regions corresponding to emotions.

[0034] Figure 7 This is a comparison of parameter results and a demonstration of facial motion effects.

[0035] Figure 8 A visual comparison of the optical flow graphs of the original and generated images. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the following embodiments will be used in conjunction with the accompanying drawings to further illustrate the invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0037] This invention attempts to extract facial expression information from human brain neural data, considering more constraints to establish a realistic face generation model, and then optimizes it using artificial intelligence methods to generate images that closely resemble human faces. It employs face-specific methods from computer graphics to optimize the collected facial data, and utilizes the DEAP dataset to pinpoint corresponding brain regions and neural bands in the human brain neural data, enhancing the constraints on the neural data and improving the signal-to-noise ratio. Then, an LSTM model is used to connect EEG and facial data, innovatively quantifying the subject's emotional parameters. Finally, a StyleGAN network is used to edit the emotional parameters based on the original face, resulting in a generated image that closely resembles the original facial expression.

[0038] like Figure 1 A complete neural decoding system comprises two parts: encoding and decoding. The encoding process records and extracts electrical signals from motor neurons in the cerebral cortex, compiling them into corresponding transition signals to extract complex information from the brain. This invention uses the publicly available EEG dataset DEAP as the EEG input. Following this, the decoding process analyzes the transition signals using a computational model, ultimately outputting the subject's behavior. This invention provides a process for decoding facial expressions using human brain neural signals, including preprocessing, training, and application.

[0039] Angle Correction

[0040] See Figure 2 The process involves rotating and correcting the face. First, the default face detector in Dlib is used to detect the face, generating a bounding box around it. Then, this bounding box is used to crop the face, as shown below. Figure 2 As shown in Figures A and B, the process involves several steps. First, a convenience function from the `imutils` package is used. This function first locates facial landmarks on the cropped image using a pre-trained face landmark detector from the `Dlib` library. Then, the convenience function uses the detected landmarks to calculate the center points of the eyes. Subsequently, the face is aligned so that the eyes are on a horizontal line, as shown in Figures A and B. Figure 2 Figure C in the diagram.

[0041] Emotion Quantification

[0042] Emotion quantification is a core component of the method; it serves as both the training labels for the neural decoder and the input parameters for real face generation. The FER2013 dataset and its accompanying CNN emotion classification model are used for emotion quantification. The FER2013 dataset contains 14,000 labeled face images. The classification head of the CNN emotion classification model is removed, and the confidence score output by the softmax activation function is directly used as the quantification parameter for facial emotion. To demonstrate the effectiveness of the proposed quantization method, as follows... Figure 3The experiment used an image ranging from happy to angry to generate a quantitative emotion curve. Figure 3 Image B in the image shows a progression from happiness to anger. Figure 3 Figure A shows the emotion curve quantified by the method of this invention. The curve for happiness decreases as the image number increases, while the curve for anger increases, indicating that the quantification method of this invention is effective.

[0043] Neural Decoding

[0044] The decoder maps EEG recordings to facial expressions through a two-stage process by learning intermediate mappings between EEG recordings and emotional features, as well as between emotional features and facial expression features. The model samples and processes EEG at a frequency of 200 Hz. This model is implemented in Python using TensorFlow. In the first stage, a stacked three-layer bLSTM learns the relationship between a 300 ms (60 time points) sequence of high-gamma and local frequency components and the corresponding individual time points (sampled at 200 Hz) representing the 0–10 interval expression features.

[0045] During testing, the decoder processes the entire neural activity sequence (high gamma and low-frequency components). In the first phase, 300ms of data are processed at a time, sliding the sequence sample by sample until an emotion sequence of equal length to the neural data is returned. The neural data is padded with an additional 150ms of data before and after the sequence to ensure the correct length of the result.

[0046] At each stage, the ADAM optimizer is used to train the model to minimize the mean squared error. The optimizer is initialized with a learning rate of 0.001, β1 = 0.9, β2 = 0.999, and ε = 1e-8. Model training stops after verifying that the loss no longer decreases. The dropout rate is set to 50% in stage 1 and 25% in stage 2 to suppress the model's overfitting tendency. Each LSTM unit has 100 hidden units. Each model uses three stacked bLSTMs and additional linear layers for regression. bLSTMs are used because this model is able to preserve long-term temporal dependencies when decoding sequences.

[0047] Face generation

[0048] The second stage processes the entire sequence at once, returning facial images of equal length, and then synthesizes these images into an expression image. The emotion quantization parameter is set as the initial value to guide StyleGAN generation, and a loss function is used to control StyleGAN to generate an image that is as similar as possible to the original image.

[0049] At this stage, when using StyleGAN to analyze the original image and generate corresponding latent variables, the learning rate is 0.08. Then, the variation range of the latent variables is modified by the emotion quantification parameter. The variation range is based on the emotion latent variables provided by the official documentation, and the perturbation value of the generated face is 0.8.

[0050] The loss function used for model training:

[0051]

[0052] in, and These are the similarities in the optical flow brightness regions of the real face and the generated face, such as... Figure 4 As shown, in order to protect the privacy of EEG providers, Mona Lisa was used as the sample for generation; based on the characteristics of this method, the region range of optical flow brightness was optimized, and facial regions with strong expression correlation, such as eyes and mouth, were selected, while other facial regions with weak expression correlation were discarded.

[0053] Application Examples

[0054] The following example uses an emotional stimulation dataset published by a hospital to demonstrate the corresponding preprocessing for video and EEG data.

[0055] For the video portion, the angle of each frame was adjusted and emotion quantified to ensure that the video genuinely conveyed clear facial emotions. The generated videos were also tested together. Figure 5 As shown, the generated effects are very similar.

[0056] In the EEG section, the preprocessing parameters of the DEAP dataset were used for debugging, and the neural signals were decomposed into multiple frequency bands. Features were extracted from each frequency band. Here, the feature used in this invention is differential asymmetry.

[0057] Figure 6 To collect EEG signals based on the brain regions corresponding to emotions, and then perform dimensionality reduction and filtering processing.

[0058] After data preprocessing, neural decoding technology is used to map the data from both sides, training a neural decoding model, and reserving a portion of the data as a validation set. After training the neural decoding model, parameter comparisons and results are presented for the original and generated faces, and the faces are animated to protect privacy. Figure 7 As can be seen, the facial changes are smooth, and each image has obvious motion effects.

[0059] Figure 8A visual comparison of the optical flow graphs of the original and generated images shows that the activity points of the generated and original images overlap, all concentrated on the eyes, mouth, and nose. The generative model of this invention understands the main parts of facial expression changes.

[0060] In addition, to prove that the faces in the images generated by this invention are real, steps 3 and 4 were repeated using the face data provided by FER2013 to obtain a series of standard values ​​of face similarity loss function, and these values ​​were compared with the images generated by this invention. The comparison of the generated image parameter results is shown in Table 1.

[0061] Table 1 compares different face similarity loss functions with the FER2013 dataset.

[0062] Loss\Dataset Standard value No. 1 No. 2 No. 3 No. 4 No. 5 OUR >60 62.48 42.41 67.43 45.12 60.42 MSE <0.01 0.012 0.02 0.003 0.005 0.003 RMSE <15% 12% 19% 5.5% 8.1% 4.2% FAR <0.01 0.004 0.005 0.001 0.006 0.001

[0063] As shown in Table 1, the face quality generated by this invention basically meets the corresponding standard values ​​for mainstream face metrics in computer vision. Here, OUR refers to the optical flow similarity of the local facial features (eyes, mouth, nose) proposed in this invention; MSE is the mean squared error of the absolute values ​​of the original and generated face coordinates, measuring coordinate similarity; RMSE is a variant of MSE that eliminates interference factors caused by the spatial position of the face, measuring coordinate similarity; and FAR refers to the similarity of the abstract facial features between the current generated image and the previous generated image. The standard values ​​were trained using the public dataset FER2013.

Claims

1. A facial expression neural decoding method based on electroencephalogram (EEG) signals, characterized in that... Includes the following steps: 1) Extract facial expression information from human brain neural data and establish a face generation model; 2) Using the DEAP dataset, the corresponding brain regions and neural bands in human brain neural data were located; The sub-step for locking the corresponding brain region and neural waveband of human brain neural data is as follows: (1) Identify the brain regions and wavelengths that trigger human emotions; (2) Visualize brain region activity in the DEAP dataset using the MNE tool; (3) The intersection of the actual brain region with the identified brain region is used to locate the brain region; 3) Connect EEG and facial data through an LSTM model, and use a CNN model to quantify the subject's emotional parameters as the initialization parameters for the LSTM; The sub-step for quantifying the subject's emotional parameters is as follows: (1) Take the matching facial expression recognition CNN model of FER2013, remove the last classification head, and take its direct output; (2) Use the output as the standard parameter for quantifying sentiment, and use this number to initialize the intermediate value of the two LSTM layers; 4) The StyleGAN network is used to edit the emotion parameters on the basis of the original face to obtain a generated image that is very close to the original face expression; The sub-step of editing the emotion parameters based on the original face using the StyleGAN network is as follows: (1) Based on the face similarity loss function, the StyleGAN model is called to generate images similar to real faces. (2) Record the specific parameters of stylegan when generating similar images, and use them as the final output samples of the LSTM model. (3) Using EEG as input and stylegan parameters as output, perform one round of LSTM training to output parameters that control stylegan to generate real faces, and finally obtain real faces.

2. The facial expression neural decoding method based on electroencephalogram (EEG) signals as described in claim 1, characterized in that... In step 1), the specific steps for extracting facial expression information from human brain neural data and establishing a face generation model are as follows: (1) The neural signal is decomposed into multiple frequency bands, including the δ band 1-3 Hz, the θ band 4-7 Hz, the α band 8-13 Hz, the β band 14-30 Hz and the γ band 31-50 Hz; (2) Extract features from each frequency band, wherein the extracted features are asymmetric in difference; (3) At the same time, time and frequency domain information are used to reflect the properties of neural signals, and the feature extraction method is wavelet transform.

Images