Method and system for generating emotional digital human based on brain-computer interface and large language model

Abstract

The application discloses a kind of based on brain-computer interface and big language model Emotional digital person generation method and system belong to artificial intelligence and man-machine interaction technical field.This method carries out pre-processing to user electroencephalogram, extracts time-frequency feature and spatial topological feature and fuses and identifies emotion;Again, emotion embedding is fused with text semantic feature, combined with knowledge base search and big language model generates emotional companion dialogue, and drives digital person expression, voice, rendering and video synthesis.The application can perceive user emotion in real time, generates emotional consistent companion digital person video.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and human-computer interaction technology, specifically relating to a method and system for generating emotional digital humans based on brain-computer interfaces and large language models. Background Technology

[0002] With the aging population and the growing prominence of mental health issues, emotional support has become a focus of social attention. According to WHO statistics, approximately 350 million people worldwide suffer from mental health problems, while professional psychological counseling resources suffer from insufficient coverage and high costs. Traditional emotional support relies on human services, which are characterized by high costs, limited coverage, and inconsistent service quality. Utilizing artificial intelligence technology to create virtual digital avatars for emotional support represents a highly promising solution.

[0003] Most existing emotion interaction systems rely on a single modality of text or speech for emotion recognition, making it difficult to accurately capture the user's true emotional state. Facial expression-based methods are easily affected by users' deliberate attempts to conceal their emotions, while speech-based methods perform poorly in noisy environments. Electroencephalogram (EEG) signals, as physiological signals that directly reflect brain activity, have the advantages of being difficult to fake and having high temporal resolution. However, existing EEG emotion recognition methods often only utilize single-dimensional features in the time or frequency domains, ignoring the multidimensional information of EEG signals in the time, frequency, and spatial domains, resulting in low recognition accuracy.

[0004] In terms of dialogue generation, existing chatbots are mostly based on predefined rules or simple sequence-to-sequence models, resulting in dialogues that lack emotional resonance and personalization. Although large language models demonstrate powerful text generation capabilities, when directly applied to emotional support scenarios, they lack the ability to perceive and respond to users' real-time emotional states, and the generated content may not match users' emotional needs.

[0005] In the field of digital human generation, most existing technologies treat facial expression generation, speech synthesis, and digital human rendering as independent modules, lacking a unified emotion-driven mechanism. This results in inconsistencies between the digital human's facial expressions, speech, and dialogue content. Furthermore, significant individual differences exist in the emotional expression patterns of different users, and existing systems lack the ability to quickly adapt to specific users, making it difficult to provide personalized emotional support services.

[0006] Furthermore, existing multimodal fusion methods often employ simple splicing or weighted summation, failing to effectively model the interaction relationships and complementary information between different modes, thus limiting the overall performance of the system.

[0007] In summary, existing technologies have significant shortcomings in terms of EEG emotion recognition accuracy, multimodal emotion fusion, emotion-consistent dialogue generation, and personalized adaptation. There is an urgent need for a method and system for generating emotionally supportive digital humans that can perceive users' EEG emotional states in real time, collaboratively generate emotion-consistent dialogues, facial expressions, and voices, and quickly adapt to individual users. Summary of the Invention

[0008] To address the aforementioned problems, this invention proposes a method and system for generating emotional digital humans based on brain-computer interfaces and large language models.

[0009] The above objectives are achieved through the following technical solutions:

[0010] This invention first provides a method for generating emotional digital humans based on brain-computer interfaces and large language models, the method comprising the following steps:

[0011] (1) The collected raw EEG signals from users were preprocessed by downsampling, baseline correction and independent component analysis artifact removal to obtain clean EEG signals;

[0012] (2) Extract time-frequency-space three-dimensional features from the clean EEG signal obtained in step (1), and input it into the emotion recognition model for emotion classification after fusion through the cross-attention mechanism to obtain the emotion probability distribution vector;

[0013] (3) Gated multimodal fusion is performed between the emotion embedding corresponding to the emotion probability distribution vector obtained in step (2) and the semantic features of the user dialogue text to obtain the emotion embedding vector and multimodal fusion features;

[0014] (4) Based on the multimodal fusion features obtained in step (3), relevant documents are retrieved from the emotional care knowledge base using the retrieval enhancement generation framework, enhanced prompts are constructed in combination with the dialogue history, and emotional care dialogue responses are generated through the large language model.

[0015] (5) Based on the emotion embedding vector obtained in step (3), generate expression parameters through multilayer perceptron mapping, and generate digital human facial expression images using generative adversarial network;

[0016] (6) Extract text features from the dialogue response obtained in step (4), concatenate the text features with the emotion embedding vector obtained in step (3) and map them into an acoustic conditional vector, and perform emotional speech synthesis through the conditional WaveNet waveform generation network to obtain a speech waveform;

[0017] (7) The emotional embedding vector obtained in step (3) is used as the conditional input emotional conditional neural radiation field. Combined with the facial expression image obtained in step (5) and the speech waveform obtained in step (6), digital human rendering and video synthesis are performed through facial region mask fusion to obtain the emotional companion digital human video.

[0018] (8) Perform multi-dimensional quality assessment on the emotional companionship digital human video obtained in step (7), including emotional consistency assessment, facial expression generation quality assessment, speech naturalness assessment and lip-sync assessment, and optimize each generation module based on the assessment results.

[0019] This invention also provides a digital human generation system for emotional companionship based on the collaboration of brain-computer interface and large language model. This system is used to perform the above-described method, including:

[0020] The EEG signal acquisition and preprocessing module is used to downsample, baseline correct, and remove artifacts through independent component analysis of the user's raw EEG signal to obtain a clean EEG signal.

[0021] The emotion recognition module is used to extract time-frequency-spatial three-dimensional features and perform cross-attention fusion on clean EEG signals, and output an emotion probability distribution vector.

[0022] The multimodal feature fusion module is used to fuse the sentiment embedding vector and text semantic features in a multimodal manner through a gating mechanism, and output multimodal fused features;

[0023] The retrieval-enhanced emotional dialogue generation module is used to retrieve relevant documents from the emotional companionship knowledge base based on multimodal fusion features and generate emotional companionship dialogue responses through a large language model.

[0024] The facial expression generation module is used to generate digital human facial expression images based on emotion embedding vectors using a generative adversarial network.

[0025] The speech synthesis module is used to concatenate dialogue response text features with emotion embedding vectors and then synthesize emotional speech waveforms through a conditional WaveNet waveform generation network.

[0026] The digital human rendering and video synthesis module is used to perform 3D rendering using emotional conditional neural radiation fields, fuse facial expression images and rendered images through facial region masking, and synthesize speech waveforms in a time-synchronous manner to output emotional companionship digital human videos.

[0027] The generation result evaluation module is used to perform multi-dimensional quality evaluation on the generated digital human videos and feed the evaluation results back to each generation module for optimization.

[0028] The advantages of this invention compared to the prior art are:

[0029] 1. A three-dimensional EEG emotion recognition method based on time-frequency-spatial feature fusion is proposed. The method extracts time-frequency features through continuous wavelet transform and extracts spatial topological features through graph convolutional network. The cross-attention mechanism is used to achieve effective fusion, which significantly improves the accuracy of emotion recognition.

[0030] 2. A multimodal emotion fusion method based on gating mechanism was designed to adaptively adjust the fusion weights of EEG emotion features and text semantic features, effectively modeling the complementary relationship between modalities.

[0031] 3. An emotion-specific retrieval-enhanced generation framework was constructed. It utilizes multimodal fusion features to retrieve highly relevant reference documents from the emotional support knowledge base and combines the generation capabilities of a large language model to achieve high-quality dialogue generation with consistent emotions.

[0032] 4. It realizes the collaborative generation of emotion-driven digital human expressions, speech and dialogue. By driving the generation of each modality through a unified emotion embedding vector, it ensures the emotional consistency of the digital human output.

[0033] 5. By introducing cross-modal contrastive learning to align the representation spaces of each modality and combining it with a meta-learning mechanism, we can achieve rapid adaptation for individual users and enhance the system's personalized service capabilities.

[0034] 6. By using MMD-based transfer learning to address the individual differences in EEG signals, the cross-subject generalization ability of the emotion recognition model was improved. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0036] Figure 2 This is a schematic diagram of the EEG signal preprocessing process.

[0037] Figure 3 This is a schematic diagram of the time-frequency-spatial three-dimensional feature extraction and cross-attention fusion structure.

[0038] Figure 4 This is a schematic diagram of the overall architecture of the system of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the relevant principles and requirements in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0040] This embodiment provides an emotional companionship digital human generation system based on brain-computer interface and large language model collaboration. The system includes an EEG signal acquisition and preprocessing module, an emotion recognition module, a multimodal feature fusion module, a retrieval-enhanced emotional dialogue generation module, an expression generation module, a speech synthesis module, a digital human rendering and video synthesis module, and a generation result evaluation module. Based on the above system, the method of this embodiment includes the following steps.

[0041] Step (1): EEG signal acquisition and preprocessing.

[0042] The goal of this step is to preprocess the raw EEG signals collected from the user to remove noise and artifacts, and obtain high-quality clean EEG signals.

[0043] (1-1) EEG signal acquisition

[0044] A multi-channel EEG acquisition device conforming to the international 10-20 system standard was used to collect the user's EEG signals during emotional companionship interactions. The raw EEG signals are represented as follows: in, This refers to the number of EEG acquisition channels, which can take different values ​​under different experimental paradigms, such as in the DEAP dataset. In the SEED dataset In the DREAMER dataset ; The number of time sampling points depends on the original sampling rate and sampling duration of the acquisition device.

[0045] (1-2) Downsampling

[0046] To standardize the sampling rate across different devices and reduce computational complexity, the original signal is downsampled: in, The target sampling rate is used. During downsampling, a low-pass anti-aliasing filter is first used to eliminate high-frequency aliasing components. Then, the sample is uniformly decimated according to the sampling rate ratio to obtain the downsampled signal. , This represents the number of time sampling points after downsampling. This represents the original sampling rate.

[0047] (1-3) Baseline Correction

[0048] To eliminate baseline drift in EEG signals, baseline correction is performed on the downsampled signals: in, This is the set of sampling point indices corresponding to the time period from 200ms before stimulus presentation to the baseline at 0ms. This represents the number of baseline sampling points contained in the set. For downsampled signals at the index The values ​​corresponding to the sampling points. Baseline correction is performed independently for each channel, subtracting the average value of the signal value of each channel from the set of baseline sampling points to eliminate DC offset and slow drift.

[0049] (1-4) Removal of artifacts from independent component analysis

[0050] Independent component analysis (ICA) was performed on the baseline-corrected signal to decompose it. in, It is a mixing matrix, where each column represents the contribution weight of an independent component to each channel; It is an independent component matrix, where each row represents the time series of an independent component.

[0051] By analyzing the time-domain waveforms, spectral characteristics, and spatial distribution topology of each independent component, interference components such as electrooculogram (EOG), electromyogram (EMG), and electrocardiogram (ECG) artifacts are identified. The rows corresponding to the artifact components are then arranged from... Remove from the middle, the corresponding column from By removing artifacts, we obtain the matrix after artifact removal. and ,in The number of independent components retained. Reconstructing clean EEG signals: The results obtained The cleaned EEG signals, after removing artifacts, will be used as input for subsequent emotion recognition steps.

[0052] Step (2): Emotion recognition based on deep learning.

[0053] This step involves cleaning the EEG signal obtained in step (1). We perform time-frequency-spatial three-dimensional feature extraction and then perform emotion classification by fusing them through a cross-attention mechanism.

[0054] (2-1) Time-frequency feature extraction

[0055] Cleaning EEG signals Perform continuous wavelet transform (CWT) on each channel to extract the joint time-frequency domain representation: in, The frequency bands correspond to the five standard frequency bands of EEG signals: delta band (1-4Hz), theta band (4-8Hz), alpha band (8-14Hz), beta band (14-30Hz), and gamma band (30-45Hz). Morlet wavelet is used as the mother wavelet to achieve a balance between time and frequency resolution.

[0056] (2-2) Construction of spatiotemporal topology graph

[0057] Based on clean EEG signals Constructing a spatiotemporal topology map of brainwaves: in, This is a three-dimensional tensor, with the first two dimensions corresponding to EEG channels and the third dimension corresponding to time. The element values ​​represent the Pearson correlation coefficients between the corresponding two channels at a given time point. The topology graph is updated step-by-step within a sliding time window, reflecting the dynamic changes in functional connectivity between brain regions.

[0058] (2-3) Convolutional Neural Network Extracts Time-Frequency Feature Vectors

[0059] Using multi-layer convolutional neural networks (CNNs) to analyze time-frequency features Perform deep feature extraction:

[0060] The CNN network contains three convolutional blocks, each consisting of a 2D convolutional layer (3×3 kernels), a batch normalization layer, a ReLU activation function, and a max pooling layer, with output channels of 64, 128, and 256, respectively. The time-frequency feature vector is obtained after global average pooling. , For feature dimensions.

[0061] (2-4) Graph convolutional networks extract spatial feature vectors

[0062] Using Graph Convolutional Networks (GCNs) to analyze spatiotemporal topological graphs Spatial feature extraction: in, The adjacency matrix is ​​constructed based on the physical locations of the electrodes in the International 10-20 system. Edge weights between adjacent electrodes are set to 1, while weights for non-adjacent electrodes are decayed according to Euclidean distance. The GCN network employs two graph convolutional layers, and spatial feature vectors are obtained after a global readout operation. .

[0063] (2-5) Cross-attention fusion

[0064] Using time-frequency features as queries and spatial features as keys and values, a cross-attention mechanism is employed for fusion: in, , , These are the learnable projection matrices for the query, key, and value, respectively. The common dimension of the query vector and the key vector is used as a scaling factor to prevent the dot product value from becoming too large; The dimensions of the value vector and the fused feature vector; Indicates matrix transpose; This is the fused feature vector.

[0065] (2-6) Emotion Classification

[0066] Sentiment classification based on fused feature vectors: in, This is the classification layer weight matrix. For bias vectors, This represents the number of sentiment categories corresponding to the current dataset's labeling system. For emotion category variables, Let be the probability distribution vector for each emotion category. For the valence or arousal tasks of DEAP and DREAMER, take . For the SEED three-class classification task, For the four categories of happiness, sadness, fear, and neutrality, take And configure the corresponding dimensions respectively. and .

[0067] The emotion classification model is trained using the cross-entropy loss function: in, For the first True labels for emotion types (one-hot encoding) The model predicts the first Emotional probability, subscript This only represents the sentiment category index. This loss function is used to train the sentiment recognition model so that the sentiment probability distribution output by the model is as close as possible to the true label distribution.

[0068] (2-7) Cross-subject transfer learning

[0069] During the training phase of the emotion recognition model, maximum mean difference (MMD) is used for domain adaptation to address the issue of individual differences among different subjects. in, and These are the fused feature sets obtained from the source and target domain EEG samples through the emotion recognition model in step (2), respectively. and These represent the number of samples in the two domains, and The first The source domain sample and the first The feature vectors obtained from the target domain samples are obtained through time-frequency feature extraction, spatial feature extraction, and cross-attention fusion. The reproducing kernel Hilbert space eigenmap function is used, employing a Gaussian radial basis kernel function. Let represent the squared L2 norm. By minimizing... This enables the emotion recognition model to generalize across subjects.

[0070] Step (3): Multimodal fusion of emotional features and textual features.

[0071] This step involves multimodal fusion of the emotion probability distribution vector obtained in step (2) with the user's dialogue text.

[0072] (3-1) Emotional Embedding

[0073] The emotion probability distribution vector obtained from step (2) The emotion category with the highest probability is embedded and encoded: in, An embedding lookup table is used to embed emotion categories, mapping discrete emotion categories to continuous dense vectors. , Embedding the dimension of emotion.

[0074] (3-2) Text semantic encoding

[0075] Semantic encoding of user-input dialogue text: in, The dialogue text entered by the user. For the encoder of the BERT pre-trained language model, the hidden state at the [CLS] position is taken as the text feature vector. , The text feature dimension is then obtained through linear projection. ,in, and These are the text projection matrix and the text projection bias, respectively. These are the text features after dimension alignment.

[0076] (3-3) Gating Fusion

[0077] Adaptive fusion of emotion embedding vectors through gating mechanism Text feature vectors aligned with dimensions : in, For the gated weight matrix, This is the gated bias vector. It is a sigmoid activation function. This represents vector concatenation. This represents element-wise multiplication. This is a gating vector that controls the relative contributions of sentiment features and text features in the corresponding dimension. This is the final multimodal fusion feature.

[0078] Step (4): Generate emotional companionship dialogue based on retrieval enhancement.

[0079] This step utilizes the multimodal fusion features obtained in step (3). Emotional companionship dialogues are generated using the Retrieval-Augmented Generation (RAG) framework.

[0080] (4-1) Retrieval of Emotional Knowledge Base

[0081] by Corresponding emotional state text tags As a query, it retrieves information from a pre-built knowledge base on emotional support. Related documents: in, A knowledge base for emotional support; To determine the number of documents to retrieve, this embodiment takes... ; This is the collection of relevant documents retrieved. The function that maps multimodal fusion features to sentiment state text descriptions is implemented based on the FAISS index.

[0082] (4-2) Enhanced prompt construction

[0083] The multimodal fusion features obtained from step (3) Corresponding emotional state text tags, search document set The text content and the history of multiple rounds of dialogue Enhanced prompts are organized into text-based prompts according to a preset prompt template: in, For the most recent A dialogue with history, and The first Round-by-round user text and system response, round index ; For emotional state text description, The preset prompt template function organizes sentiment tags, searched documents, and conversation history into structured text prompts.

[0084] (4-3) Dialogue generation

[0085] Generate dialogue responses based on enhanced prompts using a large language model: in, For large language models, an autoregressive decoding method is used to generate word-by-word meta-generators.

[0086] (4-4) Optimization of Emotional Consistency

[0087] Evaluate the quality of generated dialogue using the affective consistency reward function: in, The current sentiment label of the user is obtained from the sentiment probability distribution in step (2); For text sentiment classifier Emotional labels derived from dialogue responses; Emotional support guidance function Guide emotions based on the goals determined by the current sentiment tags and conversation history. The cosine similarity function is used. For smoothness evaluation function; For the current large language model, enhance prompts The probability distribution of the word-by-word meta-generation strategy output below. For the frozen reference large language model in the same The probability distribution of the word-by-word meta-reference strategy output below. Let KL divergence be the two. , , To balance the weights, define the dialogue optimization loss. This is used to transform maximizing rewards into minimizing losses.

[0088] Step (5): Emotion-driven digital human expression generation.

[0089] This step is based on the emotion embedding vector obtained in step (3). Generate a sequence of digital human facial expression images that change frame by frame and match the user's emotional state. Let... The total number of frames in the video to be generated, the first... Frame normalized time position is And obtained through a time mapping network .

[0090] (5-1) Expression parameter mapping

[0091] The emotion embedding vector is mapped to facial action unit (AU) expression parameters through a multilayer perceptron: in, For video frame indexing, For the first Frame normalized time position, A time-mapping network for frame-by-frame modulation of emotion embeddings. For the first Frame emotion embedding; It consists of three fully connected layers with hidden layer dimensions of 256 and 128 respectively, the activation function is LeakyReLU, and the output dimension is 46, corresponding to the 46 AU parameters in the FACS system.

[0092] (5-2) Generative Adversarial Network for Facial Expression Generation

[0093] Input the facial expression parameters and a random noise vector into the generator network: in, For the generator of the U-Net structure, Injected into each layer via AdaIN; For the first The noise vector of a frame sampled independently from a standard normal distribution; and These are the image height and width, respectively. For the first Frame RGB facial expression image.

[0094] (5-3) Combat Training

[0095] The training loss for a Generative Adversarial Network (GAN) is: ; in, For PatchGAN structure discriminator, Real facial images For the distribution of real image data, For video frame indexing; Used alone to minimize the discriminator classification loss. The two networks are used separately to minimize the generator adversarial loss and incorporated into the joint training objective, with optimization performed alternately.

[0096] Step (6): Emotional speech synthesis.

[0097] This step will be based on the dialogue response obtained in step (4). Synthesized into emotional speech.

[0098] (6-1) Generation of acoustic condition vectors Combine the dialogue response text features with the sentiment embedding vector obtained in step (3). After concatenation, it is mapped to an acoustic conditional vector: in, Textual features of dialogue responses The emotion embedding vector obtained from step (3) is... This indicates that concatenation is performed along the feature dimension. For the acoustic condition vector dimension, The acoustic mapping weight matrix, This is the acoustic conditional vector. By introducing an emotion embedding vector, the synthesized speech carries emotional prosodic features that match the user's emotional state.

[0099] (6-2) WaveNet waveform generation network dilated convolution ; in, Initialize the causal convolution function. for The input after causal convolution initialization, For the first Hidden layer state, The kernel size is [size]. The dilation rate is [value missing]. The WaveNet waveform generation network consists of 30 layers of dilated convolutions, divided into 3 recurrent blocks, with each block having a dilation rate ranging from [value missing]. arrive Increasing.

[0100] (6-3) Gated activation fusion in, For batch normalized learnable parameters.

[0101] (6-4) Autoregressive speech generation in, For the candidate quantized audio values ​​at the current time step, The conditional probability distribution given by WaveNet For the first Audio samples at each audio time step (μ-law quantized to 256 discrete values). The total number of audio samples contained in the speech waveform; the final output speech waveform. .

[0102] Step (7): Digital human rendering and video compositing.

[0103] This step will use the emotion embedding vector obtained in step (3) The frame-by-frame facial expression image sequence obtained from step (5) The audio waveform obtained from step (6) is integrated and rendered to generate a digital human video.

[0104] (7-1) Modeling of Emotional Conditioned Neural Radiation Field in, In three-dimensional space coordinates, As the perspective direction, For the emotion embedding vector, For position Volume density function value under emotional conditions For position Perspective Color function values ​​under emotional conditions. It consists of 8 fully connected layers with 256 hidden dimensions, uses ReLU activation, and employs sine and cosine encoding for position (10 spatial levels and 4 visual levels).

[0105] (7-2) Volume rendering integral. Defined from the camera origin. Along the direction of view The emitted light ,in The sampling position parameter is along the ray.

[0106] in, and These are the ray parameters corresponding to the near and far planes, respectively. This is a dummy variable in the transmittance integral. Position of light Body density under emotional conditions, The color values ​​are based on the corresponding location, viewpoint, and emotional conditions. Cumulative transmittance For light The corresponding pixel color. The numerical implementation uses a hierarchical sampling strategy to discretize the continuous integral.

[0107] (7-3) Facial region masking fusion and video synthesis

[0108] The digital human image rendered by the neural radiation field is fused with the facial expression image obtained in step (5) using a facial region mask. For the first... A frame represents the pixel color of each ray of light at the corresponding camera pose. Arranged to obtain neural radiation field rendering images Then, a pre-trained face segmentation network is used to generate facial region masks. The facial regions are replaced with expression images generated by a generative adversarial network, while the neural radiation field rendering results are preserved for non-facial regions. in, This is a facial region mask generated based on a face segmentation network, with values ​​ranging from... ; The first generated in step (5) Frames of facial expression images; The color of each light ray in this step The arrangement of the first Frame neural radiation field rendering image; This is element-wise multiplication.

[0109] The fused image sequence The speech waveform obtained from step (6) Perform timing alignment and synchronization, where, The total number of video frames to be generated, as defined in step (5), This represents the total number of audio samples in the speech waveform; the output frame rate is 25fps. If the speech sampling rate is... Then let ,in The audio sampling rate is used to explicitly ensure that the duration of the image sequence is consistent with that of the audio waveform.

[0110] Step (8): Generate result evaluation.

[0111] This step involves a multi-dimensional quality assessment of the generated emotional support digital human video, and the assessment results are used to optimize each generation module.

[0112] (8-1) Emotional Consistency Assessment

[0113] The consistency between the emotions of the generated dialogue, facial expressions, and speech and the user's EEG emotional state is assessed using the emotion consistency reward function in step (4). As a core indicator.

[0114] (8-2) Evaluation of Facial Expression Generation Quality

[0115] The quality and diversity of generated facial expression images were evaluated using the Frechet Inception Distance (FID). A lower FID value indicates that the generated image is closer to the distribution of the real image.

[0116] (8-3) Assessment of speech naturalness

[0117] The naturalness and emotional expressiveness of the synthesized speech were assessed using the Mean Opinion Score (MOS), with human evaluators rating the speech quality on a scale of 1 to 5.

[0118] (8-4) Simultaneous assessment of lip shape

[0119] The lip-sync error distance (LSE-D) was used to evaluate the degree of lip-sync between digital human facial animation and speech. The lower the LSE-D value, the better the lip-sync effect.

[0120] (8-5) Dialogue Quality Assessment

[0121] BERTScore is used to evaluate the semantic similarity between the generated dialogue response and the reference response, serving as a metric for dialogue empathy.

[0122] Based on the above multi-dimensional evaluation results, the verification indicators for each task are constructed, and the balance weights of the corresponding loss terms and the training rounds are adjusted accordingly. Then, the parameters of each generation module are updated by backpropagation according to the loss functions defined in steps (2) to (7), forming a closed-loop optimization mechanism.

[0123] Training strategies and personalized adaptation

[0124] This section describes the cross-modal contrastive learning and meta-learning personalized adaptation strategies used in the system training phase.

[0125] Cross-modal contrastive learning

[0126] First, use the projection head to map the existing step outputs to the same normalized representation space: , Press again Perform time-averaged aggregation of frame-by-frame facial expression parameters, and let... Finally, InfoNCE contrastive learning loss was used to align EEG emotional representations, textual semantic representations, and digital human-generated representations. in, , and Learnable projection heads for EEG, text, and digital human generation, respectively. This is a vector of facial expression parameters aggregated over time. and From the same sample , , Selected from, The set of negative samples is composed of the representations of other samples. Characterize any of the negative samples; For temperature hyperparameters, This refers to the aforementioned cosine similarity function.

[0127] Meta-learning Personalized Adaptation

[0128] The Model-Agnostic Meta-Learning (MAML) method is used to achieve rapid adaptation for individual users. in, To share model parameters, The meta-learning rate, Index for users, For the first 5-20 labeled support samples per user For query sample sets that do not overlap with the supporting sample set, For use of the User-specific parameters obtained through individual user support set adaptation The joint task loss consists of emotion classification, generative adversarial, cross-modal contrast, cross-subject distribution alignment, dialogue optimization, speech reconstruction, neural radiation field pixel reconstruction, and audio-video synchronization loss.

[0129] Meta-learning outer loop optimization: in, The learning rate for the outer loop. The number of users participating in the training. Index for users, For the first The adaptation parameters are obtained from the user support sample set. For the first The system uses a query sample set from individual users. Through meta-learning, it can quickly adapt to a small number of new user samples, enabling personalized emotional support services.

[0130] Joint training losses

[0131] During the system training phase, based on the aforementioned losses, let... For the first A real audio sample, For the first The actual pixel color corresponding to each sampled ray. The number of light rays sampled per batch. For the range of values Audio and video synchronization scoring function; definition , and Constraining emotional speech reconstruction, neural radiation field pixel reconstruction, and audio-video synchronization, respectively.

[0132] in, For the cross-entropy loss of emotion classification in step (2), The generative adversarial loss used only for generator optimization in step (5), For cross-modal contrastive learning loss, This is the alignment term across the participant distribution for the emotion recognition model. For the dialogue optimization loss in step (4), , and These are respectively the aforementioned speech reconstruction, neural radiation field pixel reconstruction, and audio-video synchronization loss; , , , , , and These are the balancing weights for the corresponding loss terms. Discriminator Loss Alternately optimize according to the definition in step (5), without incorporating the generator joint loss.

[0133] Training data and experimental evaluation:

[0134] Training data:

[0135] The emotion recognition module of this invention is trained and evaluated on the following publicly available EEG emotion datasets:

[0136] (1) DEAP dataset: contains 32 EEG acquisition channels, 32 subjects, each subject watched 40 music video clips as emotional stimuli, the original sampling rate was 512Hz, and the labeled dimensions included valence, arousal and dominance.

[0137] (2) SEED dataset: contains 62 EEG acquisition channels, 15 subjects, each subject watched 15 Chinese movie clips as emotional stimuli, the original sampling rate was 200Hz, and the emotions were labeled as positive, neutral and negative.

[0138] (3) DREAMER dataset: contains 14 EEG acquisition channels, 23 subjects, each subject watched 18 video clips as emotional stimuli, the original sampling rate was 128Hz, and the annotation dimensions included valence and arousal.

[0139] The training corpus for the dialogue generation module contains approximately 50,000 emotional dialogue data, covering various scenarios such as psychological counseling, emotional comfort, and daily companionship. Each data entry includes user input, emotional tags, and reference responses.

[0140] Training configuration:

[0141] Model training was performed on dual NVIDIA A100 80GB GPUs using the PyTorch 2.1 deep learning framework. The main hyperparameter settings were as follows: initial learning rate of 1e-4, cosine annealing scheduling; batch size of 32; emotion recognition module trained for 100 epochs, dialogue generation module fine-tuned for 20 epochs; AdamW optimizer used, with a weight decay coefficient of 0.01.

[0142] Evaluation indicators and comparison results:

[0143] The comparison results of the method of this invention with existing representative methods on various evaluation indicators are as follows:

[0144] Summary of the comparison results between the method of this invention and existing methods:

[0145] (1) Emotion recognition accuracy: On the DEAP dataset, the valence classification accuracy of the method of this invention is 89.7% and the arousal classification accuracy is 90.2%, which is more than 7.6 and 6.7 percentage points higher than the baseline method using only time domain features (valence 82.1%, arousal 83.5%) and the method using only frequency domain features (valence 84.3%, arousal 85.1%), respectively; On the SEED dataset, the tri-class classification accuracy is 93.4%, which is better than the DGCNN method (90.4%) and the BiDANN method (92.2%); On the DREAMER dataset, the valence classification accuracy is 87.3% and the arousal classification accuracy is 88.1%.

[0146] (2) Dialogue empathy score (BERTScore): The BERTScore F1 value of the method of this invention is 0.847, which is 6.5 and 3.4 percentage points higher than the baseline of large language model without emotion perception (0.782) and the emotion dialogue method without retrieval enhancement (0.813).

[0147] (3) Expression generation FID: The FID value of the method of the present invention is 18.6, which is better than the GAN method without emotion conditions (FID=32.4) and the expression generation method based on VAE (FID=27.1), indicating that the generated expression images are of higher quality and closer to the real distribution.

[0148] (4) Speech MOS score: The MOS score of the method of this invention is 4.12 (out of 5), which is better than the baseline WaveNet waveform generation network without emotion conditions (3.67) and the Tacotron2-based method (3.89), verifying the effect of emotion embedding vector injection on improving the expressiveness of speech emotion.

[0149] (5) Lip synchronization LSE-D: The LSE-D value of the method of the present invention is 7.82, which is better than the method without facial region mask fusion (LSE-D=9.45) and the lip synchronization method based on 2D deformation (LSE-D=8.91), demonstrating that the facial region mask fusion strategy improves the lip synchronization quality.

[0150] The experimental results above demonstrate that the method of this invention outperforms existing methods in terms of emotion recognition accuracy, dialogue empathy, facial expression generation quality, speech naturalness, and lip-sync, verifying the effectiveness of the proposed time-frequency-spatial three-dimensional feature fusion, gated multimodal fusion, emotion-conditional injection speech synthesis, and facial region masking fusion techniques.

Claims

1. A method for generating emotional digital humans based on brain-computer interfaces and large language models, characterized in that, The method includes the following steps: (1) The collected raw EEG signals from users were preprocessed by downsampling, baseline correction and independent component analysis artifact removal to obtain clean EEG signals; (2) Extract time-frequency-space three-dimensional features from the clean EEG signal obtained in step (1), and input it into the emotion recognition model for emotion classification after fusion through the cross-attention mechanism to obtain the emotion probability distribution vector; (3) Gated multimodal fusion is performed between the emotion embedding corresponding to the emotion probability distribution vector obtained in step (2) and the semantic features of the user dialogue text to obtain the emotion embedding vector and multimodal fusion features; (4) Based on the multimodal fusion features obtained in step (3), relevant documents are retrieved from the emotional care knowledge base using the retrieval enhancement generation framework, enhanced prompts are constructed in combination with the dialogue history, and emotional care dialogue responses are generated through the large language model. (5) Based on the emotion embedding vector obtained in step (3), generate expression parameters through multilayer perceptron mapping, and generate digital human facial expression images using generative adversarial network; (6) Extract text features from the dialogue response obtained in step (4), concatenate the text features with the emotion embedding vector obtained in step (3) and map them into an acoustic conditional vector, and perform emotional speech synthesis through the conditional WaveNet waveform generation network to obtain a speech waveform; (7) The emotional embedding vector obtained in step (3) is used as the conditional input emotional conditional neural radiation field. Combined with the facial expression image obtained in step (5) and the speech waveform obtained in step (6), digital human rendering and video synthesis are performed through facial region mask fusion to obtain the emotional companion digital human video. (8) Perform multi-dimensional quality assessment on the emotional companionship digital human video obtained in step (7), including emotional consistency assessment, facial expression generation quality assessment, speech naturalness assessment and lip-sync assessment, and optimize each generation module based on the assessment results.

2. The method according to claim 1, characterized in that, Step (1) specifically includes: raw EEG signals Downsampling to 128Hz yields the downsampled signal. ,in, Number of brainwave channels, This represents the original number of time sampling points. This represents the number of time sampling points after downsampling. This indicates a downsampling operation with a target sampling rate of 128Hz; Baseline correction is performed on the downsampled signal: in, This is the baseline-corrected EEG signal. This is the set of sampling point indices corresponding to the baseline time period. This represents the number of baseline sampling points contained in the set. For downsampled signals at the index The value corresponding to the sampling point; Independent component analysis was performed on the baseline-corrected signal. After identifying and removing artifacts, clean EEG signals were reconstructed. : in, It is a mixed matrix. For independent component matrices, and These are the mixture matrix and the independent component matrix after removing artifacts, respectively.

3. The method according to claim 2, characterized in that, The time-frequency-space three-dimensional feature extraction described in step (2) specifically includes: The clean EEG signals obtained in step (1) Extracting time-frequency features using continuous wavelet transform : in, The frequency bands include five bands: the delta band (1-4Hz), the theta band (4-8Hz), the alpha band (8-14Hz), the beta band (14-30Hz), and the gamma band (30-45Hz). Indicates continuous wavelet transform operation; Constructing a spatiotemporal topology map of brainwaves : This represents the operation of constructing a spatiotemporal topology map based on the correlation between brainwave channels; Extracting time-frequency feature vectors using convolutional neural networks (CNNs) Spatial feature vectors are extracted using graph convolutional networks (GCN). ,in This is an adjacency matrix constructed based on electrode locations; The time-frequency feature vector and spatial feature vector are fused using a cross-attention mechanism: in, , , These are the query vector, key vector, and value vector, respectively. The fused feature vector , , These are the query, key, and value projection matrices, respectively. The dimension of the key vector. Indicates matrix transpose. This represents the normalized exponential function; The emotion classification described in step (2) is based on fusion features: in, This is the classification weight matrix. For bias vectors, Let the vector be the probability distribution vector of the emotion. The emotion classification uses cross-entropy loss. Conduct training: in, The number of emotion categories for the current task. For emotion category variables, For the first True labels for emotions Represents the natural logarithm function. The model predicts the first Emotional probability, subscript This only represents the index of emotion categories.

4. The method according to claim 3, characterized in that, In step (2), the emotion recognition model aligns the feature distributions of the labeled subject sample set and the new subject sample set during its training phase using the maximum mean difference, thereby achieving cross-subject transfer learning. in, and These are the fusion feature sets calculated by the emotion recognition model in step (2) from the source and target domain EEG samples, respectively. and These are the feature vectors obtained by fusing time-frequency features, spatial features, and cross-attention for the corresponding samples. and For the number of samples in both domains, For the regenerating kernel Hilbert space mapping function, and This is only the summation index for the corresponding sample set. This represents the square norm 2.

5. The method according to claim 4, characterized in that, In step (3), the gated multimodal fusion specifically includes embedding and encoding the highest probability category in the emotion probability distribution vector obtained in step (2) to obtain the emotion embedding vector. : in, This is the sentiment category embedding function, where argmax represents the operation of taking the category index corresponding to the maximum probability. The emotion probability distribution vector obtained from step (2); User conversation text Perform semantic encoding to obtain text feature vectors : in, For the encoder of the BERT pre-trained language model, The dialogue text entered by the user. For text feature dimensions; To ensure that the text features and sentiment embedding vectors have the same dimension, first... The text features are projected into the sentiment embedding space, and then the sentiment embedding vector and the aligned text feature vector are fused through a gating mechanism: in, For the emotion embedding vector, and These are the text projection matrix and the text projection bias, respectively. Text features after dimension alignment; For the gated weight matrix, This is the gated bias vector. It is a sigmoid activation function. This represents a vector concatenation operation. This represents element-wise multiplication. For multimodal fusion features, This represents the gating vector.

6. The method according to claim 5, characterized in that, Step (4) specifically includes: The multimodal fusion features obtained from step (3) The corresponding emotional state text tag is "query," which is retrieved from the emotional support knowledge base. Related documents: in, The emotional state text labels corresponding to the multimodal fusion features. This represents the collection of relevant documents retrieved. This indicates an operation to retrieve relevant documents from the emotional support knowledge base. This represents a knowledge base for emotional support and companionship. Indicates the number of documents retrieved; The sentiment state text labels corresponding to the multimodal fusion features obtained in step (3), the text content of the retrieved document, and the multi-turn dialogue history are organized into textualized enhanced prompts according to a preset prompt template: in, Enhanced hints that indicate textual representation This indicates the preset prompt template function. For the most recent A dialogue with history, and The first Round-by-round user text and system response, round index ; Generate dialogue responses using a large language model : Evaluate and optimize dialogue quality using an affective consistency reward function: in, This represents the value of the affective consistency reward function. The current sentiment label of the user is obtained from the sentiment probability distribution in step (2). To utilize text sentiment classifiers Sentiment tags obtained from response text Emotional support guidance function Guide emotions based on the goals determined by current sentiment tags and conversation history. The cosine similarity function is used. For smoothness evaluation function; For the current large language model, enhance prompts The probability distribution of the word-by-word meta-generation strategy directly output below. For the frozen reference large language model in the same The probability distribution of the word-by-word meta-reference strategy output below. Let KL divergence be the two. , , To balance the weighting coefficients, the dialogue optimization loss is defined as follows: This allows the reward to be explicitly converted into a minimizeable training loss.

7. The method according to claim 6, characterized in that, In step (5), generating digital human facial expression images using a generative adversarial network specifically includes: The emotion embedding vector obtained from step (3) Mapped along the video timeline as a frame-by-frame emotion embedding sequence. Let... The total number of frames in the video to be generated, the first... The normalized time position of the frame is ,but Then Mapped to the first layer perceptron via a multilayer perceptron Frame expression parameters : in, For video frame indexing, This represents the total number of frames in the video to be generated. For the first Normalized time position of the frame. A time-mapping network for frame-by-frame modulation of emotion embeddings based on temporal location. For the first Frame emotion embedding, For mapping facial expression parameters to a network; The first Frame expression parameters With the Frame random noise Input generator, generate the first Frame of facial expression images: in, To generate a generator for adversarial networks, For the first The noise vector of a frame sampled independently from a standard normal distribution. For the first Frames of facial expression images; Through the discriminator Judge the first Frame generation image authenticity is assessed, and adversarial training loss is used to optimize generation quality: ； in, For the discriminator network, Real facial images For the distribution of real image data, The loss is used separately to optimize the discriminator. To optimize the generator separately and incorporate the generative adversarial loss into the joint training objective, This represents the mathematical expectation operation. This is the index for the aforementioned video frames.

8. The method according to claim 7, characterized in that, In step (6), the emotional speech synthesis specifically includes synthesizing the dialogue response obtained in step (4). The corresponding text features are concatenated with the sentiment embedding vector obtained in step (3) and mapped to an acoustic conditional vector. : in, Textual features of dialogue responses The emotion embedding vector obtained from step (3) is... This represents the concatenation of text features and sentiment embedding vectors. This is the acoustic mapping weight matrix; First press The acoustic conditional vector is initialized as the first-layer input through causal convolution, and then processed layer by layer through the dilated convolutional layers of the WaveNet waveform generation network: Conditional fusion using gating activation units: Generate autoregressive speech waveforms: in, , Initialize the causal convolution function; For the first Hidden layer state, To expand the convolutional layer index, The kernel size is [size]. For the first Layer expansion rate, For batch normalization operations, For normalization parameters, For the candidate quantized audio values ​​at the current time step, The conditional probability distribution given by WaveNet For the first Audio samples generated at each audio time step This indicates the dilated convolution operation. For the first Layer-gated activation output, It is a sigmoid activation function. The hyperbolic tangent activation function is used. This is element-wise multiplication.

9. The method according to claim 8, characterized in that, In step (7), the digital human rendering and video synthesis specifically includes implicit modeling of the three-dimensional space using the Neural Radiance Field (NeRF): in, In three-dimensional space coordinates, As the perspective direction, The emotion embedding vector obtained from step (3) is... For position Volume density function value under emotional conditions For position Perspective Color function values ​​under emotional conditions, This represents a multilayer sensor network representing a neural radiation field. Defined from the camera origin Along the direction of view The emitted light ,in The sampling position parameters are along the ray; pixel colors are synthesized through volume rendering integration. : in, To start from the camera origin Along direction The emitted light, and These are the ray parameters corresponding to the near and far planes, respectively. This is a dummy variable in the transmittance integral. Position of light Body density under emotional conditions, The color values ​​are based on the corresponding location, viewpoint, and emotional conditions. Cumulative transmittance For light The corresponding pixel color; The facial expression image obtained in step (5) is fused with the neural radiation field rendered image using a facial region mask: in, This is a facial region mask generated based on a face segmentation network. The first generated in step (5) Frame of facial expression images, To make the first The colors of each ray corresponding to the camera pose The resulting neural radiation field rendering image. For element-wise multiplication, Indicates the first The final image after frame fusion; The fused image sequence is then combined with the speech waveform obtained in step (6). Perform time-synchronized synthesis to output a digital human video providing emotional support; In the training phase of each neural network model in steps (2) to (7), the method also employs a cross-modal contrastive learning strategy to align the representation spaces of EEG emotional representation, text semantic representation, and digital human generation representation, and uses a meta-learning adaptation strategy to achieve rapid adaptation for individual users, specifically including: The InfoNCE contrastive learning loss is used to align the representation spaces of each modality: in, , First press Perform time-averaged aggregation of the facial expression parameters frame by frame, and then let The three are the normalized representations obtained by passing the existing outputs from steps (2), (3), and (5) to (6) through the corresponding projection heads. This is a vector of facial expression parameters aggregated over time. and Selected from the above different modal representations of the same sample, The set of negative samples is composed of the representations of other samples. Characterize any one of the negative samples. For temperature hyperparameters, The aforementioned cosine similarity function; Using Model-Agnostic Meta-Learning (MAML) to achieve rapid adaptation for individual users: ； in, To share model parameters, The learning rate for the inner loop. The learning rate of the outer loop element. Provide a small set of labeled support samples for specific users. For a query sample set that does not overlap with the supporting sample set, The parameters are obtained by adapting from the supporting sample set. and These are the models before and after adaptation, The joint task loss function is composed of emotion classification, generative adversarial, cross-modal contrast, cross-subject distribution alignment, dialogue optimization, speech reconstruction, neural radiation field pixel reconstruction, and audio-video synchronization loss.

10. A digital human generation system for emotional companionship based on brain-computer interface and large language model collaboration, characterized in that, The system is used to perform the method according to any one of claims 1 to 9, comprising: The EEG signal acquisition and preprocessing module is used to downsample, baseline correct, and remove artifacts through independent component analysis of the user's raw EEG signal to obtain a clean EEG signal. The emotion recognition module is used to extract time-frequency-spatial three-dimensional features and perform cross-attention fusion on clean EEG signals, and output an emotion probability distribution vector. The multimodal feature fusion module is used to fuse the sentiment embedding vector and text semantic features in a multimodal manner through a gating mechanism, and output multimodal fused features; The retrieval-enhanced emotional dialogue generation module is used to retrieve relevant documents from the emotional companionship knowledge base based on multimodal fusion features and generate emotional companionship dialogue responses through a large language model. The facial expression generation module is used to generate digital human facial expression images based on emotion embedding vectors using a generative adversarial network. The speech synthesis module is used to concatenate dialogue response text features with emotion embedding vectors and then synthesize emotional speech waveforms through a conditional WaveNet waveform generation network. The digital human rendering and video synthesis module is used to perform 3D rendering using emotional conditional neural radiation fields, fuse facial expression images and rendered images through facial region masking, and synthesize speech waveforms in a time-synchronous manner to output emotional companionship digital human videos. The generation result evaluation module is used to perform multi-dimensional quality evaluation on the generated digital human videos and feed the evaluation results back to each generation module for optimization.

Images