Dynamic Listener Facial Expression Response Model Based on Dialogue Emotion

Abstract

This invention relates to the field of computer vision technology for emotional interaction, specifically a dynamic listener facial expression response model based on dialogue emotion. The model includes: data alignment, feature learning, probabilistic modeling, and sequence generation units. It acquires a speaker-listener video pair dataset, extracts dialogue text and listener facial video frames, and generates corresponding emotion label sequences through fine-grained emotion analysis and frame-by-frame expression recognition. Timeline alignment is then completed to form synchronously labeled data. A variational quantization autoencoder architecture is used to learn facial video frame features, constructing a high-dimensional emotion-visual discrete latent representation space. The model models the dynamic conditional probability distribution of speaker emotion and listener response, generating a temporally continuous and smoothly transitioning sequence of listener facial expression responses based on new dialogue text and emotion sequences. This model achieves accurate temporal matching between emotion and facial response, optimizing the naturalness and coherence of expression generation.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology for emotional interaction, and more particularly to a dynamic listener facial expression response model based on dialogue emotion. Background Technology

[0002] Current technologies for generating listener facial expressions in dialogue-based emotional interaction scenarios mostly extract the overall emotional features of the dialogue text directly, match them with preset static facial expression templates, perform coarse-grained emotion judgment on the dialogue text, and perform single-frame independent expression recognition on the listener's facial video. They do not carry out synchronized data processing for speaker-listener video pairs, nor do they achieve temporal correlation processing between text emotion and facial expression.

[0003] Existing technologies cannot establish a temporal correspondence between the speaker's emotional output and the listener's facial reactions. They lack accurate synchronously labeled data to support the modeling of the relationship between emotion and visual features. Using conventional feature mapping methods to process listener facial video frames cannot form a high-dimensional feature representation space that adapts to emotional interaction. It is difficult to depict the dynamic relationship between the speaker's contextual emotions and the listener's facial reactions. The listener facial expression sequences generated during the inference stage have problems such as temporal breaks and abrupt transitions.

[0004] It is necessary to achieve precise alignment of fine-grained sentiment labels in the dialogue text with frame-by-frame sentiment labels on the listener's face on the timeline, forming synchronously labeled data that matches the speaker's emotions with the listener's responses. This requires the use of a variational quantization autoencoder architecture to compress and learn features from the listener's facial video frames, construct a high-dimensional discrete latent representation space of emotion-visual features, and establish a dynamic conditional probability distribution relationship between the speaker's contextual emotion features and the listener's facial response features within this space, thereby enabling the continuous generation of the listener's facial expressions driven by new dialogue text. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a dynamic listener facial expression response model based on dialogue emotion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a dynamic listener facial expression response model based on dialogue emotion, comprising: The data alignment unit acquires a training dataset containing speaker-listener video pairs, extracts corresponding dialogue text data and listener facial video frame sequences from each video pair in the training dataset, performs fine-grained sentiment analysis on the dialogue text data to generate a text sentiment tag sequence corresponding to each semantic unit in the dialogue text, performs frame-by-frame facial expression recognition on the listener facial video frame sequence to generate a facial sentiment tag sequence corresponding to each frame of facial image, and aligns the text sentiment tag sequence and the facial sentiment tag sequence on the time axis to form synchronous labeled data of speaker emotion and listener response. The feature learning unit, based on the synchronously labeled data of speaker emotion and listener response, uses a variational quantization autoencoder architecture to compress and learn features from the listener's facial video frame sequence, and constructs a high-dimensional discrete latent representation space of emotion-visual features. The probabilistic modeling unit models the dynamic conditional probability distribution relationship between the speaker's contextual emotional features and the listener's facial reaction features within the discrete latent representation space. The sequence generation unit, based on the dynamic conditional probability distribution relationship obtained from modeling, receives new speaker dialogue text and its emotional sequence during the inference stage, and predicts and generates a corresponding temporally continuous and smoothly transitioning sequence of listener facial expression responses.

[0007] As a further aspect of the present invention, fine-grained sentiment analysis is performed on the dialogue text data to generate a text sentiment tag sequence corresponding to each semantic unit in the dialogue text, including: The dialogue text data is encoded using a pre-trained language model to obtain a context feature vector representation of the text; The contextual feature vector representation of the text is input into the sentiment classification layer to generate preliminary text sentiment labels at the sentence level. Based on a predefined sentiment category dictionary and semantic dependency relationships, the dialogue text data is segmented and semantic role labeled to identify each independent semantic unit. Attention is fused between the vector representation of each independent semantic unit and the sentence-level text sentiment label to calculate the contribution weight of each independent semantic unit to the overall sentence sentiment. Based on the contribution weights, sentence-level text sentiment tags are redistributed and fine-tuned to generate text sentiment tags that correspond one-to-one with each independent semantic unit and have confidence levels. These tags are then arranged in the order in which the semantic units appear in the text to form a text sentiment tag sequence.

[0008] As a further aspect of the present invention, facial expression recognition is performed frame-by-frame on the listener's facial video frame sequence to generate a facial emotion tag sequence corresponding to each frame of the facial image, including: A pre-trained facial landmark detection model is used to process each frame of the listener's facial video frame sequence to locate the coordinates of the facial landmarks. Based on the coordinates of facial key points, each frame of facial image is normalized and aligned to obtain a standardized facial region image. The standardized facial region image is input into a pre-trained facial expression recognition neural network; The facial expression recognition neural network extracts facial visual features and outputs a facial emotion probability distribution vector. Non-maximum suppression is applied to the facial emotion probability distribution vector, and the emotion category with the highest probability value is selected as the facial emotion label for each frame of facial image. Arrange all facial emotion tags in chronological order of video frames to form a facial emotion tag sequence.

[0009] As a further aspect of the present invention, the text emotion tag sequence and the facial emotion tag sequence are aligned along the time axis to form synchronously labeled data of speaker emotion and listener response, including: Obtain the audio stream corresponding to the dialogue text data, or extract the corresponding speaker audio from the video pair; Speech activity detection is performed on the audio stream to determine the start and end timestamps of each semantic unit on the timeline; Based on the start and end timestamps of each semantic unit, its corresponding text sentiment tags are mapped onto a continuous timeline, forming a density distribution of text sentiment tags over time. For each frame in the facial emotion tag sequence, whose timestamp is known, find all text emotion tags within a preset time window that are closest to the timestamp of each frame in the temporal density distribution of the text emotion tags. The weighted fusion result of all text sentiment tags within the preset time window is calculated and used as the speaker context sentiment tag aligned with each frame of facial image; The aligned speaker contextual emotion tags are combined with the facial emotion tags of each frame to form a synchronized emotion pair; All frames in a video pair and their corresponding synchronized emotion pairs are collected in chronological order to form synchronized labeled data of speaker emotion and listener response.

[0010] As a further aspect of the present invention, based on the synchronously labeled data of speaker emotion and listener response, a variational quantization autoencoder architecture is used to compress and learn features from the listener's facial video frame sequence to construct a high-dimensional discrete latent representation space of emotion-visual features, including: Frame images of the listener's facial video frame sequence are extracted as input from the synchronously labeled data of speaker's emotion and listener's response; The frame image is encoded into a continuous high-dimensional feature vector using an encoder network; A correlation is established between the high-dimensional feature vector output by the encoder network and the speaker context sentiment tags in the synchronously labeled data, and the sentiment tag information is embedded into the encoding process of the high-dimensional feature vector. A learnable discrete codebook is introduced, which consists of multiple codebook vectors; Calculate the similarity between the high-dimensional feature vector and each codebook vector in the discrete codebook, and select the index of the codebook vector with the highest similarity as the quantization result; The selected codebook vector is used as the quantized representation of the high-dimensional feature vector, i.e., the discrete latent representation; Using a decoder network, the listener's facial image is reconstructed from the discrete latent representation as input; The encoder network, discrete codebook, and decoder network are trained together by reconstruction loss, quantization loss, and sentiment embedding consistency loss, and finally a high-dimensional sentiment-visual feature discrete latent representation space that integrates sentiment information is constructed.

[0011] As a further aspect of the present invention, within the discrete latent representation space, modeling the dynamic conditional probability distribution relationship between the speaker's contextual emotional features and the listener's facial reaction features includes: During the training phase, synchronous emotion pairs from multiple consecutive time steps are extracted in chronological order from the synchronously labeled data of speaker emotion and listener response. For each time step, a discrete latent representation corresponding to the listener's facial image at the current time step is obtained from the discrete latent representation space as a listener response feature; The speaker contextual sentiment labels for the current time step and several previous time steps are obtained from the synchronous sentiment pairs to form a speaker sentiment context within a time window. The speaker's emotional context within the time window is encoded into a fixed-dimensional emotional context vector; Using the listener response features of the previous time step and the emotional context vector of the current time step as conditions, a conditional probability model is established to predict the listener response features of the current time step. The conditional probability model is implemented through a recurrent neural network, whose hidden state integrates historical listener response characteristics and emotional context information. The conditional probability model is trained by maximizing the log-likelihood of the conditional probability at all time steps on the synchronously labeled data of the speaker's emotion and the listener's response, thereby modeling the dynamic conditional probability distribution relationship.

[0012] As a further aspect of the present invention, the step of receiving new speaker dialogue text and its emotional sequence during the inference stage based on the dynamic conditional probability distribution relationship obtained by modeling, and predicting and generating a corresponding temporally continuous and smoothly transitioning sequence of listener facial expression responses, includes: During the reasoning phase, new speaker dialogue text is input; Fine-grained sentiment analysis is performed on the new speaker dialogue text to generate a corresponding text sentiment tag sequence; The text sentiment tag sequence is mapped onto a continuous time axis, and speaker context sentiment tags for each time step in the inference phase are generated at the same time resolution as the training phase. Initialize an empty listener response feature sequence and set the listener response features at the initial time step to a preset discrete latent representation of a neutral expression; Starting from the initial time step, prediction is performed sequentially for each time step: obtain the speaker context sentiment labels for the current time step and several previous time steps to form a sentiment context vector; The listener response features predicted at the previous time step and the emotional context vector at the current time step are input into the modeled dynamic conditional probability distribution relationship. From the dynamic conditional probability distribution relationship, sample or take the maximum probability value to obtain the listener response feature predicted at the current time step, which is a codebook index in the discrete latent representation space; The codebook vector corresponding to the obtained codebook index is used as the listener response feature predicted at the current time step and added to the listener response feature sequence. The execution steps are iterated until all time steps are processed to obtain a complete sequence of predicted listener response features.

[0013] As a further aspect of the present invention, after obtaining the complete predictive listener response feature sequence, it further includes: The facial image generation unit inputs each listener response feature in the predicted listener response feature sequence, i.e. the codebook vector corresponding to each codebook index, into the trained decoder network. The decoder network decodes each codebook vector and reconstructs it into a frame of the listener's facial image; All the reconstructed facial images of the listeners are combined in their corresponding time sequence to generate a temporally continuous video sequence of facial expressions. The generated facial expression video sequence is post-processed with temporal smoothing to ensure that the expression changes between adjacent frames are visually smooth and to eliminate jitter or abrupt changes.

[0014] As a further aspect of the present invention, the conditional probability model is implemented through a recurrent neural network, whose hidden state integrates historical listener response features and emotional context information, specifically including: The hidden state of the recurrent neural network is used to represent the fusion information of the listener's facial expression response history and emotional context history up to the current time step. At each time step, the input of the recurrent neural network is the embedded representation of the codebook vector corresponding to the listener response features of the previous time step, concatenated with the emotional context vector of the current time step. The recurrent neural network updates the hidden state at the current time step based on the current input and the hidden state at the previous time step. In the output layer, the hidden state at the current time step is mapped to the log probability distribution of all codebook vectors in the discrete latent representation space through a fully connected layer; Sample from the logarithmic probability distribution, or select the codebook index with the highest logarithmic probability, as the listener response feature predicted at the current time step.

[0015] As a further aspect of the present invention, the reconstruction loss used in the training process of the variational quantization autoencoder architecture is the pixel-level mean square error loss between the original facial image and the reconstructed facial image. Quantization loss is the mean square error loss between the high-dimensional feature vector output by the encoder and the selected codebook vector; The sentiment embedding consistency loss is the cross-entropy loss between the aligned speaker context sentiment label and the sentiment category prediction decoded from the quantized discrete latent representation.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: Fine-grained sentiment analysis is performed on the dialogue text to generate a text sentiment tag sequence corresponding to each semantic unit. Frame-by-frame facial expression recognition is performed on the listener's facial video frame sequence to generate a facial sentiment tag sequence. The text sentiment tag sequence and the facial sentiment tag sequence are aligned on the timeline to form synchronously labeled data of speaker's emotion and listener's response. This allows for an accurate presentation of the temporal correspondence between the speaker's emotional output and the listener's facial response. The matching dimension of text sentiment and facial visual emotion aligns with the actual temporal logic of the dialogue interaction, thus optimizing the temporal correspondence accuracy and sentiment matching fit of the sentiment tags.

[0017] Based on the variational quantization autoencoder architecture, compression and feature learning are performed on the listener's facial video frame sequence. A high-dimensional discrete latent representation space of emotion-visual features is constructed. Within this discrete latent representation space, the dynamic conditional probability distribution relationship between the speaker's contextual emotion features and the listener's facial reaction features is modeled. The discrete latent representation space can regularize the distribution pattern of high-dimensional emotion-visual features. The variational quantization autoencoder architecture can complete the efficient compression and targeted learning of facial video frame features. The dynamic conditional probability distribution relationship can fully characterize the dynamic correlation logic between the speaker's emotional changes and the listener's facial reactions. The listener's facial expression reaction sequence generated during the inference stage can achieve continuous temporal transition, and the smoothness and naturalness of expression changes are optimized. Attached Figure Description

[0018] Figure 1 This is a timing diagram of the dynamic listener facial expression response model based on dialogue emotion described in this invention; Figure 2 A flowchart for generating facial emotion label sequences for frame-by-frame facial expression recognition; Figure 3 A graph showing the relationship between discrete codebook size K and model reconstruction quality and training speed; Figure 4 Training loss convergence curves for recurrent neural network models; Figure 5 A heatmap showing the intensity of the relationship between speaker's emotion and listener's response. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0021] See Figure 1 This invention provides a dynamic listener facial expression response model based on dialogue emotion, and the specific implementation is as follows: The model comprises four core units: a data alignment unit, a feature learning unit, a probabilistic modeling unit, and a sequence generation unit. The data alignment unit acquires a training dataset containing speaker-listener video pairs and extracts dialogue text data and listener facial video frame sequences. It generates text sentiment label sequences through fine-grained sentiment analysis of the text and generates facial sentiment label sequences through frame-by-frame facial expression recognition of the listener video frames. Finally, these two sequences are aligned on the timeline to form synchronously labeled data representing speaker emotion and listener response. The feature learning unit, based on this synchronously labeled data, uses a variational quantization autoencoder architecture to compress and learn features from the listener facial video frames, constructing a high-dimensional discrete latent representation space that integrates emotional information and visual features. The probabilistic modeling unit models the dynamic conditional probability distribution relationship between speaker contextual emotional features and listener facial response features within this discrete latent representation space. Based on this dynamic conditional probability distribution relationship, the sequence generation unit receives new speaker dialogue text and its emotional sequence during the inference phase and predicts and generates corresponding temporally continuous and smoothly transitioning listener facial expression response sequences.

[0022] In one embodiment of the present invention, fine-grained sentiment analysis is performed on the dialogue text data to generate a text sentiment tag sequence corresponding to each semantic unit in the dialogue text. See also... Figure 2This process employs a pre-trained language model to encode the dialogue text data, obtaining a contextual feature vector representation of the text. This contextual feature vector representation is then input into a sentiment classification layer to generate preliminary text sentiment labels at the sentence level. Based on a predefined sentiment category dictionary and semantic dependency relationships, the dialogue text data is segmented and semantically labeled, identifying each independent semantic unit. The vector representation of each independent semantic unit is then fused with the sentence-level text sentiment labels through attention fusion. The contribution weight of each independent semantic unit to the overall sentence sentiment is calculated, and the sentence-level text sentiment labels are redistributed and fine-tuned based on this contribution weight, generating text sentiment labels that correspond one-to-one with each independent semantic unit and have confidence levels. These labels are arranged according to the order of their appearance in the text, forming a text sentiment label sequence. Finally, facial expression recognition is performed frame-by-frame on the listener's facial video frame sequence, generating a facial sentiment label sequence corresponding to each frame of the facial image. This process employs a pre-trained facial landmark detection model to process each frame of the listener's facial video sequence, locating the coordinates of facial landmarks. Based on these coordinates, each frame is normalized and aligned to obtain a standardized facial region image. This standardized facial region image is then input into a pre-trained facial expression recognition neural network. This network extracts facial visual features and outputs a facial emotion probability distribution vector. Non-maximum suppression is applied to this vector, and the emotion category with the highest probability value is selected as the facial emotion label for each frame. All facial emotion labels are then arranged in chronological order according to the video frames to form a facial emotion label sequence.

[0023] In specific implementations, the dynamic listener facial expression response model based on dialogue sentiment involves fine-grained sentiment analysis of dialogue text data to generate text sentiment label sequences, and frame-by-frame facial expression recognition of listener facial video frames to generate facial sentiment label sequences. In the text sentiment analysis implementation, a pre-trained language model is used to encode the input dialogue text data. The encoding process transforms the text into a high-dimensional feature vector representation containing contextual semantic information. These feature vectors capture the dependencies between words and the global context. The obtained text context feature vector representation is input into a specially designed sentiment classification layer, typically a fully connected neural network layer whose output dimension matches the preset number of sentiment categories. The sentiment classification layer performs sentiment classification on the overall text content at the sentence level, generating a preliminary text sentiment label. This preliminary label reflects the dominant sentiment tendency of the entire sentence. In some embodiments, based on a predefined sentiment category dictionary and the semantic dependencies of the text, further word segmentation and semantic role labeling are performed on the dialogue text data to identify each independent semantic unit constituting the sentence. An independent semantic unit can be a phrase or word with complete sentiment meaning. It is understandable that the vector representation of each independent semantic unit is obtained by truncating or recompiling the output of the intermediate layer of the language model. An attention fusion operation is then performed between the vector representation of each independent semantic unit and the previously generated text sentiment tags at the sentence level. The attention fusion mechanism calculates the correlation score between the feature vector of each independent semantic unit and the embedding vector of the sentence-level sentiment tag, using this as the contribution weight of each independent semantic unit to the overall sentence sentiment. Based on the calculated contribution weights, the sentiment intensity represented by the sentence-level text sentiment tags is redistributed and fine-tuned among multiple independent semantic units, generating fine-grained text sentiment tags that correspond one-to-one with each independent semantic unit and are accompanied by a confidence score. All generated fine-grained text sentiment tags are arranged strictly according to the order of appearance of their corresponding independent semantic units in the original dialogue text data, ultimately forming an ordered sequence of text sentiment tags.

[0024] In the implementation of facial expression recognition, the processing object is a sequence of video frames of the listener's face. A pre-trained facial keypoint detection model is applied to each frame of the sequence. This model can accurately locate the pixel coordinates of key facial feature points such as the corners of the eyes, mouth, and nose. Based on the coordinates of the detected facial keypoints, geometric normalization and spatial alignment are performed on each frame of the original facial image. The processing includes rotation, scaling, and translation, ensuring that the facial regions in different frames are at the same position and scale in the image, thus obtaining a standardized facial region image. This step eliminates interference caused by changes in head pose and distance from the camera. The standardized facial region image is then input into a pre-trained facial expression recognition neural network. This neural network is a deep convolutional neural network whose structure is used to efficiently extract visual features of facial regions. The last layer of the network is usually a softmax classifier, which outputs a multi-dimensional facial emotion probability distribution vector. Each element in the vector corresponds to a probability value of a preset basic emotion category. Non-maximum suppression (NMS) is applied to the facial emotion probability distribution vector output by the facial expression recognition neural network. NMS compares the probability values ​​of each category in the probability distribution vector and selects the emotion category with the highest probability value as the final facial emotion label for the current frame's facial image. All facial emotion labels identified frame by frame are arranged according to the original temporal order of the video frames, forming a facial emotion label sequence that strictly corresponds to the video stream's timeline. In some embodiments, the pre-trained facial expression recognition neural network is trained on a large-scale facial expression dataset and is capable of recognizing multiple basic emotions, including happiness, sadness, surprise, anger, disgust, fear, and neutrality. Optionally, the facial keypoint detection model and the facial expression recognition neural network can be jointly trained end-to-end to optimize the synergy of feature extraction. It can be understood that the generated text emotion label sequence and facial emotion label sequence provide the foundational data for the subsequent time alignment step, and the labels of both sequences are based on the same or a mappable emotion category system.

[0025] In one embodiment of the present invention, the text sentiment tag sequence and the facial sentiment tag sequence are aligned on the time axis to form synchronous annotation data of speaker emotion and listener response. This process first acquires the audio stream corresponding to the dialogue text data, or extracts the corresponding speaker audio from the video pair, performs speech activity detection on the audio stream, and determines the start and end timestamps of each semantic unit on the time axis. Based on the start and end timestamps of each semantic unit, its corresponding text sentiment tag is mapped onto a continuous time axis, forming a temporal density distribution of text sentiment tags. For each frame in the facial sentiment tag sequence, whose timestamp is known, all text sentiment tags within a preset time window closest to the timestamp of each frame are found in the temporal density distribution of text sentiment tags. The weighted fusion result of all text sentiment tags within the preset time window is calculated as the speaker context sentiment tag aligned with each frame's facial image. The aligned speaker context sentiment tag is combined with the facial sentiment tag of each frame to form a synchronous sentiment pair. All frames in the video pair and their corresponding synchronous sentiment pairs are collected in chronological order to form synchronous annotation data of speaker emotion and listener response.

[0026] In practice, the text sentiment tag sequence and the facial sentiment tag sequence are aligned on the timeline to form synchronized annotation data of speaker emotion and listener response. The specific implementation relies on bridging audio time information. An audio stream corresponding to the dialogue text data is acquired, or the corresponding speaker audio is extracted from the original speaker-listener video pair. The speaker audio and the listener facial video frame sequence in the video pair are inherently synchronized on the timeline. Speech activity detection is performed on the acquired speaker audio. The speech activity detection algorithm identifies segments of speech activity in the audio stream and determines the precise start and end timestamps of each semantic unit on the timeline. The start and end timestamps of a semantic unit define the time interval of the emotion expressed by that unit. Based on the start and end timestamps of each semantic unit, its corresponding fine-grained text sentiment tag is mapped onto a continuous timeline. The mapping method assigns a time interval attribute to each text sentiment tag; within the time interval, the sentiment tag is considered valid, thus forming a density distribution of text sentiment tags on a continuous timeline. For each frame of facial image in the facial emotion tag sequence, its corresponding timestamp can be directly calculated from the video's frame rate information and is known. Within the established temporal density distribution of text emotion tags, all text emotion tags within a preset time window are searched, centered on the timestamp of the current facial image frame. The length of the preset time window is a configurable hyperparameter that covers a reasonable range where facial responses may lag behind speech stimuli. The weighted fusion result of all text emotion tags found within the preset time window is calculated. Weighted fusion considers the time difference between each text emotion tag's own timestamp and the timestamp of the current facial image frame; the smaller the time difference, the greater the weight. The result of weighted fusion is a comprehensive emotion vector or a normalized emotion tag, which is used as the speaker context emotion tag precisely aligned with the current frame of facial image. In some embodiments, weighted fusion can employ a soft allocation method based on a Gaussian kernel function, expressed as:

[0027] in: This represents the speaker's contextual sentiment vector obtained after alignment. It represents the number of text sentiment tags retrieved within a preset time window. It is the first Each text sentiment tag corresponds to a one-hot encoded vector or distributed representation vector. It is the first The weight of each label, It is the timestamp of the current facial image frame. Is with the first The time point associated with a text sentiment tag (e.g., the midpoint of its corresponding semantic unit). This is the Gaussian kernel width parameter that controls the rate of weight decay. The calculated aligned speaker context sentiment label is combined with the facial sentiment label obtained from the current frame to form a synchronized sentiment pair containing "speaker context - listener response". All frames in the entire video pair and their corresponding synchronized sentiment pairs are collected and arranged strictly in chronological order, ultimately forming a synchronized labeled dataset of speaker sentiment - listener response for model training. Optionally, speech activity detection can employ traditional methods based on energy and zero cross-rate, or it can employ a speech endpoint detection model based on deep learning. In some embodiments, a specific example scenario is that a semantic unit "That's great!" in a dialogue is labeled with the sentiment "happy", with a time interval of 1.0 to 1.5 seconds. The listener video has a frame at 1.3 seconds, with a preset time window of ±0.2 seconds. Then, within the interval of 1.1 to 1.5 seconds, the "happy" label will be retrieved and participate in the weighted fusion calculation of the 1.3-second frame. If there are no other sentiment labels within this time window, then the aligned speaker context sentiment label for that frame is "happy". It is understandable that by using this method based on temporal density distribution and sliding window weighted fusion, discrete text sentiment tag sequences based on semantic units are transformed into speaker contextual sentiment streams that are strictly aligned with consecutive video frames, thus achieving fine alignment of cross-modal data in the temporal dimension.

[0028] In one embodiment of the present invention, based on synchronously labeled data of speaker emotion and listener response, a variational quantization autoencoder architecture is used to compress and learn features from a sequence of listener facial video frames, constructing a high-dimensional discrete latent representation space of emotion-visual features. Frame images of the listener facial video frame sequence are extracted from the synchronously labeled data as input. An encoder network encodes the frame images into continuous high-dimensional feature vectors. A correlation is established between the high-dimensional feature vectors output by the encoder network and the speaker contextual emotion labels in the synchronously labeled data, embedding the emotion label information into the encoding process of the high-dimensional feature vectors. A learnable discrete codebook is introduced, consisting of multiple codebook vectors. The similarity between the high-dimensional feature vector and each codebook vector in the discrete codebook is calculated. The index of the codebook vector with the highest similarity is selected as the quantization result, and the selected codebook vector is used as the quantized representation of the high-dimensional feature vector, i.e., the discrete latent representation. A decoder network is used, with the discrete latent representation as input, to reconstruct the listener's facial image. By jointly training the encoder network, discrete codebook, and decoder network using reconstruction loss, quantization loss, and sentiment embedding consistency loss, a high-dimensional discrete latent representation space incorporating sentiment-visual features is ultimately constructed. Specifically, the reconstruction loss is the pixel-level mean squared error loss between the original and reconstructed facial images; the quantization loss is the mean squared error loss between the high-dimensional feature vector output by the encoder and the selected codebook vector; and the sentiment embedding consistency loss is the cross-entropy loss between the aligned speaker context sentiment label and the sentiment category prediction decoded from the quantized discrete latent representation.

[0029] In practice, based on synchronously labeled data of speaker emotion and listener response, a variational quantization autoencoder architecture is used to compress and learn features from the listener's facial video frame sequence to construct a high-dimensional discrete latent representation space of emotion-visual features. The process involves extracting each frame of the listener's facial video frame sequence from the synchronously labeled data as model input. These input images are standard facial region images that have undergone alignment and normalization. An encoder network encodes each input facial image frame into a continuous high-dimensional feature vector. The encoder network is typically a deep convolutional neural network, with a structure including multiple convolutional layers, pooling layers, and fully connected layers, used to extract abstract visual features from the image. Simultaneously with the encoder network outputting the high-dimensional feature vector, speaker contextual emotion labels aligned with the current frame image from the synchronously labeled data are embedded into the encoding process. These speaker contextual emotion labels are first converted into distributed vector representations, and then fused with features from the intermediate layers of the encoder network through a projection layer or influenced by an attention mechanism. This ensures that the generated high-dimensional feature vector not only contains facial visual information but also incorporates the corresponding speaker emotional context. A learnable discrete codebook is introduced, randomly initialized before training, and consists of K fixed-dimensional codebook vectors, each representing a basic visual-emotional prototype in the discrete latent representation space. The similarity between the high-dimensional feature vector output by the encoder network and each codebook vector in the discrete codebook is calculated. Similarity is typically measured using cosine similarity or the reciprocal of the Euclidean distance. The index of the codebook vector with the highest similarity to the high-dimensional feature vector is selected as the quantization result; this index is a discrete integer. The selected codebook vector is used as the quantized representation of the high-dimensional feature vector, which is the discrete latent representation of the current facial image frame. A decoder network, typically a deconvolutional neural network or a convolutional neural network with upsampling layers, is used as input to reconstruct a listener's facial image with the same dimensions as the original input image from the discrete latent representation. By combining multiple loss functions to train the encoder network, discrete codebook, and decoder network, a high-dimensional discrete latent representation space of emotion-visual features incorporating emotional information is ultimately constructed. The training objective is to enable the entire architecture to effectively compress facial images into a discrete space and reconstruct them while maintaining consistency with the emotion labels. In some embodiments, the reconstruction loss uses the pixel-level mean squared error between the original facial image and the reconstructed facial image; the quantization loss uses the mean squared error between the high-dimensional feature vector output by the encoder and the selected codebook vector; and the emotion embedding consistency loss uses the cross-entropy loss between the aligned speaker context emotion labels and the emotion category prediction decoded from the quantized discrete latent representation. It can be understood that the total loss function is a weighted sum of these loss terms, used to guide the update of the model parameters.Optionally, the size K of the discrete codebook is an important hyperparameter. A larger K provides a finer discrete representation but increases model complexity and training difficulty. See Table 1, which shows a simplified example illustrating the calculated values ​​of different loss terms and their impact on model updates in a training batch.

[0030] Table 1: Example of loss terms in training batches of variational quantization autoencoders

[0031] The specific calculation of the sentiment embedding consistency loss involves an auxiliary sentiment classifier that takes quantized discrete latent representations (codebook vectors) as input and predicts a sentiment category distribution. Sentiment embedding consistency loss It can be calculated using the following formula:

[0032] in: This indicates a loss of emotional embedding consistency. It is the total number of emotion categories. It is the first one-hot encoded vector of the aligned speaker contextual sentiment label. The value of each element (0 or 1). It is the th in the sentiment probability distribution predicted by the sentiment classifier The probability of each category. In some embodiments, the sentiment classifier is a shallow, fully connected neural network that shares a discrete latent representation as input with the decoder network but has independent parameters. During training, the encoder network, discrete codebook, decoder network, and sentiment classifier are jointly optimized to minimize the total loss through backpropagation. It is understood that the vectors in the discrete codebook are also updated during training via gradient descent to better represent the data distribution. Optionally, in the quantization step, to ensure that the gradient can be backpropagated to the encoder through a non-differentiable index selection operation, a pass-through estimator trick can be used, directly copying the gradient of the decoder input to the encoder output during backpropagation.

[0033] See Figure 3In the latent space construction and hyperparameter tuning stages of the dynamic listener facial expression response model based on dialogue emotion, the discrete codebook size K exhibits a significant inverse correlation with model performance. Specifically, the discrete codebook size K is plotted on the horizontal axis, while the vertical axis represents the model's reconstruction quality and training speed (normalized). Reconstruction quality reflects the model's ability to reconstruct features from listener facial video frame sequences, while training speed reflects the model's computational efficiency in the discrete latent representation space construction and feature learning stages. As the discrete codebook size K gradually increases from its initial low value to the range of 1000, the green line representing reconstruction quality shows a continuous upward trend, confirming that a larger discrete codebook can provide more refined visual-emotional prototype representations. This significantly improves the model's reconstruction accuracy of listener facial images in the discrete latent representation space, enabling it to better reproduce facial visual features that incorporate the speaker's emotional context. Conversely, the orange line representing training speed shows a continuous downward trend, indicating that increasing the codebook size increases the complexity of similarity calculation between high-dimensional feature vectors and codebook vectors, while also increasing the cost of updating and maintaining the discrete codebook, leading to a gradual decrease in the model's training efficiency during latent space construction. In hyperparameter tuning practice, a trade-off between model performance and training cost must be made based on this associated feature: if prioritizing reconstruction quality to achieve accurate prediction of listener facial expression responses is desired, a larger discrete codebook size K can be selected; if reducing resource consumption during the training phase is necessary to improve model iteration efficiency, the value of K can be appropriately reduced. During the process, it is necessary to combine the convergence of reconstruction loss, quantization loss and emotional embedding consistency loss to determine the optimal K value that balances reconstruction quality and training efficiency, and finally construct a discrete latent representation space that has both fine emotional-visual representation capabilities and is adapted to training resource configuration.

[0034] In one embodiment of the present invention, a dynamic conditional probability distribution relationship between speaker contextual emotion features and listener facial reaction features is modeled within a discrete latent representation space. During the training phase, synchronous emotion pairs from multiple consecutive time steps are extracted sequentially from the synchronously labeled data of speaker emotion and listener reaction. For each time step, a discrete latent representation corresponding to the listener's facial image at the current time step is obtained from the discrete latent representation space as the listener reaction feature. Speaker contextual emotion labels for the current time step and several previous time steps are obtained from the synchronous emotion pairs, forming a speaker emotion context within a time window. This speaker emotion context within the time window is encoded into a fixed-dimensional emotion context vector. Using the listener reaction features from the previous time step and the emotion context vector at the current time step as conditions, a conditional probability model predicting the listener reaction features at the current time step is established. This conditional probability model is implemented through a recurrent neural network, whose hidden state integrates historical listener reaction features and emotion context information. A conditional probability model is trained by maximizing the log-likelihood of the conditional probabilities across all time steps on the synchronously labeled data of speaker emotion and listener response, thereby modeling the dynamic conditional probability distribution relationship. The hidden state of this recurrent neural network is used to represent the fusion information of the listener's facial expression response history and emotional context history up to the current time step. At each time step, the input of the recurrent neural network is the concatenation of the embedded representation of the codebook vector corresponding to the listener response feature of the previous time step and the emotional context vector of the current time step. The recurrent neural network updates the hidden state of the current time step based on the current input and the hidden state of the previous time step. In the output layer, the hidden state of the current time step is mapped to the log probability distribution of all codebook vectors in the discrete latent representation space through a fully connected layer. Samples are taken from this log probability distribution, or the codebook index with the maximum log probability is selected as the listener response feature predicted for the current time step.

[0035] In practice, the dynamic conditional probability distribution relationship between speaker contextual emotion features and listener facial response features is modeled within a discrete latent representation space. The process begins with the processing of synchronously labeled speaker emotion-listener response data during the training phase. From the synchronously labeled data, synchronous emotion pairs from multiple consecutive time steps are extracted in chronological order, forming a sequence sample for probabilistic modeling. Each synchronous emotion pair at a time step contains an aligned speaker contextual emotion label and a facial emotion label corresponding to the listener's facial image, along with the original image. For each specific time step in the sequence sample, a discrete latent representation corresponding to the listener's facial image at the current time step is obtained from the constructed high-dimensional emotion-visual feature discrete latent representation space through a query operation. This discrete latent representation is a codebook index, and its corresponding codebook vector serves as the listener's response feature at the current time step. Speaker contextual emotion labels for the current time step and several previous historical time steps are obtained from the synchronous emotion pairs, forming the speaker's emotional context within a fixed-length time window. The length of the time window defines the range of influence of historical emotional information on the current response. The speaker's emotional context within this time window is encoded into a fixed-dimensional emotional context vector. This encoding process can be implemented using a recurrent neural network (RNN), a one-dimensional convolutional network, or a simple feedforward neural network. This network maps a series of discrete emotional label sequences into a continuous context representation vector. In practice, a conditional probability model predicting the listener's response features at the current time step is established, conditioned on the listener's response features from the previous time step and the emotional context vector at the current time step. The conditional probability model defines the probability distribution of the current listener's response features taking various possible values ​​in the discrete latent representation space, given historical listener responses and the current speaker's emotional context. The conditional probability model is implemented using a recurrent neural network. The hidden state of the RNN represents the fusion information of the listener's facial expression response history and emotional context history up to the current time step. At each time step, the input to the RNN is the concatenated vector of the codebook vector corresponding to the listener's response features from the previous time step and the emotional context vector at the current time step. The RNN updates and outputs the hidden state at the current time step based on the concatenated input vector and the hidden state of the RNN from the previous time step. In the output layer of the recurrent neural network, the hidden state at the current time step is mapped through a fully connected layer to the log probability distribution of all codebook vectors in the discrete latent representation space. This distribution covers all possible listener response prototypes. From the obtained log probability distribution, a codebook index can be selected by random sampling, or the codebook index with the largest log probability value can be directly selected as the listener response feature predicted at the current time step. The conditional probability model is trained by maximizing the log-likelihood of the conditional probability across all time steps on the synchronously labeled speaker emotion-listener response data, thereby modeling the dynamic conditional probability distribution relationship.In some embodiments, a specific example scenario involves processing a dialogue segment containing three consecutive time steps. Referring to Table 2, example data extracted from synchronously labeled data and used for modeling is shown.

[0036] Table 2: Example data table of continuous time steps used for dynamic conditional probability modeling

[0037] In the example shown in Table 2, when making a prediction at time step t=3, the input to the recurrent neural network is the codebook vector embedding corresponding to the listener response feature (codebook index 78) predicted at time step t=2, concatenated with the sentiment context vector (encoded by [surprise, neutral]) at time step t=3. Based on this input and its hidden state at time step t=2, the recurrent neural network calculates the hidden state at time step t=3 and further outputs the log probability distribution over 512 codebook vectors, finally selecting the codebook index 42 with the highest probability as the prediction. The training objective of maximizing the conditional log-likelihood can be formalized as minimizing the negative log-likelihood loss. :

[0038] in: This represents the negative log-likelihood loss. It is the total number of time steps in the sequence. It is a time step Index of real listener response signatures Indicates time step All previous historical listener response characteristic sequences, It is a time step The emotional context vector, This represents all trainable parameters of a conditional probability model (recurrent neural network). It is the model-predicted, true index under given conditions. The probability. It can be understood that the training process updates the parameters of the recurrent neural network through the backpropagation algorithm. This allows the model to learn dynamic mapping patterns from historical responses and the current speaker's emotion to the current listener's response. In some embodiments, the recurrent neural network can employ a long short-term memory network or gated recurrent units to better capture long-distance temporal dependencies. Optionally, the encoding network for the emotional context vector can be jointly trained end-to-end with the recurrent neural network. It can be understood that through this sequence-based conditional probability modeling, the model not only learns the static correlation between listener facial expressions and speaker's emotional changes, but also captures the dynamic evolution of listener facial expression responses over time. Optionally, a teacher-forced strategy can be used during training, using the actual listener response features from the previous time step instead of the predicted values ​​as input to accelerate model convergence.

[0039] See Figure 4 The loss convergence curve intuitively reflects the training convergence characteristics and learning effect of the model. Specifically, when modeling the dynamic conditional probability distribution of speaker contextual emotional features and listener facial reaction features in the discrete latent representation space, the training phase takes the synchronous labeled data of speaker emotion and listener reaction as input, extracts continuous synchronous emotional pairs by time step, and uses the real listener reaction features (codebook vector embedding) of the previous time step and the emotional context vector of the current time step as RNN input. The model parameters are updated through the time backpropagation algorithm to minimize the negative log-likelihood loss, thereby learning the dynamic mapping law from historical listener reaction to current speaker emotion to current listener reaction. From the curve characteristics, during the training iterations from 1 to 10, the negative log-likelihood loss value shows an ideal trend of continuous monotonic decrease and gradual convergence: the loss value is highest in the initial iteration (1st round) (about 2.85), indicating that the model's ability to fit the conditional probability distribution under the initial parameters is weak; as the iterations increase, the loss value decreases rapidly, dropping to about 2.15 in the 2nd round and about 1.68 in the 3rd round, reflecting that the model quickly learns the emotion-response association pattern in the early stage; in subsequent rounds (4th-10th rounds), the rate of decrease in the loss value slows down and gradually approaches a stable low value (about 0.45 in the 10th round), indicating that the model completes fine-tuning in the later stage, and the fitting accuracy of the dynamic conditional probability distribution continues to improve, finally achieving full convergence. The convergence curve validates the training effectiveness of the recurrent neural network conditional probability model: by maximizing the conditional log-likelihood on synchronously labeled data (equivalent to minimizing the negative log-likelihood loss), the model successfully captures the temporal dependence and dynamic evolution of listener facial expressions as the speaker's emotions change, providing a reliable probabilistic modeling foundation for generating temporally continuous and smoothly transitioning listener facial expression response sequences during the inference phase. The teacher-enforced strategy used during training (using real historical responses as input) accelerates model convergence and ensures the smooth descent of the loss curve, consistent with the training patterns of long-term sequence modeling.

[0040] In one embodiment of the present invention, based on the dynamic conditional probability distribution relationship obtained through modeling, new speaker dialogue text and its sentiment sequence are received during the inference phase, and a corresponding temporally continuous and smoothly transitioning sequence of listener facial expression responses is predicted and generated. During the inference phase, new speaker dialogue text is input, and fine-grained sentiment analysis is performed on the text to generate a corresponding text sentiment label sequence. This text sentiment label sequence is mapped onto a continuous time axis, and speaker contextual sentiment labels for each time step in the inference phase are generated at the same temporal resolution as in the training phase. An empty listener response feature sequence is initialized, and the listener response features for the initial time step are set to a preset discrete latent representation corresponding to a neutral expression. Starting from the initial time step, predictions are performed sequentially for each time step, obtaining the speaker context sentiment labels for the current time step and several previous time steps, forming a sentiment context vector. The listener response features predicted from the previous time step and the sentiment context vector for the current time step are input into a pre-modeled dynamic conditional probability distribution. From this dynamic conditional probability distribution, the maximum probability value is sampled or taken to obtain the listener response features predicted for the current time step, which is a codebook index in the discrete latent representation space. The codebook vector corresponding to the obtained codebook index is used as the listener response features predicted for the current time step and added to the listener response feature sequence. The above steps are iteratively executed until all time steps are processed, resulting in a complete predicted listener response feature sequence. Next, each listener response feature in the predicted listener response feature sequence, i.e. the codebook vector corresponding to each codebook index, is input into the trained decoder network. The decoder network decodes and reconstructs each codebook vector into a frame of listener facial image. All the reconstructed listener facial images are combined in their corresponding time order to generate a temporally continuous facial expression video sequence. The generated facial expression video sequence is then subjected to temporal smoothing post-processing to ensure that the expression changes between adjacent frames are visually smooth.

[0041] In practice, based on the dynamic conditional probability distribution obtained from modeling, new speaker dialogue text and its sentiment sequence are received during the inference phase, and a corresponding temporally continuous and smoothly transitioning sequence of listener facial expression responses is predicted and generated. At the beginning of the inference phase, a new, unprocessed speaker dialogue text is input, which serves as the basis for generating the expected listener responses. The same fine-grained sentiment analysis process as the training phase is performed on the new speaker dialogue text. This process includes encoding using a pre-trained language model, sentence-level sentiment classification, semantic unit recognition, and attention weight allocation, ultimately generating a text sentiment label sequence corresponding one-to-one with each independent semantic unit in the input text. The generated text sentiment label sequence is mapped onto a continuous timeline. This mapping process requires allocating temporal information to each text sentiment label. This can be done by assuming a fixed speech rate or using an additional speech synthesis front-end to estimate the duration of each semantic unit, and generating speaker context sentiment labels corresponding to each discrete time step in the inference phase at the same temporal resolution as the model processing during the training phase (e.g., 30 frames per second). Initialize an empty listener response feature sequence and set the listener response features at the initial time step to a preset neutral expression in the constructed high-dimensional emotion-visual feature discrete latent representation space and its codebook vector. This preset codebook index is usually determined in the training data by statistically analyzing the index corresponding to the neutral expression that appears most frequently.

[0042] Starting from the initial time step, iterative prediction operations are performed sequentially for each time step. For the current time step, speaker contextual sentiment labels from the current time step and several previous historical time steps are obtained, forming a sentiment context within a time window of the same length as defined in the training phase. This sentiment context within the time window is input into the same sentiment context encoding network as in the training phase. The encoding network encodes the discrete sentiment label sequence into a fixed-dimensional sentiment context vector. The listener response features (a codebook index) predicted from the previous time step are converted into their corresponding codebook vector embedding representation through an embedding layer. This embedding representation is then concatenated with the sentiment context vector of the current time step to form an input vector that integrates historical responses and the current context. The concatenated input vector is then input into a modeled and trained dynamic conditional probability distribution, which is parameterized by a trained recurrent neural network model. From the logarithmic probability distribution of all codebook vectors in the discrete latent representation space, output by the dynamic conditional probability distribution, the predicted listener response features for the current time step are obtained through sampling or maximizing probability operations. The predicted listener response features are specific codebook indices in the discrete latent representation space. In some embodiments, the sampling operation can employ temperature-coefficient-based multinomial sampling, which balances response diversity and reasonableness. Calculated using the following formula:

[0043] in: Indicates the codebook index The probability of sampling. The dynamic conditional probability distribution relationship (output layer of a recurrent neural network) serves as the codebook index. The generated raw score, It is the size of the discrete codebook (the total number of codebook vectors). It is a temperature parameter that controls the sharpness of the sampling distribution. When the value approaches 0, the sampling process approaches a greedy operation of taking the maximum probability value. As the sample size increases, the sampling distribution becomes more uniform. The codebook vector corresponding to the sampled or selected codebook index is taken from the trained discrete codebook and used as the specific vector of the predicted listener response feature for the current time step. This codebook index or vector is then added to the continuously growing listener response feature sequence. The above steps, from obtaining the emotional context vector to adding the predicted features to the sequence, are iteratively executed until all time steps generated from the input text have been processed. Finally, a complete predicted listener response feature sequence, temporally aligned with the input speaker's emotional sequence, is obtained. This sequence consists of a series of codebook indices or codebook vectors.

[0044] After obtaining the complete predicted listener response feature sequence, the facial image generation unit begins its work. For each listener response feature in the predicted listener response feature sequence, i.e., each codebook index, its corresponding codebook vector is retrieved from the pre-trained discrete codebook through a query operation. Each retrieved codebook vector is then sequentially input into a pre-trained variational quantization autoencoder architecture decoder network, which has the exact same structure and parameters as the decoder network used in the training phase. The decoder network decodes each input codebook vector and reconstructs it into a frame of listener facial image. The reconstructed facial images are consistent in size and format with the standardized facial region images used during training. All listener facial images reconstructed by the decoder network are arranged strictly according to their corresponding temporal order to generate a temporally continuous facial expression video sequence. The generated facial expression video sequence undergoes temporal smoothing post-processing. Temporal smoothing post-processing can employ inter-frame interpolation algorithms or temporal filtering techniques to smoothly transition the positions of facial key points or image pixel values ​​between adjacent frames, ensuring that expression changes between adjacent frames are visually smooth and natural, eliminating visual jitter or abrupt jumps that may be caused by discrete sampling or prediction jumps. In some embodiments, temporal smoothing post-processing uses a one-dimensional Gaussian filter to convolve the values ​​at the same pixel location in multiple consecutive frames. It can be understood that the resulting facial expression video sequence can be directly superimposed on the facial model of the virtual listener character, or fused with a neutral listener background video. Optionally, during the iterative prediction process of generating the listener response feature sequence, a bundle search strategy can be used instead of simple greedy sampling to find a globally superior response sequence.

[0045] See Figure 5In the dynamic listener facial expression response model based on dialogue emotion, the correlation strength between speaker emotion and listener response is quantified using a Pearson correlation coefficient heatmap. Specifically, the row dimension corresponds to the speaker's six emotional states (neutral, joy, sadness, anger, surprise, and fear), and the column dimension corresponds to the listener's six facial expression responses (neutral, joy, sadness, anger, surprise, and fear). The value of each cell in the heatmap is the Pearson correlation coefficient between the corresponding speaker emotion and listener response. The color intensity visually represents the correlation strength: dark red represents a strong positive correlation (coefficient close to 1), dark blue represents a strong negative correlation (coefficient close to -1), and light white / light blue / light red represent a weak correlation (coefficient close to 0). From the core features of the heatmap, the autocorrelation coefficients on the diagonal are all close to 1.00, verifying a strong positive correlation between the speaker's emotions and the listener's similar facial expressions, which is consistent with the basic law of emotional resonance in human dialogue. At the same time, the off-diagonal areas show clear emotional interaction patterns: for example, the speaker's sadness and the listener's surprise show a strong positive correlation of 0.61, the speaker's surprise and the listener's sadness show a strong positive correlation of 0.70, the speaker's fear and the listener's neutrality show a strong positive correlation of 0.53, while the speaker's neutrality and the listener's joy show a strong negative correlation of -0.65, and the speaker's fear and the listener's joy show a strong negative correlation of -0.70. These quantitative results provide data support for the modeling of dynamic conditional probability distributions in the model, and can be used to optimize the correlation modeling logic between the speaker's contextual emotional features and the listener's facial reaction features. During parameter configuration, the correlation coefficient is calculated based on the speaker emotion-listener response synchronous annotation data generated during the model training phase. The time alignment window is set to the same time resolution as the training phase (e.g., 30 frames / second). The emotion category adopts 6 predefined basic emotion labels to ensure the consistency between the correlation strength calculation and the model training process.

[0046] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A dynamic listener facial expression response model based on dialogue emotion, characterized in that, The model includes the following steps: The data alignment unit acquires a training dataset containing speaker-listener video pairs, extracts corresponding dialogue text data and listener facial video frame sequences from each video pair in the training dataset, performs fine-grained sentiment analysis on the dialogue text data to generate a text sentiment tag sequence corresponding to each semantic unit in the dialogue text, performs frame-by-frame facial expression recognition on the listener facial video frame sequence to generate a facial sentiment tag sequence corresponding to each frame of facial image, and aligns the text sentiment tag sequence and the facial sentiment tag sequence on the time axis to form synchronous labeled data of speaker emotion and listener response. The feature learning unit, based on the synchronously labeled data of speaker emotion and listener response, uses a variational quantization autoencoder architecture to compress and learn features from the listener's facial video frame sequence, and constructs a high-dimensional discrete latent representation space of emotion-visual features. The probabilistic modeling unit models the dynamic conditional probability distribution relationship between the speaker's contextual emotional features and the listener's facial reaction features within the discrete latent representation space. The sequence generation unit, based on the dynamic conditional probability distribution relationship obtained from modeling, receives new speaker dialogue text and its emotional sequence during the inference stage, and predicts and generates a corresponding temporally continuous and smoothly transitioning sequence of listener facial expression responses.

2. The dynamic listener facial expression response model based on dialogue emotion as described in claim 1, characterized in that, Fine-grained sentiment analysis is performed on the dialogue text data to generate a text sentiment tag sequence corresponding to each semantic unit in the dialogue text, including: The dialogue text data is encoded using a pre-trained language model to obtain a context feature vector representation of the text; The contextual feature vector representation of the text is input into the sentiment classification layer to generate preliminary text sentiment labels at the sentence level. Based on a predefined sentiment category dictionary and semantic dependency relationships, the dialogue text data is segmented and semantic role labeled to identify each independent semantic unit. Attention is fused between the vector representation of each independent semantic unit and the sentence-level text sentiment label to calculate the contribution weight of each independent semantic unit to the overall sentence sentiment. Based on the contribution weights, sentence-level text sentiment tags are redistributed and fine-tuned to generate text sentiment tags that correspond one-to-one with each independent semantic unit and have confidence levels. These tags are then arranged in the order in which the semantic units appear in the text to form a text sentiment tag sequence.

3. The dynamic listener facial expression response model based on dialogue emotion as described in claim 2, characterized in that, Perform frame-by-frame facial expression recognition on the listener's facial video frame sequence to generate a facial emotion tag sequence corresponding to each frame of the facial image, including: A pre-trained facial landmark detection model is used to process each frame of the listener's facial video frame sequence to locate the coordinates of the facial landmarks. Based on the coordinates of facial key points, each frame of facial image is normalized and aligned to obtain a standardized facial region image. The standardized facial region image is input into a pre-trained facial expression recognition neural network; The facial expression recognition neural network extracts facial visual features and outputs a facial emotion probability distribution vector. Non-maximum suppression is applied to the facial emotion probability distribution vector, and the emotion category with the highest probability value is selected as the facial emotion label for each frame of facial image. Arrange all facial emotion tags in chronological order of video frames to form a facial emotion tag sequence.

4. The dynamic listener facial expression response model based on dialogue emotion as described in claim 3, characterized in that, Aligning the text emotion tag sequence with the facial emotion tag sequence along the time axis creates synchronized annotation data of speaker emotion and listener response, including: Obtain the audio stream corresponding to the dialogue text data, or extract the corresponding speaker audio from the video pair; Speech activity detection is performed on the audio stream to determine the start and end timestamps of each semantic unit on the timeline; Based on the start and end timestamps of each semantic unit, its corresponding text sentiment tags are mapped onto a continuous timeline, forming a density distribution of text sentiment tags over time. For each frame in the facial emotion tag sequence, whose timestamp is known, find all text emotion tags within a preset time window that are closest to the timestamp of each frame in the temporal density distribution of the text emotion tags. The weighted fusion result of all text sentiment tags within the preset time window is calculated and used as the speaker context sentiment tag aligned with each frame of facial image; The aligned speaker contextual emotion tags are combined with the facial emotion tags of each frame to form a synchronized emotion pair; All frames in a video pair and their corresponding synchronized emotion pairs are collected in chronological order to form synchronized labeled data of speaker emotion and listener response.

5. The dynamic listener facial expression response model based on dialogue emotion as described in claim 4, characterized in that, Based on the synchronously labeled speaker emotion-listener response data, a variational quantization autoencoder architecture is used to compress and learn features from the listener's facial video frame sequence, constructing a high-dimensional discrete latent representation space of emotion-visual features, including: Frame images of the listener's facial video frame sequence are extracted as input from the synchronously labeled data of speaker's emotion and listener's response; The frame image is encoded into a continuous high-dimensional feature vector using an encoder network; A correlation is established between the high-dimensional feature vector output by the encoder network and the speaker context sentiment tags in the synchronously labeled data, and the sentiment tag information is embedded into the encoding process of the high-dimensional feature vector. A learnable discrete codebook is introduced, which consists of multiple codebook vectors; Calculate the similarity between the high-dimensional feature vector and each codebook vector in the discrete codebook, and select the index of the codebook vector with the highest similarity as the quantization result; The selected codebook vector is used as the quantized representation of the high-dimensional feature vector, i.e., the discrete latent representation; Using a decoder network, the listener's facial image is reconstructed from the discrete latent representation as input; The encoder network, discrete codebook, and decoder network are trained together by reconstruction loss, quantization loss, and sentiment embedding consistency loss, and finally a high-dimensional sentiment-visual feature discrete latent representation space that integrates sentiment information is constructed.

6. The dynamic listener facial expression response model based on dialogue emotion as described in claim 5, characterized in that, Within the discrete latent representation space, modeling the dynamic conditional probability distribution relationship between speaker contextual emotional features and listener facial reaction features includes: During the training phase, synchronous emotion pairs from multiple consecutive time steps are extracted in chronological order from the synchronously labeled data of speaker emotion and listener response. For each time step, a discrete latent representation corresponding to the listener's facial image at the current time step is obtained from the discrete latent representation space as a listener response feature; The speaker contextual sentiment labels for the current time step and several previous time steps are obtained from the synchronous sentiment pairs to form a speaker sentiment context within a time window. The speaker's emotional context within the time window is encoded into a fixed-dimensional emotional context vector; Using the listener response features of the previous time step and the emotional context vector of the current time step as conditions, a conditional probability model is established to predict the listener response features of the current time step. The conditional probability model is implemented through a recurrent neural network, whose hidden state integrates historical listener response characteristics and emotional context information. The conditional probability model is trained by maximizing the log-likelihood of the conditional probability at all time steps on the synchronously labeled data of the speaker's emotion and the listener's response, thereby modeling the dynamic conditional probability distribution relationship.

7. The dynamic listener facial expression response model based on dialogue emotion as described in claim 6, characterized in that, Based on the dynamic conditional probability distribution relationship obtained through modeling, the system receives new speaker dialogue text and its emotional sequence during the inference stage, and predicts and generates a corresponding temporally continuous and smoothly transitioning sequence of listener facial expression responses, including: During the reasoning phase, new speaker dialogue text is input; Fine-grained sentiment analysis is performed on the new speaker dialogue text to generate a corresponding text sentiment tag sequence; The text sentiment tag sequence is mapped onto a continuous time axis, and speaker context sentiment tags for each time step in the inference phase are generated at the same time resolution as the training phase. Initialize an empty listener response feature sequence and set the listener response features at the initial time step to a preset discrete latent representation of a neutral expression; Starting from the initial time step, prediction is performed sequentially for each time step: obtain the speaker context sentiment labels for the current time step and several previous time steps to form a sentiment context vector; The listener response features predicted at the previous time step and the emotional context vector at the current time step are input into the modeled dynamic conditional probability distribution relationship. From the dynamic conditional probability distribution relationship, sample or take the maximum probability value to obtain the listener response feature predicted at the current time step, which is a codebook index in the discrete latent representation space; The codebook vector corresponding to the obtained codebook index is used as the listener response feature predicted at the current time step and added to the listener response feature sequence. The execution steps are iterated until all time steps are processed to obtain a complete sequence of predicted listener response features.

8. The dynamic listener facial expression response model based on dialogue emotion as described in claim 7, characterized in that, After obtaining the complete predictive listener response feature sequence, the following is also included: The facial image generation unit inputs each listener response feature in the predicted listener response feature sequence, i.e. the codebook vector corresponding to each codebook index, into the trained decoder network. The decoder network decodes each codebook vector and reconstructs it into a frame of the listener's facial image; All the reconstructed facial images of the listeners are combined in their corresponding time sequence to generate a temporally continuous video sequence of facial expressions. The generated facial expression video sequence is post-processed with temporal smoothing to ensure that the expression changes between adjacent frames are visually smooth and to eliminate jitter or abrupt changes.

9. The dynamic listener facial expression response model based on dialogue emotion as described in claim 8, characterized in that, The conditional probability model is implemented through a recurrent neural network, whose hidden states integrate historical listener response characteristics and emotional context information, specifically including: The hidden state of the recurrent neural network is used to represent the fusion information of the listener's facial expression response history and emotional context history up to the current time step. At each time step, the input of the recurrent neural network is the embedded representation of the codebook vector corresponding to the listener response features of the previous time step, concatenated with the emotional context vector of the current time step. The recurrent neural network updates the hidden state at the current time step based on the current input and the hidden state at the previous time step. In the output layer, the hidden state at the current time step is mapped to the log probability distribution of all codebook vectors in the discrete latent representation space through a fully connected layer; Sample from the logarithmic probability distribution, or select the codebook index with the highest logarithmic probability, as the listener response feature predicted at the current time step.

10. The dynamic listener facial expression response model based on dialogue emotion as described in claim 9, characterized in that, The reconstruction loss used by the variational quantization autoencoder architecture during training is the pixel-level mean square error loss between the original facial image and the reconstructed facial image. Quantization loss is the mean square error loss between the high-dimensional feature vector output by the encoder and the selected codebook vector; The sentiment embedding consistency loss is the cross-entropy loss between the aligned speaker context sentiment label and the sentiment category prediction decoded from the quantized discrete latent representation.

Images