Digital image facial expression real-time conversion method based on deep learning

Abstract

The invention discloses a digital image facial expression real-time conversion method based on deep learning, and the method comprises the following steps: S1, collecting video images, voice audios and text input in user interaction, carrying out the preprocessing, and generating a multi-modal feature sequence; s2, extracting a mouth feature sequence through convolution and a bidirectional gating structure; s3, generating an expression parameter set based on an Audio 2Face model; s4, calling an Unreal animation engine to construct a face driver, and generating an expression frame sequence; s5, analyzing the multi-modal features to generate a rendering instruction; and S6, driving the digital image face to render according to the rendering instruction, and updating network parameters based on error feedback. According to the method, the multi-modal interaction data and the deep learning method are fused, and the Audio2Face model and the Unreal engine are combined, so that the generation precision and the real-time response capability of the facial expression of the digital image are improved.

Description

Technical Field

[0001] This invention relates to the field of facial expression modeling technology, and in particular to a method for real-time conversion of digital image facial expressions based on deep learning. Background Technology

[0002] With the continuous development of artificial intelligence technology, digital humans, virtual avatars, and other forms of human-computer interaction have been widely used in fields such as virtual live streaming, online customer service, and immersive teaching. To enhance user immersion and emotional interaction, real-time facial expression-driven technology has become an important technical means to improve the naturalness and interactivity of digital avatars. Current common methods mainly rely on rule-based modeling, manual feature extraction, or traditional animation scripts for expression control, making it difficult to achieve a balance between high timeliness, high fidelity, and multimodal perception.

[0003] In existing technologies, some methods rely on cameras to capture facial key points or signature muscle movement trajectories, driving 3D model deformation through static parameters. However, such methods suffer from unstable recognition under different users, complex lighting, or occluded scenarios, making it difficult to adapt to the interactive needs of real-time, multi-source input. Simultaneously, the potential value of non-visual modal information such as speech and text in the expression generation process is not fully utilized, resulting in digital avatars lacking contextual understanding and emotional expression capabilities in their facial responses.

[0004] In recent years, the rise of deep learning technology has provided new ideas for real-time facial expression conversion. Through structures such as convolutional neural networks and recurrent neural networks, dynamic features in time series can be effectively extracted, and combined with models such as Audio2Face to achieve audio-driven expression parameter generation. However, existing methods generally suffer from the following problems: First, the feature extraction process does not fully integrate multimodal information from video, audio, and text, making it difficult to accurately recreate the expression intent within the context; second, the generated expression parameters are not fused with semantic state for inference, resulting in a lack of contextual relevance and consistency in the generated expressions; third, the expression generation process lacks a feedback optimization mechanism, leading to insufficient generalization ability of the model across different users or usage scenarios.

[0005] Therefore, how to provide a method for real-time conversion of digital facial expressions based on deep learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a real-time facial expression conversion method for digital avatars based on deep learning. This invention fully integrates multimodal interactive data from video images, audio, and text input, combines a deep neural network with convolutional and bidirectional gating structures for lip-shape feature extraction, utilizes the Audio2Face model to generate expression parameters, and drives the digital avatar's facial skeleton nodes through the Unreal Engine to achieve high-precision expression synthesis. At the same time, it introduces semantic parsing and feedback error optimization mechanisms, which have the advantages of fast response speed, high expression naturalness, strong contextual consistency, and strong cross-modal generalization ability.

[0007] A method for real-time facial expression conversion of digital images based on deep learning according to an embodiment of the present invention includes the following steps: S1. Collect user interaction data, including video images, voice audio, and text commands, at a fixed frequency and preprocess it to generate a multimodal feature sequence; S2. Using a convolutional neural network containing bidirectional gated recurrent units, perform one-dimensional convolution and pooling operations on the multimodal feature sequence, extract the bidirectional state vector of the pooling result along the time dimension, and construct the lip shape feature sequence. S3. Based on the Audio2Face model, a facial simulation framework is built, and a low-dimensional mapping operation is performed on the lip shape feature sequence. Based on the mapping result, a continuous encoding sequence is constructed and an expression decoding operation is performed to obtain the expression parameter set. S4. Based on the set of facial expression parameters, a facial driving sequence is constructed using the Unreal Engine to control the skeletal nodes of the preset digital image face to perform position offset calculations and generate a sequence of facial expression frames. S5. Perform semantic parsing on the multimodal feature sequence, construct an intent expression set, and generate a state label set by combining the expression frame sequence. Perform a fusion mapping operation on the state label set and the expression parameter set to construct rendering instructions. S6. Drive the digital image face to render according to the rendering instructions, generate an expression video sequence and calculate the error with the expression frame sequence, and update the parameters of the convolutional neural network and the facial simulation framework based on the error results.

[0008] Optionally, the digital image represents a virtual character form structure constructed based on 3D modeling, including a facial mesh, a set of bound skeletal nodes, and material texture information. The skeletal nodes represent the bound control points in the facial mesh structure of the digital image used to perform facial expression deformation.

[0009] Optionally, S1 specifically includes: S11. Collect video image sequences, voice audio signals and text command data during user interaction at a fixed frequency, and perform format standardization processing to generate a set of video frames, audio signals and character sequences. S12. Perform grayscale normalization and histogram equalization on the video frame set, extract the face region and perform face alignment to construct the video feature sequence; S13. Perform frame segmentation and Hamming window weighting processing on the audio signal, extract spectral features using the Mel-Cepstral algorithm, and perform Z-score normalization on the spectral features to construct an audio feature sequence. S14. Perform word segmentation and encoding operations on the character sequence, and use a preset word embedding dictionary to convert the character fragments at each time step into fixed-length embedding vectors to construct the text feature sequence; S15. Perform time alignment processing on the video feature sequence, audio feature sequence and text feature sequence to construct a multimodal feature sequence.

[0010] Optionally, S2 specifically includes: S21. Using a convolutional neural network containing bidirectional gated recurrent units, the multimodal feature sequence is divided into multiple fixed-length sliding windows in chronological order, and a one-dimensional convolution operation is performed in each sliding window. The convolution kernel slides along the time dimension to extract local change features and generate a convolution response sequence. S22. Perform max pooling on the convolutional response sequence to compress local variation features along the time dimension and construct the pooling result sequence; S23. The positive and negative state vectors of the pooling result sequence are extracted along the time dimension by a bidirectional gated recurrent unit, and a vector concatenation operation is performed on the positive and negative state vectors. The concatenation results are arranged in time order to generate a lip shape feature sequence.

[0011] Optionally, the extraction process of the forward state vector and the reverse state vector specifically includes: The pooling result sequence is input into the forward structure of the bidirectional gated recurrent unit in chronological order. At each time step, the pooling result vector is concatenated with the hidden state vector of the previous time step. Two rounds of linear transformation and Sigmoid function activation operations are performed on the concatenated result to generate update gate and reset gate respectively. In the first time step, the preset initialization vector is used as the hidden state vector of the previous time step. The retention ratio of the hidden state vector from the previous time step is adjusted based on the reset threshold, and then fused with the pooling result vector of the current time step to generate a candidate state vector. The candidate state vector and the hidden state vector of the previous time step are weighted and fused according to the update threshold to generate the hidden state vector of the current time step as the positive state vector. The pooling result sequence is input into the reverse structure of the bidirectional gated recurrent unit in reverse time order. At each time step, the pooling result vector is concatenated with the hidden state vector of the next time step, and the calculation of update gate and reset gate, candidate state vector generation and weighted fusion operations are performed to generate the reverse state vector. In the last time step, the preset initialization vector is used as the hidden state vector of the next time step. The forward state vector and the corresponding reverse state vector at each time step are concatenated along the feature dimension, and the concatenation results are arranged in chronological order to construct a lip shape feature sequence.

[0012] Optionally, S3 specifically includes: S31. A facial simulation framework is built based on the Audio2Face model. Temporal feature mapping is performed on the lip shape feature sequence according to the time step. Latent distribution parameters are extracted by variational autoencoder. S32. Perform reparameterization sampling based on the latent distribution parameters, construct the latent representation vector in time order, and generate a continuous coding sequence; S33. Input the continuous encoded sequence into the decoding path of the facial simulation framework, perform the expression feature reconstruction operation at each time step, and generate a set of expression parameters. S34. Perform adversarial discrimination operation on the expression parameter set and the preset parameter set along the time dimension through generative adversarial network, and update the parameters of the facial simulation framework according to the discrimination result; S35. Using the updated facial simulation framework, feature mapping, reparameterization sampling, and facial expression feature reconstruction operations are performed based on the lip shape feature sequence to regenerate the facial expression parameter set.

[0013] Optionally, the facial simulation framework uses the Audio2Face model as its core, including a variational autoencoder and a generative adversarial network. The specific process of generating the expression parameter set includes: The lip shape feature sequence is input into the encoding path of the variational autoencoder, and one-dimensional convolution and GELU function activation operations are performed sequentially at each time step. Batch normalization is then performed on the activation results to generate latent distribution parameters. Parallel linear mapping paths are built based on each potential distribution parameter. At each time step, the mean vector and log-variance vector of the potential distribution parameters are generated to construct a set of distribution parameters. A pseudo-random perturbation vector following a standard normal distribution is introduced. For each pair of distribution parameters, a reparameterization sampling operation is performed, and the sampling results are processed by a moving average to generate a continuous encoded sequence. The continuous encoded sequence is input into the decoding path of the Audio2Face model. At each time step, a two-layer perceptron transformation is performed on the latent representation vector. The ReLU function and the Sigmoid function are used for nonlinear mapping, respectively. After each layer transformation, residual connection and batch normalization operations are performed to extract low-dimensional representations to generate a set of facial expression parameters. The facial expression parameters include the position offset of key skeletal control points and muscle weight values. The expression parameter set and the preset parameter set are constructed into sample pairs according to time steps and input into the discriminator structure of the generative adversarial network using a three-layer one-dimensional convolutional network. After each convolutional layer, the LeakyReLU function is activated and Dropout is processed, and the discrimination score of the final convolutional result is extracted through a fully connected network. With the goal of minimizing the discrimination score, an adversarial loss function is constructed, and the parameters of the encoding and decoding paths in the face simulation framework are iteratively updated using a backpropagation mechanism until the adversarial loss function converges. After the loss function converges, the updated facial simulation framework is used to perform latent representation extraction and expression feature decoding operations on the lip feature sequence at time steps to regenerate the expression parameter set.

[0014] Optionally, S4 specifically includes: S41. In the preset driving structure of Unreal Engine, extract the position offset and muscle weight value of the key bone control points at each time step in the expression parameter set. S42. Based on the offset of the skeletal control point position, perform a node position update operation in the set of bound skeletal nodes of the preset digital image, and adjust the displacement amplitude and direction of the skeletal nodes in sync with the muscle control weight value. S43. After completing the update of the skeletal nodes at each time step, the driver calculation interface in the Unreal Engine is called. Based on the skeletal constraint rules and the position changes of the bound skeletal nodes, the facial mesh of the digital image is driven to perform facial expression deformation calculations, and keyframe indexes for the corresponding time step are generated. The skeletal constraint rules represent a set of predefined constraint logic used to limit and guide the spatial range, motion relationship and deformation mode of the bound skeletal nodes when controlling the facial deformation of the virtual character in the Unreal Engine. S44. Execute the rendering cache construction operation based on the keyframe index, and combine the bound bone node state and mesh deformation results of each frame to generate a facial rendering image and construct a frame cache sequence. S45. Organize the frame buffer sequence into an expression frame sequence in chronological order.

[0015] Optionally, S5 specifically includes: S51. Perform semantic parsing on the text features and audio features at each time step in the multimodal feature sequence to extract word meaning tags, tone categories and sentence structures, and construct a semantic representation sequence. S52. Perform time alignment processing on the semantic representation sequence and the video feature sequence, extract the image features and semantic representations of each time step in the expression frame sequence, combine the features, and divide the image state, voice state and text state according to preset rules to construct a state label set. S53. Perform a mapping operation on the set of state labels and the set of facial expression parameters. At each time step, perform feature fusion processing on the state expression and the corresponding facial expression parameters, and construct a set of control instructions based on the fused features.

[0016] Optionally, S6 specifically includes: S61. Call the graphics rendering interface that controls the facial movements of the digital image. Based on the expression parameters and status labels of each time step in the control instruction set, perform position interpolation calculation and rotation posture construction operations on the bound bone nodes of the digital image's face to generate expression images for the corresponding time steps and construct an expression video sequence. S62. Calculate the image error of the facial expression video sequence and the facial expression frame sequence according to the time step, and construct the feedback error sequence; S63. Based on the feedback error sequence, the gradient propagation mechanism is used to update the parameters of the convolutional neural network and the face simulation framework.

[0017] The beneficial effects of this invention are: First, this invention constructs a unified processing mechanism for multimodal data. By standardizing the format, extracting features, and aligning the time of video images, speech audio, and text data respectively, it significantly improves the temporal consistency and semantic matching degree between different modal inputs, laying a high-quality feature foundation for subsequent expression generation and semantic rendering.

[0018] Secondly, lip-shape features are extracted using a convolutional neural network containing bidirectional gated recurrent units, and facial expression parameters are mapped and generated using a facial simulation framework based on the Audio2Face model. This is combined with the Unreal Engine to drive the digital image's facial skeleton nodes to achieve dynamic expression synthesis. At the same time, semantic parsing and status labels are integrated to construct rendering instructions, achieving accurate response to user intent and natural transition of expressions.

[0019] Finally, this invention introduces an error feedback mechanism. By calculating the error between the generated facial expression video sequence and the intermediate facial expression frame sequence, the network parameters and generation structure are continuously optimized, which effectively enhances the system's adaptive ability and the real-time performance and accuracy of the output facial expressions, and improves the overall intelligence level of the digital image interaction experience. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0021] Figure 1 This is a flowchart of a method for real-time conversion of digital facial expressions based on deep learning, as proposed in this invention. Figure 2 This is a flowchart of the expression parameter generation and structure optimization process for a real-time digital image facial expression conversion method based on deep learning proposed in this invention. Figure 3 This is a flowchart illustrating the closed-loop process of expression frame rendering and error feedback optimization for a real-time digital image facial expression conversion method based on deep learning proposed in this invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0023] refer to Figure 1-3 A method for real-time facial expression conversion of digital images based on deep learning includes the following steps: S1. Collect user interaction data, including video images, voice audio, and text commands, at a fixed frequency and preprocess it to generate a multimodal feature sequence; S2. Using a convolutional neural network containing bidirectional gated recurrent units, perform one-dimensional convolution and pooling operations on the multimodal feature sequence, extract the bidirectional state vector of the pooling result along the time dimension, and construct the lip shape feature sequence. S3. Based on the Audio2Face model, a facial simulation framework is built, and a low-dimensional mapping operation is performed on the lip shape feature sequence. Based on the mapping result, a continuous encoding sequence is constructed and an expression decoding operation is performed to obtain the expression parameter set. S4. Based on the set of facial expression parameters, a facial driving sequence is constructed using the Unreal Engine to control the skeletal nodes of the preset digital image face to perform position offset calculations and generate a sequence of facial expression frames. S5. Perform semantic parsing on the multimodal feature sequence, construct an intent expression set, and generate a state label set by combining the expression frame sequence. Perform a fusion mapping operation on the state label set and the expression parameter set to construct rendering instructions. S6. Drive the digital image face to render according to the rendering instructions, generate an expression video sequence and calculate the error with the expression frame sequence, and update the parameters of the convolutional neural network and the facial simulation framework based on the error results.

[0024] In this embodiment, the digital image represents a virtual character form structure constructed based on 3D modeling, including a facial mesh, a set of bound bone nodes, and material texture information. The bone nodes represent the bound control points in the facial mesh structure of the digital image used to perform facial expression deformation.

[0025] In this embodiment, S1 specifically includes: S11. Collect video image sequences, voice audio signals and text command data during user interaction at a fixed frequency, and perform format standardization processing to generate a set of video frames, audio signals and character sequences. S12. Perform grayscale normalization and histogram equalization on the video frame set, extract the face region and perform face alignment to construct the video feature sequence; S13. Perform frame segmentation and Hamming window weighting processing on the audio signal, extract spectral features using the Mel-Cepstral algorithm, and perform Z-score normalization on the spectral features to construct an audio feature sequence. S14. Perform word segmentation and encoding operations on the character sequence, and use a preset word embedding dictionary to convert the character fragments at each time step into fixed-length embedding vectors to construct the text feature sequence; S15. Perform time alignment processing on the video feature sequence, audio feature sequence and text feature sequence to construct a multimodal feature sequence.

[0026] In this embodiment, S13 specifically includes: S131. Perform frame division operation on the audio signal according to the set frame length and frame shift, and sample the Hamming window function in each frame to perform weighted processing on the time domain signal to generate a weighted time domain frame sequence. S132. Perform a fast Fourier transform on the weighted time-domain frame sequence to obtain the spectral information corresponding to each frame; S133. The sampling Mel filter bank processes the spectral information, compresses and transforms the frequency axis according to the human auditory perception scale, obtains the Mel power spectrum of each frame, performs logarithmic operation and discrete cosine transform on the Mel power spectrum, and extracts the Mel frequency cepstral coefficients as the spectral feature vector of each frame. S134. Arrange all spectral feature vectors in chronological order into a two-dimensional matrix, and perform Z-score standardization on each feature dimension, subtracting the mean of that dimension and dividing by the standard deviation of that dimension. S135. Construct an audio feature sequence from the standardized frame-level features according to time steps.

[0027] In this embodiment, the Fast Fourier Transform specifically includes: The length of each frame signal in the weighted time-domain frame sequence is uniformly processed. For frames that are not long enough, zero-padding is performed at the end to construct a frame sequence of equal length. Perform Fourier transform calculations on each frame signal in the equal-length frame sequence to obtain the amplitude response and phase response of that frame at different frequencies; Amplitude compression processing is performed on all amplitude responses of each frame to construct the corresponding spectral information; The spectral information of all frames is combined into a spectral feature set in chronological order.

[0028] In this embodiment, S2 specifically includes: S21. Using a convolutional neural network containing bidirectional gated recurrent units, the multimodal feature sequence is divided into multiple fixed-length sliding windows in chronological order, and a one-dimensional convolution operation is performed in each sliding window. The convolution kernel slides along the time dimension to extract local change features and generate a convolution response sequence. S22. Perform max pooling on the convolutional response sequence to compress local variation features along the time dimension and construct the pooling result sequence; S23. The positive and negative state vectors of the pooling result sequence are extracted along the time dimension by a bidirectional gated recurrent unit, and a vector concatenation operation is performed on the positive and negative state vectors. The concatenation results are arranged in time order to generate a lip shape feature sequence.

[0029] In this embodiment, the extraction process of the positive state vector and the negative state vector specifically includes: The pooling result sequence is input into the forward structure of the bidirectional gated recurrent unit in chronological order. At each time step, the pooling result vector is concatenated with the hidden state vector of the previous time step. Two rounds of linear transformation and Sigmoid function activation operations are performed on the concatenated result to generate update gate and reset gate respectively. In the first time step, the preset initialization vector is used as the hidden state vector of the previous time step. The retention ratio of the hidden state vector from the previous time step is adjusted based on the reset threshold, and then fused with the pooling result vector of the current time step to generate a candidate state vector. The candidate state vector and the hidden state vector of the previous time step are weighted and fused according to the update threshold to generate the hidden state vector of the current time step as the positive state vector. The pooling result sequence is input into the reverse structure of the bidirectional gated recurrent unit in reverse time order. At each time step, the pooling result vector is concatenated with the hidden state vector of the next time step, and the calculation of update gate and reset gate, candidate state vector generation and weighted fusion operations are performed to generate the reverse state vector. In the last time step, the preset initialization vector is used as the hidden state vector of the next time step. The forward state vector and the corresponding reverse state vector at each time step are concatenated along the feature dimension, and the concatenation results are arranged in chronological order to construct a lip shape feature sequence.

[0030] In this embodiment, S3 specifically includes: S31. A facial simulation framework is built based on the Audio2Face model. Temporal feature mapping is performed on the lip shape feature sequence according to the time step. Latent distribution parameters are extracted by variational autoencoder. S32. Perform reparameterization sampling based on the latent distribution parameters, construct the latent representation vector in time order, and generate a continuous coding sequence; S33. Input the continuous encoded sequence into the decoding path of the facial simulation framework, perform the expression feature reconstruction operation at each time step, and generate a set of expression parameters. S34. Perform adversarial discrimination operation on the expression parameter set and the preset parameter set along the time dimension through generative adversarial network, and update the parameters of the facial simulation framework according to the discrimination result; S35. Using the updated facial simulation framework, feature mapping, reparameterization sampling, and facial expression feature reconstruction operations are performed based on the lip shape feature sequence to regenerate the facial expression parameter set.

[0031] In this embodiment, the facial simulation framework is based on the Audio2Face model and includes a variational autoencoder and a generative adversarial network. The specific process of generating the expression parameter set includes: The lip shape feature sequence is input into the encoding path of the variational autoencoder, and one-dimensional convolution and GELU function activation operations are performed sequentially at each time step. Batch normalization is then performed on the activation results to generate latent distribution parameters. Parallel linear mapping paths are built based on each potential distribution parameter. At each time step, the mean vector and log-variance vector of the potential distribution parameters are generated to construct a set of distribution parameters. A pseudo-random perturbation vector following a standard normal distribution is introduced. For each pair of distribution parameters, a reparameterization sampling operation is performed, and the sampling results are processed by a moving average to generate a continuous encoded sequence. The continuous encoded sequence is input into the decoding path of the Audio2Face model. At each time step, a two-layer perceptron transformation is performed on the latent representation vector. The ReLU function and the Sigmoid function are used for nonlinear mapping, respectively. After each layer transformation, residual connection and batch normalization operations are performed to extract low-dimensional representations to generate a set of facial expression parameters. The facial expression parameters include the position offset of key skeletal control points and muscle weight values. The expression parameter set and the preset parameter set are constructed into sample pairs according to time steps and input into the discriminator structure of the generative adversarial network using a three-layer one-dimensional convolutional network. After each convolutional layer, the LeakyReLU function is activated and Dropout is processed, and the discrimination score of the final convolutional result is extracted through a fully connected network. With the goal of minimizing the discrimination score, an adversarial loss function is constructed, and the parameters of the encoding and decoding paths in the face simulation framework are iteratively updated using a backpropagation mechanism until the adversarial loss function converges. After the loss function converges, the updated facial simulation framework is used to perform latent representation extraction and expression feature decoding operations on the lip feature sequence at time steps to regenerate the expression parameter set.

[0032] In this embodiment, the reparameterization sampling operation specifically includes: The standard deviation vector is obtained by performing an exponential operation on the logarithmic variance vector and then taking the square root. Call the pseudo-random number generator to sample a random normal vector in the same dimension as the standard deviation vector; Perform element-wise multiplication on the random normal vector and the standard deviation vector, and perform element-wise addition on the mean vector to obtain the sampling encoding vector for the corresponding time step; All sampled coding vectors are concatenated in chronological order to generate a continuous coding sequence.

[0033] In this embodiment, S4 specifically includes: S41. In the preset driving structure of Unreal Engine, extract the position offset and muscle weight value of the key bone control points at each time step in the expression parameter set. S42. Based on the offset of the skeletal control point position, perform a node position update operation in the set of bound skeletal nodes of the preset digital image, and adjust the displacement amplitude and direction of the skeletal nodes in sync with the muscle control weight value. S43. After completing the update of the skeletal nodes at each time step, the driver calculation interface in the Unreal Engine is called. Based on the skeletal constraint rules and the position changes of the bound skeletal nodes, the facial mesh of the digital image is driven to perform facial expression deformation calculations, and keyframe indexes for the corresponding time step are generated. The skeletal constraint rules represent a set of predefined constraint logic used to limit and guide the spatial range, motion relationship and deformation mode of the bound skeletal nodes when controlling the facial deformation of the virtual character in the Unreal Engine. S44. Execute the rendering cache construction operation based on the keyframe index, and combine the bound bone node state and mesh deformation results of each frame to generate a facial rendering image and construct a frame cache sequence. S45. Organize the frame buffer sequence into an expression frame sequence in chronological order.

[0034] In this embodiment, S5 specifically includes: S51. Perform semantic parsing on the text features and audio features at each time step in the multimodal feature sequence to extract word meaning tags, tone categories and sentence structures, and construct a semantic representation sequence. S52. Perform time alignment processing on the semantic representation sequence and the video feature sequence, extract the image features and semantic representations of each time step in the expression frame sequence, combine the features, and divide the image state, voice state and text state according to preset rules to construct a state label set. S53. Perform a mapping operation on the set of state labels and the set of facial expression parameters. At each time step, perform feature fusion processing on the state expression and the corresponding facial expression parameters, and construct a set of control instructions based on the fused features.

[0035] In this embodiment, S6 specifically includes: S61. Call the graphics rendering interface that controls the facial movements of the digital image. Based on the expression parameters and status labels of each time step in the control instruction set, perform position interpolation calculation and rotation posture construction operations on the bound bone nodes of the digital image's face to generate expression images for the corresponding time steps and construct an expression video sequence. S62. Calculate the image error of the facial expression video sequence and the facial expression frame sequence according to the time step, and construct the feedback error sequence; S63. Based on the feedback error sequence, the gradient propagation mechanism is used to update the parameters of the convolutional neural network and the face simulation framework.

[0036] In this embodiment, S62 specifically includes: S621. Align the facial expression video sequence and the facial expression frame sequence according to time steps to construct an image pairing set; S622. Perform grayscale conversion and size normalization on each pair of images in the image pairing set to generate an aligned image set; S623. Perform pixel-by-pixel difference operation on each pair of aligned images, calculate the gray-level difference between the two images at each pixel position, and generate a set of pixel difference images. S624. Perform mean square error calculation on the pixel difference image set. Calculate the mean square value of all pixel positions in the pixel difference image at each time step to generate an error value sequence. S625. Arrange all error value sequences in chronological order to construct a feedback error sequence.

[0037] Example 1: To verify the feasibility of this invention in practice, it was applied to a real-time facial expression conversion task for a digital avatar in a virtual interactive system. The system uses a camera, microphone, and text input terminal to collect video, audio, and semantic information from the user during interaction, driving a pre-set digital virtual avatar to display facial expressions.

[0038] In real-world usage scenarios, users describe the scene via voice, accompanied by natural facial movements and lip-reading. The system needs to accurately understand the semantics, recognize the lip movements, and drive the digital avatar's facial expressions in real time based on tone, keywords, and emotional expression. For example, when a user says "This is amazing," the system should automatically generate surprised expressions such as an open mouth and raised eyebrows, maintaining a natural transition.

[0039] The system first acquires video images at a frequency of 25 frames per second, simultaneously acquiring 16kHz audio signals and obtaining user input text via keyboard or speech-to-text conversion. The acquired video image sequence undergoes face detection, alignment, and grayscale normalization to extract feature vectors and construct a video feature sequence. The audio signal is processed through frame segmentation, windowing, and MFCC extraction, and its spectral vector is obtained through Z-score normalization. The text input content is processed through word segmentation and word embedding to construct a text feature sequence. Finally, the three types of features are aligned at each time step to generate a multimodal feature sequence.

[0040] In this embodiment, the system employs a convolutional neural network structure containing bidirectional gated recurrent units to extract context state vectors from multimodal feature sequences and generate a lip-shape feature set. During the training phase, a training set of over 1000 annotated speech and facial expression data has been entered, including expression categories such as "smiling," "surprised," "confused," and "happy," generating corresponding expression parameter sets. The facial simulation framework is based on the Audio2Face model and incorporates a variational autoencoder to achieve feature compression, constructing continuous encoded sequences in the expression parameter space to control the facial movements of the digital avatar.

[0041] To achieve a more natural visual presentation, the system pre-defines a facial mesh model with bound skeletal nodes within the Unreal Engine. Node displacement is controlled by expression parameters to generate continuous expression frames. The semantic analysis section uses an LSTM semantic decoder to parse the multimodal sequences, extracting the intended meaning and combining it with emotion tags to generate rendering instructions consistent with the semantics. The final output video frames are then compared frame-by-frame with real-life facial expression footage, evaluated across multiple dimensions including lip alignment accuracy, expression synchronization delay, and facial muscle naturalness.

[0042] Table 1. Performance Evaluation Data of Facial Expression Transformation under Typical Sentences

[0043] As shown in Table 1, the method of this invention achieved a mouth alignment accuracy of over 93% in most sentences, and the expression generation delay remained within 100ms, ensuring no noticeable lag in real-world interactive experiences. The average score for expression naturalness was above 9.0, indicating that the generated muscle movements were smooth and consistent with human facial physiological characteristics. Figure 1 The high consistency scores indicate a good match between the semantic parsing module and the rendering instructions.

[0044] Furthermore, the system outputs facial animation frame sequences at a rate of 30 frames per second in the Unreal Engine rendering engine, which can meet the real-time requirements of conventional virtual human interaction and virtual live streaming applications. Compared with traditional virtual avatar systems based on rule mapping or static facial expression libraries, the method of this invention has higher facial expression accuracy, lower response latency, and stronger semantic matching capabilities.

[0045] In summary, this embodiment fully verifies the practicality and superior performance of the present invention in complex user interaction scenarios, and provides technical support for digital image driving and virtual human expression simulation in multimodal human-computer interaction.

[0046] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A deep learning-based digital avatar facial expression real-time conversion method, characterized in that, Includes the following steps: S1. Collect user interaction data, including video images, voice audio, and text commands, at a fixed frequency and preprocess it to generate a multimodal feature sequence; S2. Using a convolutional neural network containing bidirectional gated recurrent units, perform one-dimensional convolution and pooling operations on the multimodal feature sequence, extract the bidirectional state vector of the pooling result along the time dimension, and construct the lip shape feature sequence. S3. Based on the Audio2Face model, a facial simulation framework is built, and a low-dimensional mapping operation is performed on the lip shape feature sequence. Based on the mapping result, a continuous encoding sequence is constructed and an expression decoding operation is performed to obtain the expression parameter set. S4. Based on the set of facial expression parameters, a facial driving sequence is constructed using the Unreal Engine to control the skeletal nodes of the preset digital image face to perform position offset calculations and generate a sequence of facial expression frames. S5. Perform semantic parsing on the multimodal feature sequence, construct an intent expression set, and generate a state label set by combining the expression frame sequence. Perform a fusion mapping operation on the state label set and the expression parameter set to construct rendering instructions. S6. Drive the digital image face to render according to the rendering instructions, generate an expression video sequence and calculate the error with the expression frame sequence, and update the parameters of the convolutional neural network and the facial simulation framework based on the error results.

2. The digital avatar facial expression real-time conversion method based on deep learning according to claim 1, characterized in that, The digital image represents a virtual character form structure constructed based on 3D modeling, including a facial mesh, a set of bound skeletal nodes, and material texture information. The skeletal nodes represent the bound control points in the facial mesh structure of the digital image used to perform facial expression deformation.

3. The digital avatar facial expression real-time conversion method based on deep learning according to claim 1, characterized in that, S1 specifically includes: S11. Collect video image sequences, voice audio signals and text command data during user interaction at a fixed frequency, and perform format standardization processing to generate a set of video frames, audio signals and character sequences. S12. Perform grayscale normalization and histogram equalization on the video frame set, extract the face region and perform face alignment to construct the video feature sequence; S13. Perform frame segmentation and Hamming window weighting processing on the audio signal, extract spectral features using the Mel-Cepstral algorithm, and perform Z-score normalization on the spectral features to construct an audio feature sequence. S14. Perform word segmentation and encoding operations on the character sequence, and use a preset word embedding dictionary to convert the character fragments at each time step into fixed-length embedding vectors to construct the text feature sequence; S15. Perform time alignment processing on the video feature sequence, audio feature sequence and text feature sequence to construct a multimodal feature sequence.

4. The digital avatar facial expression real-time conversion method based on deep learning according to claim 1, characterized in that, S2 specifically includes: S21. Using a convolutional neural network containing bidirectional gated recurrent units, the multimodal feature sequence is divided into multiple fixed-length sliding windows in chronological order, and a one-dimensional convolution operation is performed in each sliding window. The convolution kernel slides along the time dimension to extract local change features and generate a convolution response sequence. S22. Perform max pooling on the convolutional response sequence to compress local variation features along the time dimension and construct the pooling result sequence; S23. The positive and negative state vectors of the pooling result sequence are extracted along the time dimension by a bidirectional gated recurrent unit, and a vector concatenation operation is performed on the positive and negative state vectors. The concatenation results are arranged in time order to generate a lip shape feature sequence.

5. The digital avatar facial expression real-time conversion method based on deep learning according to claim 4, characterized in that, The extraction process of the positive and negative state vectors specifically includes: The pooling result sequence is input into the forward structure of the bidirectional gated recurrent unit in chronological order. At each time step, the pooling result vector is concatenated with the hidden state vector of the previous time step. Two rounds of linear transformation and Sigmoid function activation operations are performed on the concatenated result to generate update gate and reset gate respectively. In the first time step, the preset initialization vector is used as the hidden state vector of the previous time step. The retention ratio of the hidden state vector from the previous time step is adjusted based on the reset threshold, and then fused with the pooling result vector of the current time step to generate a candidate state vector. The candidate state vector and the hidden state vector of the previous time step are weighted and fused according to the update threshold to generate the hidden state vector of the current time step as the positive state vector. The pooling result sequence is input into the reverse structure of the bidirectional gated recurrent unit in reverse time order. At each time step, the pooling result vector is concatenated with the hidden state vector of the next time step, and the calculation of update gate and reset gate, candidate state vector generation and weighted fusion operations are performed to generate the reverse state vector. In the last time step, the preset initialization vector is used as the hidden state vector of the next time step. The forward state vector and the corresponding reverse state vector at each time step are concatenated along the feature dimension, and the concatenation results are arranged in chronological order to construct a lip shape feature sequence.

6. The digital avatar facial expression real-time conversion method based on deep learning according to claim 1, characterized in that, S3 specifically includes: S31. A facial simulation framework is built based on the Audio2Face model. Temporal feature mapping is performed on the lip shape feature sequence according to the time step. Latent distribution parameters are extracted by variational autoencoder. S32. Perform reparameterization sampling based on the latent distribution parameters, construct the latent representation vector in time order, and generate a continuous coding sequence; S33. Input the continuous encoded sequence into the decoding path of the facial simulation framework, perform the expression feature reconstruction operation at each time step, and generate a set of expression parameters. S34. Perform adversarial discrimination operation on the expression parameter set and the preset parameter set along the time dimension through generative adversarial network, and update the parameters of the facial simulation framework according to the discrimination result; S35. Using the updated facial simulation framework, feature mapping, reparameterization sampling, and facial expression feature reconstruction operations are performed based on the lip shape feature sequence to regenerate the facial expression parameter set.

7. The digital avatar facial expression real-time conversion method based on deep learning according to claim 6, characterized in that, The facial simulation framework is based on the Audio2Face model and includes a variational autoencoder and a generative adversarial network. The specific process of generating the expression parameter set includes: The lip shape feature sequence is input into the encoding path of the variational autoencoder, and one-dimensional convolution and GELU function activation operations are performed sequentially at each time step. Batch normalization is then performed on the activation results to generate latent distribution parameters. Parallel linear mapping paths are built based on each potential distribution parameter. At each time step, the mean vector and log-variance vector of the potential distribution parameters are generated to construct a set of distribution parameters. A pseudo-random perturbation vector following a standard normal distribution is introduced. For each pair of distribution parameters, a reparameterization sampling operation is performed, and the sampling results are processed by a moving average to generate a continuous encoded sequence. The continuous encoded sequence is input into the decoding path of the Audio2Face model. At each time step, a two-layer perceptron transformation is performed on the latent representation vector. The ReLU function and the Sigmoid function are used for nonlinear mapping, respectively. After each layer transformation, residual connection and batch normalization operations are performed to extract low-dimensional representations to generate a set of facial expression parameters. The facial expression parameters include the position offset of key skeletal control points and muscle weight values. The expression parameter set and the preset parameter set are constructed into sample pairs according to time steps and input into the discriminator structure of the generative adversarial network using a three-layer one-dimensional convolutional network. After each convolutional layer, the LeakyReLU function is activated and Dropout is processed, and the discrimination score of the final convolutional result is extracted through a fully connected network. With the goal of minimizing the discrimination score, an adversarial loss function is constructed, and the parameters of the encoding and decoding paths in the face simulation framework are iteratively updated using a backpropagation mechanism until the adversarial loss function converges. After the loss function converges, the updated facial simulation framework is used to perform latent representation extraction and expression feature decoding operations on the lip feature sequence at time steps to regenerate the expression parameter set.

8. The digital avatar facial expression real-time conversion method based on deep learning according to claim 1, characterized in that, S4 specifically includes: S41. In the preset driving structure of Unreal Engine, extract the position offset and muscle weight value of the key bone control points at each time step in the expression parameter set. S42. Based on the offset of the skeletal control point position, perform a node position update operation in the set of bound skeletal nodes of the preset digital image, and adjust the displacement amplitude and direction of the skeletal nodes in sync with the muscle control weight value. S43. After completing the update of the skeletal nodes at each time step, the driver calculation interface in the Unreal Engine is called. Based on the skeletal constraint rules and the position changes of the bound skeletal nodes, the facial mesh of the digital image is driven to perform facial expression deformation calculations, and keyframe indexes for the corresponding time step are generated. The skeletal constraint rules represent a set of predefined constraint logic used to limit and guide the spatial range, motion relationship and deformation mode of the bound skeletal nodes when controlling the facial deformation of the virtual character in the Unreal Engine. S44. Execute the rendering cache construction operation based on the keyframe index, and combine the bound bone node state and mesh deformation results of each frame to generate a facial rendering image and construct a frame cache sequence. S45. Organize the frame buffer sequence into an expression frame sequence in chronological order.

9. The digital avatar facial expression real-time conversion method based on deep learning according to claim 1, characterized in that, S5 specifically includes: S51. Perform semantic parsing on the text features and audio features at each time step in the multimodal feature sequence to extract word meaning tags, tone categories and sentence structures, and construct a semantic representation sequence. S52. Perform time alignment processing on the semantic representation sequence and the video feature sequence, extract the image features and semantic representations of each time step in the expression frame sequence, combine the features, and divide the image state, voice state and text state according to preset rules to construct a state label set. S53. Perform a mapping operation on the set of state labels and the set of facial expression parameters. At each time step, perform feature fusion processing on the state expression and the corresponding facial expression parameters, and construct a set of control instructions based on the fused features.

10. A method for real-time conversion of digital facial expressions based on deep learning according to claim 1, characterized in that, S6 specifically includes: S61. Call the graphics rendering interface that controls the facial movements of the digital image. Based on the expression parameters and status labels of each time step in the control instruction set, perform position interpolation calculation and rotation posture construction operations on the bound bone nodes of the digital image's face to generate expression images for the corresponding time steps and construct an expression video sequence. S62. Calculate the image error of the facial expression video sequence and the facial expression frame sequence according to the time step, and construct the feedback error sequence; S63. Based on the feedback error sequence, the gradient propagation mechanism is used to update the parameters of the convolutional neural network and the face simulation framework.

Images