Speech-driven digital human body shape generation method based on latent space feature fusion

Abstract

The application discloses a speech-driven digital human mouth shape generation method based on latent space feature fusion, and belongs to the technical field of artificial intelligence and image synthesis; mainly improves the quality and time sequence continuity of a speech-driven digital human mouth shape generation image; the scheme of the application is that after modal encoding of speech audio and video images is carried out respectively, an image reconstruction process is guided by speech features in a latent space constructed by an image encoder, and a mouth shape change image frame sequence consistent with the speech features is generated; a complete process from user speech input to digital human response is realized, intelligent expression capability of the digital human in a human-computer interaction process is enhanced, and therefore more natural and intelligent digital human speech expression is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and image processing, and particularly relates to a speech-driven digital mouth shape generation method based on latent space feature fusion. BACKGROUND

[0002] Virtual digital people can replace the real person to generate virtual digital people teachers, virtual digital interviewers and other images of the real person. With the gradual application of virtual digital people in media communication, digital education, virtual images and other scenarios, speech-driven mouth shape image generation gradually becomes an important technical link to support speech synchronization expression. Existing methods usually build the association between speech and image to realize speech-driven mouth shape image generation, which to some extent improves the naturalness and synchronization of virtual people expression. However, there are still the following problems in the existing virtual digital people generation process: input processing is not standardized: the pre-processing of speech signals and images is insufficient in some methods, resulting in lack of uniformity of input data in time sequence and spatial scale, and poor generation effect; multi-modal feature extraction is insufficient: existing methods often fail to capture dynamic semantic information of speech and detailed structure of image at the same time, resulting in insufficient accuracy and naturalness of generated mouth shape; fusion mechanism is imperfect: the cooperation of speech and image features in latent space is insufficient, which easily causes frame discontinuity and incoherent mouth shape change of generated images; limited generation quality: in terms of detail restoration and overall consistency, the generated results of some methods have phenomena such as blur, distortion or incomplete matching with speech;

[0003] Therefore, in order to improve the naturalness and stability of speech-driven digital mouth shape generation, it is necessary to design an image generation method based on latent space feature fusion. SUMMARY

[0004] In order to overcome the shortcomings of the prior art and improve the quality and time sequence continuity of speech-driven digital mouth shape generation image, the application provides a speech-driven digital mouth shape generation method based on latent space feature fusion. The application enhances the dynamic change expression of the mouth shape area while ensuring the overall structural consistency through more efficient speech coding, image feature extraction and cross-modal fusion mechanism, thereby improving the quality and time sequence continuity of the finally generated image.

[0005] The speech-driven digital mouth shape generation method based on latent space feature fusion comprises the following steps:

[0006] Step S1: constructing a data set;

[0007] Collecting face speech videos, processing the face speech videos to obtain original speech audio and original video files;

[0008] Performing speech preprocessing on the original speech audio to obtain speech features;

[0009] performing image preprocessing on the original video file to obtain face bounding box information of each frame of image in the original video file, a mask image of each frame of image and an initial latent space feature of each frame of image;

[0010] based on all voice features, face bounding box information of all frames of image, mask images of all frames of image and initial latent space features of all frames of image, a data set is constructed;

[0011] Step S2: constructing a latent space fusion module; the output of the latent space fusion module is an image-voice cross-modal fusion feature ; the latent space fusion module is trained using the training set, and when the loss function After the region is stable for a plurality of iteration cycles and no longer appears large fluctuations, it is considered that the training of the latent space fusion module is completed, and the optimal latent space fusion module is obtained;

[0012] Step S3: processing the to-be-processed face voice video to obtain a to-be-processed voice audio and a to-be-processed video file respectively; performing voice preprocessing on the to-be-processed voice audio to obtain a to-be-processed voice feature; performing image preprocessing on the to-be-processed video file to obtain face bounding box information of each frame of image of the to-be-processed video file, a mask image of each frame of image of the to-be-processed video file and an initial latent space feature of each frame of image of the to-be-processed video file; inputting the to-be-processed voice feature, the face bounding box information of each frame of image of the to-be-processed video file, the mask image of each frame of image of the to-be-processed video file and the initial latent space feature of each frame of image of the to-be-processed video file into the optimal latent space fusion module to obtain an image-voice cross-modal fusion feature of the to-be-processed face voice video; inputting the image-voice cross-modal fusion feature of the to-be-processed face voice video into an image decoder to obtain an image frame of the to-be-processed face voice video; processing the image frame of the to-be-processed face voice video using an image fusion method to obtain a voice-driven lip change digital human video.

[0013] Further, the image fusion method is: according to the mask image of the lip region of each frame of image, replacing the lip region in the face bounding box information of each frame of image with a lip region of the image frame to obtain a voice-driven face fusion image; the lip region of the image frame is the pixel value of the image frame multiplied by the pixel value of the mask image corresponding to the image frame; arranging the obtained voice-driven face fusion image in sequence to obtain a fusion image sequence; using a video generation tool to generate a voice-driven lip change digital human video from the fusion image sequence.

[0014] Further, the image decoder is based on a decoder of a variational autoencoder (VAE); the output of the image decoder is an image frame; and the input of the image decoder is an image-voice cross-modal fusion feature.

[0015] Further, the step of performing speech preprocessing on the original speech audio to obtain speech features comprises:

[0016] performing standardization processing, slicing processing and padding processing on the original speech audio in sequence to obtain audio clips; inputting the audio clips into a feature encoding module to obtain speech features of each audio clip;

[0017] The standardization processing comprises format standardization and sampling rate standardization on the original speech audio, to ensure that the input data is structured and standardized.

[0018] The slicing processing comprises dividing the original speech audio after the standardization processing into a plurality of continuous sub-clips Audio Clips with consistent lengths.

[0019] The padding processing comprises, for the sub-clips obtained after the slicing processing, filling zero values for the sub-clips with incomplete lengths at the end to obtain audio clips with consistent lengths.

[0020] Further, the feature encoding module is a speech feature encoder based on a convolutional neural network CNN; the input of the feature encoding module is the audio clip; and the output of the feature encoding module is the speech feature.

[0021] Further, the process of performing image preprocessing on the original video file to obtain face bounding box information of each frame of image in the original video file, a mask image of each frame of image and an initial latent space feature of each frame of image comprises:

[0022] obtaining each frame of image of the original video file, converting each frame of image into a standard format and saving in time sequence, so that the image sequence can be correctly loaded and used subsequently; detecting each frame of image using a face detection algorithm to obtain face bounding box information; performing convolution processing on each frame of image with the obtained face bounding box information to obtain a mask image of a mouth area of each frame of image; cropping each frame of image with the obtained face bounding box information to obtain a face image of each frame of image; inputting the face image of each frame of image into an image auto-encoding module to obtain an initial latent space feature of each frame of face image; and the image auto-encoding module is an encoder of a variational auto-encoder VAE.

[0023] Further, the original speech audio is in a wav format.

[0024] Further, the latent space fusion module comprises a convolution module and an attention module; the convolution module is based on a U-Net network architecture; and the output of the convolution module is the image latent space feature.

[0025] ​The attention module is based on a cross-attention mechanism of a Transformer; the Query input of the attention module is an image latent space feature ; the input of Key and the input of Value of the attention module are both speech features A;

[0026] The attention mapping of the attention module is:

[0027] The image latent space feature and the speech feature A are respectively linearly transformed to obtain a query matrix Q, a key matrix K and a value matrix V;

[0028] The attention feature is calculated as:

[0029]

[0030] wherein Q, K and V are respectively a query matrix, a key matrix and a value matrix obtained by linearly transforming the image latent space feature and the speech feature , is a scaling factor.

[0031] The attention feature is added to the image latent space feature by a residual structure to obtain an image-speech cross-modal fusion feature :

[0032]

[0033] Through the residual structure, the original structural information of the image latent space is retained, and the fusion of the speech semantic information is realized, so that a balance between content expression and dynamic semantic guidance of the generated image is ensured.

[0034] Further, the loss function of the latent space fusion module is:

[0035]

[0036] wherein is an output result of the latent space fusion module, is a reconstruction target of the latent space fusion module, and the loss function is calculated by using an L2 norm.

[0037] The beneficial effects of the present application are: since the speech audio and video image are respectively modally encoded, the image reconstruction process is guided by the speech features in the latent space constructed by the image encoder, the sequence of lip shape change image frames consistent with the speech features is generated, the complete process from user speech input to digital human response is realized, the intelligent expression ability of the digital human in the human-computer interaction process is enhanced, and therefore more natural and intelligent digital human speech expression is realized; the generated results are evaluated by using the image quality evaluation index LPIPS, the average LPIPS value of the method of the present application is about 0.09, which is obviously reduced compared with the prior art, and the lower the LPIPS value, the more natural synchronization of the lip shape change and the speech signal, and the closer to the real portrait in vision. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The present application is a general flowchart of the method. DETAILED DESCRIPTION

[0039] The present application will be further described below in combination with the drawings and examples.

[0040] The speech-driven digital human lip shape generation method based on latent space feature fusion comprises the following steps:

[0041] Step 1: pre-processing the original speech audio;

[0042] The original speech audio is sequentially subjected to standardization processing, slicing processing and padding processing to obtain an audio segment; the audio segment is input into a feature encoding module to obtain the speech features of each audio segment;

[0043] The standardization processing is to standardize the format and sampling rate of the original speech audio, so as to ensure the structured and standardized input data;

[0044] The slicing processing is to divide the original speech audio after standardization processing into a plurality of continuous sub-segments Audio Clips with consistent length, so as to effectively reduce the video memory pressure and improve the batch inference efficiency;

[0045] The padding processing is to pad the sub-segments obtained after slicing processing with zero values for the sub-segments with incomplete length at the end to obtain audio segments with consistent length; the padding processing ensures that all the sub-segments meet the fixed length input requirement and avoids interruption of inference;

[0046] The original speech audio is in wav format;

[0047] The feature encoding module is a convolutional neural network (CNN) based speech feature encoder. The input of the feature encoding module is an audio segment, and the output of the feature encoding module is speech features. The feature encoding module extracts the features of the speech signal in the audio segment through convolutional operation, and retains the key information of the speech.

[0048] Step 2: Preprocessing the original video file;

[0049] Preprocessing the original video file;

[0050] The process of preprocessing the original video file is as follows: obtaining each frame of image of the original video file, converting each frame of image into a standard format, and saving in chronological order, so that the image sequence can be correctly loaded and used subsequently;

[0051] Using a face detection algorithm to detect each frame of image to obtain face bounding box information; using coordinates to identify the position of the face in each image; the face bounding box information is used to provide spatial constraints for subsequent face region cropping and feature extraction, ensuring that the processing is focused on the face region of the person, improving the image generation accuracy and efficiency;

[0052] Convolution processing the face bounding box information of each frame of image to obtain a mask image of the mouth area of each frame of image;

[0053] Cropping each frame of image with face bounding box information to obtain a face image of each frame of image;

[0054] Inputting the face image of each frame of image into an image auto-encoding module to obtain the initial latent space feature of each frame of face image;

[0055] The image auto-encoding module is a variational auto-encoder (VAE) encoder. The variational auto-encoder (VAE) encoder maps the input high-resolution image to the latent space to generate a compact latent representation;

[0056] The image auto-encoding module extracts local and global features of the image through multiple layers of convolution and nonlinear activation operation, while gradually reducing the spatial resolution and increasing the channel dimension in the downsampling process, and outputs the initial latent space feature. The initial latent space feature contains both the overall structure information of the image and the key local details. Through the image auto-encoding module, the input face image is converted into the initial latent space feature in the image auto-encoding module, so that the image information is efficiently represented;

[0057] Step 3: Using a latent space fusion module to perform multi-modal fusion on the speech features and the initial latent space features;

[0058] The latent space fusion module comprises a convolution module and an attention module; the convolution module is based on a U-Net network architecture; the output of the convolution module is an image latent space feature ;

[0059] The attention module is based on a cross-attention mechanism of a Transformer; the Query input of the attention module is the image latent space feature ; the Key / Value input of the attention module is the speech feature A;

[0060] The attention mapping of the attention module is:

[0061] The image latent space feature and the speech feature A are respectively linearly transformed to obtain a query matrix Q, a key matrix K and a value matrix V;

[0062] The attention feature is calculated as:

[0063]

[0064] wherein Q, K and V are respectively a query matrix, a key matrix and a value matrix obtained by linearly transforming the image latent space feature and the speech feature , is a scaling factor;

[0065] The output attention feature is added to the input image feature by a residual addition to obtain an image-speech cross-modal fusion feature :

[0066]

[0067] Through the residual structure, the original structure information of the image latent space is preserved, and the fusion of the speech semantic information is realized, so that a balance between content expression and dynamic semantic guidance of the generated image is ensured;

[0068] Step 4: latent space decoding and image fusion generation stage;

[0069] The image-speech cross-modal fusion feature is input into an image decoder; the image decoder is based on a decoder of a variational autoencoder VAE; the output of the image decoder is an image frame;

[0070] An image fusion method is used to process the image frame to obtain a digital human video with speech-driven lip shape changes;

[0071] The step of processing image frames by using an image fusion method to obtain a digital person video with speech-driven mouth shape changes is as follows:

[0072] According to the mask image of the mouth shape area of each image, the mouth shape area in the face bounding box information of each image is replaced by the mouth shape area of the image frame to obtain a speech-driven face fusion image; the mouth shape area of the image frame is the multiplication of the pixel value of the image frame and the pixel value of the mask image corresponding to the image frame; the obtained speech-driven face fusion image is arranged in order to obtain a fusion image sequence; and a video generation tool is used to generate a digital person video with speech-driven mouth shape changes from the fusion image sequence.

[0073] Loss function of latent space fusion module is:

[0074]

[0075] wherein, is the output result of the latent space fusion module, is the reconstruction target of the latent space fusion module, and the loss function is calculated by using an L2 norm; during the training process, when the value of the loss function is reduced and is stable in multiple iteration periods without large fluctuations, it is considered that the training of the latent space fusion module is completed, and the optimal latent space fusion module is obtained.

[0076] The present application verifies the speaker video and the speech segment which do not participate in the training. In order to quantify the generation quality, the present application uses the image quality evaluation index LPIPS commonly used in the generation model for testing. LPIPS (Learned Perceptual Image Patch Similarity) is an index for measuring the perceptual similarity of images. The lower the value, the smaller the gap between the generated image and the real image in the human visual system, and the better the generation quality. LPIPS is a learned image similarity measure. Its principle is to compare the differences of images in the deep neural network feature space to simulate the perception of human eyes to image differences, which has high objectivity and reference value.

[0077] In the speech-driven mouth shape generation task, a lower LPIPS means that the mouth shape change is more naturally synchronized with the speech signal, and is visually closer to the real person.

[0078] The present application is compared with the existing technology Wav2Lip method for speech-driven mouth shape generation. The LPIPS index comparison results are shown in Table 1.

[0079] Table 1: LPIPS comparison results

[0080]

[0081] The results show that the average LPIPS value of the generated image is about 0.09, indicating that the method can achieve a relatively ideal effect in terms of naturalness and consistency.

[0082] Compared with the prior art, the present application has the following advantages:

[0083] Input standardization: improve the adaptability and robustness of the system under different data scenarios;

[0084] Multi-modal fusion enhancement: effectively combine speech and image features to improve the accuracy and naturalness of generated mouth shapes;

[0085] Cross-modal attention guidance: achieve fine-grained mapping of speech dynamics and mouth shape changes;

[0086] Warm-up learning rate scheduling: significantly improve the stability of the training process and the convergence effect of the model;

[0087] High-quality decoding and local fusion: generate results that are real, coherent, and have no obvious artifacts;

[0088] Video-level continuity: ensure smooth transitions between frames and avoid jumps and distortions;

[0089] Objective index verification: evaluate the generation effect quantitatively through the LPIPS index, with an average LPIPS of about 0.09, further proving the effectiveness of the present application in visual quality.

[0090] Therefore, the present application can realize speech-driven dynamic generation of facial mouth shapes while ensuring consistency and naturalness of facial structures, and has a wide application prospect, and can be applied to virtual digital people, intelligent interaction, virtual education, immersive media and other fields.

Claims

1. A method for generating a voice-driven digital avatar based on latent space feature fusion, characterized in that, The method comprises the following steps: Step S1: constructing a data set; Collecting a face voice video, processing the face voice video, obtaining an original voice audio and an original video file; Performing voice preprocessing on the original voice audio to obtain voice features; Performing image preprocessing on the original video file to obtain face bounding box information of each frame of image, a mask image of each frame of image and initial latent space features of each frame of image in the original video file; Constructing a data set based on all voice features, face bounding box information of all frames of image, mask images of all frames of image and initial latent space features of all frames of image; Step S2: constructing a latent space fusion module; The output of the latent space fusion module is an image-speech cross-modal fusion feature ; The latent space fusion module is trained using the training set, and when a loss function of the latent space fusion module After the region is stable for multiple iteration cycles and no large fluctuations occur, it is considered that the training of the latent space fusion module is completed, and an optimal latent space fusion module is obtained. Step S3: processing a to-be-processed face voice video to obtain a to-be-processed voice audio and a to-be-processed video file respectively; performing voice preprocessing on the to-be-processed voice audio to obtain to-be-processed voice features; Performing image preprocessing on the to-be-processed video file to obtain face bounding box information of each frame of image in the to-be-processed video file, a mask image of each frame of image in the to-be-processed video file and initial latent space features of each frame of image in the to-be-processed video file; Inputting the to-be-processed voice features, face bounding box information of each frame of image in the to-be-processed video file, mask images of each frame of image in the to-be-processed video file and initial latent space features of each frame of image in the to-be-processed video file into the optimal latent space fusion module to obtain image voice cross-modal fusion features of the to-be-processed face voice video; Inputting the image voice cross-modal fusion features of the to-be-processed face voice video into an image decoder to obtain image frames of the to-be-processed face voice video; processing the image frames of the to-be-processed face voice video using an image fusion method to obtain a digital human video with voice-driven lip changes.

2. The voice-driven digital population generation method based on latent space feature fusion according to claim 1, characterized in that, The latent space fusion module comprises a convolution module and an attention module; the convolution module is based on a U-Net network architecture; the output of the convolution module is an image latent space feature ​ The attention module is based on a cross-attention mechanism of a Transformer; input of Query of the attention module is image latent space features ; input of Key and input of Value of the attention module are both speech features A; The attention mapping of the attention module is: Image latent space features respectively A linear transformation is performed on the speech feature A to obtain the query matrix Q, the key matrix K, and the value matrix V, respectively. Computing attention features : ； wherein Q, K and V are respectively a query matrix, a key matrix and a value matrix obtained by performing linear transformation on image latent space features and speech features , is a scaling factor; Attention feature Image latent space feature Residual addition is performed to obtain image-speech cross-modal fusion feature : ； Through the residual structure, the image latent space original structure information is retained, and the fusion of voice semantic information is realized, so that balance between content expression and dynamic semantic guidance of the generated image is ensured. 3.The voice-driven digital population generation method based on latent space feature fusion according to claim 1, characterized in that, The loss function of the latent space fusion module Is: ； wherein, is the output result of the latent space fusion module, is the reconstruction target of the latent space fusion module, and the loss function is calculated using an L2 norm.

4. The voice-driven digital population generation method based on latent space feature fusion according to claim 1, characterized in that, The image fusion method is: according to the mask image of the lip region of each frame of image, the lip region in the face bounding box information of each frame of image is replaced with the lip region of the image frame to obtain a voice-driven face fusion image; the lip region of the image frame is the pixel value of the image frame multiplied by the pixel value of the mask image corresponding to the image frame; Arranging the obtained voice-driven face fusion images in sequence to obtain a fusion image sequence; using a video generation tool to generate a digital human video with voice-driven lip changes from the fusion image sequence.

5. The voice-driven digital population generation method based on latent space feature fusion according to claim 1, characterized in that, The image decoder is based on a decoder of a variational autoencoder (VAE); the output of the image decoder is an image frame; and the input of the image decoder is an image voice cross-modal fusion feature.

6. The latent space feature fusion-based speech-driven digital avatar generation method according to claim 1, characterized in that, The step of performing voice preprocessing on the original voice audio to obtain voice features is: Performing standardization processing, slicing processing and padding processing on the original voice audio in sequence to obtain an audio segment; inputting the audio segment into a feature encoding module to obtain voice features of each audio segment; The standardization processing is format standardization and sampling rate standardization on the original voice audio to ensure that the input data is structured and standardized. The slicing processing is to divide the original speech audio after the standardization processing into a plurality of continuous sub-clips Audio Clips with consistent lengths. The padding processing is to pad the sub-clips obtained after the slicing processing with zero values for the sub-clips with incomplete lengths, so as to obtain audio clips with consistent lengths.

7. The latent space feature fusion-based speech-driven digital avatar generation method according to claim 6, characterized in that, The feature encoding module is a speech feature encoder based on a convolutional neural network CNN. The input of the feature encoding module is an audio clip, and the output of the feature encoding module is speech features. 8.The voice-driven digital population generation method based on latent space feature fusion according to claim 1, characterized in that, The process of performing image preprocessing on the original video file to obtain face bounding box information of each frame of image, a mask image of each frame of image and initial latent space features of each frame of image in the original video file is as follows: Each frame of image of the original video file is obtained, each frame of image is converted into a standard format and saved in time sequence, so that the image sequence can be correctly loaded and used subsequently; a face detection algorithm is used to detect each frame of image to obtain face bounding box information; Each frame of image with the obtained face bounding box information is subjected to convolution processing to obtain a mask image of a mouth area of each frame of image; Each frame of image with the obtained face bounding box information is cropped to obtain a face image of each frame of image; the face image of each frame of image is input into an image auto-encoding module to obtain initial latent space features of each frame of face image; the image auto-encoding module is an encoder of a variational auto-encoder VAE. 9.A terminal device, comprising a processor, a memory, and a computer program stored in the memory; characterized in that, The processor executes the computer program to implement the speech-driven digital mouth shape generation method based on latent space feature fusion according to any one of claims 1-8.

10. A computer readable storage medium having stored therein a computer program; characterized in that, The computer program is executed by the processor to implement the speech-driven digital mouth shape generation method based on latent space feature fusion according to any one of claims 1-8.

Images