A method for automatically generating lip shape and expression by inputting speech

Abstract

The present invention belongs to the field of speech processing technology, and relates to a method for automatically generating lip shape and expression from input speech, including: extracting features from the input speech to obtain an audio feature sequence; constructing an audio-generated video model, including: inputting speech videos of human faces in different languages ​​and regions into a Diffusion Transformer model that uses neural dynamic time warping, attention-based time alignment, and audio and video time position coding synchronization, and training to obtain an audio-generated video model; inputting the audio feature sequence into the audio-generated video model to obtain a lip shape and expression synchronization video. The present invention can generate corresponding lip shape and expression synchronization videos based on the input speech, can improve user experience, further improve the synchronization accuracy of the generated video, improve the real-time processing capability of the model, and the generated lip shape and expression synchronization videos are more natural, can work across languages ​​and cultural backgrounds, and adapt to content requirements in different regions around the world.

Description

Technical Field

[0001] The invention belongs to the technical field of speech processing, and in particular relates to a method for automatically generating lip shapes and expressions by inputting speech. Background Art

[0002] Currently, audio and video synchronization technology has been applied in many fields and has made certain progress, but there is still much room for improvement in synchronization accuracy, real-time processing capabilities, natural interaction simulation, and voice adaptability in different regions around the world. Summary of the invention

[0003] In order to solve the above problems in the prior art, the present invention provides a method for automatically generating lip shapes and expressions by inputting speech, aiming to provide a more efficient, more natural and more adaptable audio and video synchronization solution.

[0004] The present invention provides a method for automatically generating lip shape and expression from input speech, comprising: extracting features of the input speech to obtain an audio feature sequence; inputting the audio feature sequence into a sound-generated video model to obtain a lip shape and expression synchronization video; constructing the sound-generated video model, comprising: inputting face speech videos of different languages ​​and regions into a Diffusion Transformer model that uses neural dynamic time warping, attention-based time alignment, and audio and video time position coding synchronization, and training to obtain the sound-generated video model. The Diffusion Transformer model is a composite model that adds a Diffusion denoising model to the Transformer framework.

[0005] Specifically, feature extraction is performed on the input speech to obtain an audio feature sequence, including: frequency domain filtering of the digital audio data of the input speech to obtain a denoised audio signal; the denoised audio signal is input into a wav2vec model composed of an encoder network and a context network to extract the audio feature sequence.

[0006] Specifically, the encoder network or context network is a convolutional neural network based on the Transformer architecture.

[0007] Specifically, the denoised audio signal is input into a wav2vec model composed of an encoder network and a context network to extract an audio feature sequence, including: converting the denoised audio signal into a latent representation code in a latent space through the encoder network, wherein the latent representation code at each time step includes information of a current time point and a previous time point; inputting the latent representation code into the context network to extract context information of multiple time steps and generate a context representation code; and determining the audio feature sequence based on the context representation codes of multiple consecutive time steps.

[0008] Specifically, face speech videos of different languages ​​and regions are input into a Diffusion Transformer model that uses neural dynamic time warping, attention-based time alignment, and audio and video time position coding synchronization, and an audio-video model is trained, including: face speech videos of different languages ​​and regions are input into a Diffusion Transformer model to learn the mapping relationship between audio features and video frames; the Diffusion Transformer model aligns the audio features and video features of the video using neural dynamic time warping and attention-based time alignment algorithms; the audio and video are synchronized by adding audio and video time position coding to each layer of the Diffusion Transformer model, wherein the audio time position coding represents the position of the audio segment on the generated video timeline, and the video time position coding represents the position of the video frame on the generated video timeline.

[0009] Specifically, face speech videos in different languages ​​and regions are input into a Diffusion Transformer model to learn the mapping relationship between audio features and video frames, including: compressing face speech videos in different languages ​​and regions and extracting spatiotemporal patches; inputting the spatiotemporal patches and voice information of each frame of video into the Diffusion Transformer model; aligning the voice information of each frame of video with the spatiotemporal patches; connecting audio features of nearby frames to determine the audio features of a generated frame, wherein the nearby frames are video frames whose time distance from the generated frame does not exceed a preset value; adding an audio attention layer to the Diffusion Transformer model to inject the audio features of the generated frame into the generated video.

[0010] Specifically, the Diffusion Transformer model aligns the audio features and video features of the video by using neural dynamic time warping and attention-based time alignment algorithm, including: dynamically time warping the neural network of the Diffusion Transformer model, and aligning the audio features with the video features for the first time by learning nonlinear transformation; calculating the similarity matrix between the audio features and the video features, and according to the similarity matrix, the Diffusion Transformer model learns the alignment path of the audio features and the video features; calculating the attention weights between the audio features and the video features, and according to the attention weights, the Diffusion Transformer model dynamically adjusts the degree of attention paid to the audio features and the video features at each time step; and according to the alignment path and the degree of attention paid by the Diffusion Transformer model to the audio features and the video features, the Diffusion Transformer model aligns the audio features and the video features for the second time.

[0011] Specifically, constructing an audio-video model also includes: using a multi-head self-attention algorithm to synchronize the temporal features, spatial features, and audio features between video frames with the temporal features, spatial features, and audio features as independent dimensions.

[0012] Specifically, constructing the audio-video model also includes: during the Diffusion Transformer model training process, introducing sample noise into the generated video, and removing the sample noise in the generated video, wherein the sample noise includes image noise and audio noise.

[0013] Specifically, constructing the audio-video model also includes: inputting videos of different time lengths and aspect ratios into the Diffusion Transformer model for training.

[0014] Beneficial effects of the present invention:

[0015] (1) The present invention can generate corresponding lip-synced videos based on input speech, and can improve user experience in various digital audio and video industry application scenarios such as online video platforms, virtual anchors, video production, virtual reality, augmented reality, game development, distance education, and video conferencing.

[0016] (2) The present invention further improves the audio and video synchronization accuracy of the generated lip-synced expression video, can process the input speech in real time, and the generated lip-synced expression video is more natural.

[0017] (3) The present invention can be applied to voice input in different languages ​​and regions around the world, and can generate lip-synced expression videos corresponding to the characteristics of similar voices, thereby enhancing the adaptability of synchronization technology, allowing it to work across languages ​​and cultural backgrounds and meet the content needs of different regions around the world.

[0018] Other features and advantages of the present invention will be described in the following description. The purpose and other advantages of the present invention can be realized and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0020] Figure 1 The present invention provides a method for automatically generating lip shapes and facial expressions by inputting speech. DETAILED DESCRIPTION

[0021] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0022] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] The embodiment of the present invention provides a method for automatically generating lip shape and expression by inputting speech. In order to explain the present invention more clearly, the following is combined with Figure 1 Each step in the embodiment of the present invention is described in detail, including steps S1 to S3, and each step is described in detail as follows:

[0024] S1: Extract features from the input speech to obtain an audio feature sequence.

[0025] In some specific implementations, feature extraction is performed on the input speech to obtain an audio feature sequence, including: frequency domain filtering the digital audio data of the input speech to obtain a denoised audio signal; inputting the denoised audio signal into a wav2vec model composed of an encoder network and a context network to extract the audio feature sequence.

[0026] For example, an audio signal is received by a microphone array and converted from analog to digital to obtain a digital audio signal. The digital audio signal is pre-processed using frequency domain filtering technology to remove environmental noise and non-speech components.

[0027] In some specific embodiments, the encoder network or context network is a convolutional neural network (CNN) based on the Transformer architecture. Transform is a transformation model based on deep learning, which is suitable for processing data sequences or time series data, and has shown good performance in processing complex time series data and multimodal data. Convolutional neural networks are a type of feedforward neural network that includes convolution calculations and has a deep structure, which can extract data features.

[0028] In some specific implementations, the denoised audio signal is input into a wav2vec model composed of an encoder network and a context network to extract an audio feature sequence, including: converting the denoised audio signal into a latent representation code of a latent space through the encoder network, wherein the latent representation code of each time step includes information of the current time point and the previous time point; inputting the latent representation code into the context network, extracting multiple context time step information, and generating a context representation code; determining the audio feature sequence according to the context representation codes of multiple consecutive time steps. For example, the latent space of a video is represented as , represents the latent representation of a video frame, Indicates the corresponding video frame, F , H , W and Respectively represent the number of video frames to be changed, the height, width and number of channels of the video frames. Converted into a token sequence, where the total number of tokens is , Indicates the number of independent frames, Indicates the height of the video frame. Indicates the width of the video frame. Indicates depth.

[0029] Through the above wav2vec model, the denoised audio signal is mapped to a feature vector. The role of the encoder network is to convert the audio data into a latent representation encoding in the latent space. This representation is achieved by learning the audio data features in the context of a language model. In this context, the latent space representation of each time step contains not only the information of the current time point, but also the information of the previous time step. The role of the context network is to further process the latent space representation provided by the encoder network to generate a contextual representation. These contextual representation encodings contain a wider receptive field and can therefore contain information from multiple time steps, thereby enabling a more comprehensive understanding of the contextual relationship of the audio signal.

[0030] In order to enhance the naturalness of the generation, the speed information (such as speaking speed) extracted from the audio data is embedded into the feature representation through the speed encoder. A , assuming the speed encoder is , then the speed characteristic S Expressed as . The speed of speech is controlled by weighted summation method, specifically, , F is the audio feature, α is the weight factor used to control the influence of speed information, It is the audio feature after adjusting the speaking speed.

[0031] A regularization term is set in the loss function of the wav2vec model to prevent the wav2vec model from overfitting. For example, the loss function of the wav2vec model is:

[0032] in, L k Indicates k The amount of data point loss, T is the total time step length of the audio feature sequence, i and k represents the time step number, is the sigmoid activation function, is the context network at time step i The resulting hidden layer represents the encoding, is at the time step The corresponding audio feature vector, is a regularization parameter used to balance the two different loss parts. It represents the expected value operation, which represents the average under a certain probability distribution. The first part of the loss function of the wav2vec model It measures the model's ability to predict future time steps. The performance of the characteristics of is a regularization term to prevent the model from overfitting.

[0033] S2: Construct an audio-visual model.

[0034] The descriptive information of the image (for example, the picture is what a middle-aged man looks like) is encoded as a descriptive feature vector.

[0035] The description information (for example, "the picture is what a middle-aged man looks like") is preprocessed, including word segmentation, stop word removal, and other operations. in is the preprocessed text description, Tis the information of the original text. The pre-trained text encoding model (BERT) is used to encode the pre-processed text description into a feature vector. ,in V d is the text description vector.

[0036] The method includes: inputting face speech videos of different languages ​​and regions into the Diffusion Transformer model that uses neural dynamic time warping, attention-based time alignment, and audio and video time position encoding synchronization to train the audio-video model. The Diffusion Transformer model is a composite model that adds a Diffusion denoising model to the Transformer framework, referred to as DiT. The collection of face speech videos includes: collecting a large number of short video clips containing people's heads speaking from the Internet in accordance with the law. These videos contain rich character information, including but not limited to appearance, gender, age, facial expressions, etc. The GPT-4 model is used to generate detailed description information for each video. The description information includes the appearance characteristics, gender, age group, facial expressions, head posture, etc. of the person. These description information is automatically generated through natural language processing technology to ensure its accuracy and richness. The GPT-4 model generates detailed text descriptions by analyzing the video. These description information not only covers basic character features, but also includes more specific and detailed features, such as facial contours, details of eyes and mouth, and slight changes in facial expressions. In this way, the richer the details of the description information, the more specific and rich the definition of the character information in the subsequently generated video. The collected videos and generated description information are preprocessed, including video frame extraction, audio synchronization, text encoding of description information, etc. The preprocessed data will be used as input for model training. The final constructed dataset contains the following information for each video clip: video frame sequence, corresponding audio data, and detailed character description information (generated by GPT-4).

[0037] The Diffusion Transformer model is based on the Transformer architecture. The Diffusion Transformer model efficiently learns the complex mapping relationship between audio features and video frames through the self-attention mechanism. This architecture ensures that every detail in the audio can be reflected in the generated video frames, thereby achieving high-quality synchronized output.

[0038] In some specific implementations, face speech videos in different languages ​​and regions are input into a Diffusion Transformer model that uses neural dynamic time warping, attention-based time alignment, and audio and video time position coding synchronization to train an audio-video model, including: face speech videos in different languages ​​and regions are input into a Diffusion Transformer model to learn the mapping relationship between audio features and video frames; the Diffusion Transformer model aligns the audio features and video features of the video using neural dynamic time warping and attention-based time alignment algorithms; the audio and video are synchronized by adding audio and video time position coding to each layer of the Diffusion Transformer model, wherein the audio time position coding represents the position of the audio segment on the generated video timeline, and the video time position coding represents the position of the video frame on the generated video timeline.

[0039] In some specific implementations, face speech videos in different languages ​​and regions are input into a Diffusion Transformer model to learn the mapping relationship between audio features and video frames, including steps S21 and S22.

[0040] S21: Compress the face speech videos of different languages ​​and regions and extract the spatiotemporal patches; input the spatiotemporal patches and the voice information of each frame of video into the Diffusion Transformer model; align the voice information of each frame of video with the spatiotemporal patches to ensure the consistency of video frames and voice information in time and space.

[0041] S22: Connect the audio features of the nearby frames to determine the audio features of the generated frame.

[0042] Aligned video frames, using appearance encoder to obtain the volume features of characters (spatial position distribution of nose, eyes, etc.) and global description (Overall facial information, facial expressions, gender, age group, etc.), extract the head posture of the corresponding frame number through the motion encoder and facial expressions .

[0043] The volume features, global description, head posture and facial expression information are unified through the attention mechanism to obtain a unified feature representation code .

[0044] The pronunciation and intonation in the speech are the main driving signs for generating character actions. The audio features extracted from the input speech are connected through the various blocks of the pre-trained wav2vec. The speech may be affected by the previous or subsequent audio clips. Therefore, the speech features of each generated frame are defined by connecting the features of nearby frames.

[0045] Among them, the nearby frames are video frames whose time distance from the generated frame does not exceed a preset value; an audio attention layer is added to the DiffusionTransformer model to inject the audio features of the generated frame into the generated video.

[0046] For example, . Indicates that at time step f Output the sequence or output generated by the Diffusion Transformer model. Represents an aggregation function or other operation applicable to the Transformer model to integrate information from input data points. arrive Indicates from fm arrive f+m The data points at the time step are Output processed by the function.

[0047] During decoding, the size of the generated video is controlled by arranging randomly initialized spatiotemporal patches in a grid of appropriate size.

[0048] The Transformer network not only learns the mapping of audio features to video features, but also ensures the consistency of this mapping in the time dimension. Special time alignment techniques and attention mechanisms are added to ensure that the model can identify and align each moment in the audio signal with its corresponding video frame.

[0049] In some specific implementations, the Diffusion Transformer model aligns the audio features and video features of the video using neural dynamic time warping and attention-based time alignment algorithms, including steps S23 to S26.

[0050] S23: Dynamic time warping of the neural network of the Diffusion Transformer model is performed to align the audio features with the video features for the first time by learning nonlinear transformations.

[0051] In lip synchronization based on voice input, the main technical problem is how to accurately identify and align each moment in the audio signal with its corresponding video frame. This involves two challenges. On the one hand, there is the problem of time alignment: the audio signal and the video frame are not synchronized in time, and the traditional alignment method is not sufficient to handle complex timing changes, resulting in the generated lip-sync video being unnatural and out of sync. On the other hand, there is the problem of feature attention: when processing multimodal data of audio and video, how to ensure that the model can effectively focus on the important features between the two and perform appropriate feature fusion. This embodiment adopts Neural Dynamic Time Warping (N-DTW), combined with the dynamic time warping method of neural network, to align audio and video features by learning nonlinear transformations, making the alignment process more flexible and precise.

[0052] S24: Calculate the similarity matrix between the audio features and the video features. Based on the similarity matrix, the Diffusion Transformer model learns the alignment path of the audio features and the video features. Attention-Based Time Alignment uses the attention mechanism to directly align the audio and video features. By calculating the similarity matrix between the audio features and the video features, the Diffusion Transformer model can learn a better alignment path.

[0053] S25: Calculate the attention weights between the audio features and the video features. According to the attention weights, the Diffusion Transformer model dynamically adjusts the degree of attention paid to the audio features and the video features at each time step.

[0054] Based on cross-modal attention, a cross-modal attention algorithm is designed. By calculating the attention weights between audio features and video features, the Diffusion Transformer model can dynamically adjust the degree of attention to audio features and video features to achieve more accurate alignment and fusion.

[0055] S26: According to the alignment path and the degree of attention paid by the Diffusion Transformer model to the audio features and the video features, the Diffusion Transformer model performs a second alignment on the audio features and the video features.

[0056] Using N-DTW and attention-based time alignment methods, the alignment accuracy of audio and video on the timeline can be significantly improved. The Diffusion Transformer model can better handle complex timing changes, and the generated lip-sync videos are more natural and synchronized.

[0057] In addition, the Multi-Head Self-Attention algorithm can be used at the same time to perform attention fusion and dynamically adjust the attention weights of audio features and video features to achieve accurate content matching and precise time alignment, so that the Diffusion Transformer model can focus on different features in audio and video signals at the same time, thereby capturing more diverse temporal dependencies.

[0058] Dynamically adjusting the attention weights of audio and video features enables the Diffusion Transformer model to effectively capture important features and perform appropriate feature fusion when processing multimodal data. The generated lip-sync video is not only synchronized in time, but also accurately matches the audio signal in content.

[0059] The above-mentioned attention algorithm is used to automatically adjust according to the rhythm of the audio and the dynamic changes of the video to ensure the naturalness and smoothness of the synchronized output.

[0060] In lip synchronization based on speech input, the main technical problem is how to accurately handle the temporal dependency between audio and video. Audio and video have complex temporal relationships, and traditional models find it difficult to effectively capture these relationships, resulting in the generated lip-synced videos being unsynchronized and unnatural.

[0061] In order to deal with the temporal dependency between audio and video, the method is to introduce a special temporal modeling method in the Diffusion Transformer model. This method adds parameters for the temporal relationship to each layer of the Diffusion Transformer model, so that the model can take into account its position and role in the entire sequence when processing each audio clip. Such a design not only enhances the model's ability to understand the time series, but also further improves the accuracy and naturalness of audio and video synchronization. Specifically, this embodiment introduces time position coding in the Diffusion Transformer model. Time position coding is added to the Diffusion Transformer model to clearly indicate the position of each audio clip and video frame on the time axis. The time position coding can be generated by sine and cosine functions, or obtained by learning, so that the Diffusion Transformer model can better capture the timing information. By introducing the time position coding, the following technical effects can be achieved: time position coding and time convolution enhance the Diffusion Transformer model's understanding of local and global time relationships, making the alignment of audio and video on the time axis more accurate.

[0062] In some specific implementations, constructing an audio-video model also includes: using a multi-head self-attention algorithm, taking the temporal features, spatial features and audio features of video frames as independent dimensions, and synchronizing the temporal features, spatial features and audio features between video frames. By adding the independent dimension of audio features, when capturing the temporal relationship related to the generation of video frames, this embodiment not only considers the temporal features and character features between video frames, but also introduces audio features to enhance the Diffusion Transformer model's ability to synchronize speech and visual content. Through this integrated approach, the model can understand the dynamic relationship between video frames and speech signals at a deeper level, thereby generating a more natural, more coherent and more realistic video sequence. Among them, spatial features refer to the image features of video frames.

[0063] In some specific implementations, constructing an audio-video model further includes: during the Diffusion Transformer model training process, introducing sample noise into the generated video, and removing the sample noise in the generated video, wherein the sample noise includes image noise and audio noise. The effect of this method is that it can effectively eliminate the noise and unnatural changes that may be generated during the video generation process while maintaining the quality of the video content. Each step of denoising is performed under the guidance of the Diffusion Transformer model, ensuring that the final generated video frame is not only visually clear and natural, but also highly synchronized with the input audio signal.

[0064] For example, based on the diffusion model, a denoising model is established in the Transformer framework, Gaussian noise is introduced into the latent representation code, perturbation samples are generated, the denoising model is trained, and a smaller noise is predicted, and the denoising is completed by iteration. After a series of denoising steps, the final video is restored and a clear and natural video frame is decoded from the encoding of the latent space. Through multiple iterative optimizations, it is ensured that the final generated video frame is not only visually clear and natural, but also achieves high-quality synchronization with the input audio signal. Through the video decoder, the final continuous video data is generated according to the denoised features. Among them, the perturbation sample is , represents a perturbation sample at time t following multiple Markov chains, As a noise scheduler, where t represents the diffusion time step, z represents the potential code of the video frame, and the perturbation samples are generated by z. is a non-zero adjustment parameter, represents a standard normal distribution with a mean of 0 and a standard deviation of 1.

[0065] By training denoising, the prediction gets less noise:

[0066] ,in, represents a parameterized probability distribution function, where the parameter set is θ , represents a Markov chain that follows a time step t The perturbation sample of Indicates that at time step t -1 denoising sample, The mean of a normal distribution is Function of Represents the time step t The disturbance about The covariance of the distribution of . Here Using the denoising model Achieve and train with simple goals.

[0067] right After t-1 times of denoising, the denoised sample z0 is obtained. z0 represents the denoised sample, which is obtained by the denoising model Restored from a noisy sample. In the inference phase, by iteratively applying the denoising model, a noisy initial sample can be gradually denoised to restore a clean sample z0, which represents the final generated image or data. is a denoising model that is used to predict noise. In the diffusion model, the goal of the training process is to and time step t , predict as accurately as possible the noise added to the original data . is the noise drawn from the standard normal distribution N(0,1) and is a constant.

[0068] The variational lower bound reduces the log-likelihood to

[0069] ,in, Represents a set of parameters The loss function is usually to be minimized. It is the Kullback-Leibler divergence, which measures the difference between two probability distributions. represents the parameterized probability distribution function, is a method for estimating the true distribution The variational distribution of Indicates that at time step t perturbation samples of ; Indicates that at time step t -1 perturbation sample, z0 represents the denoised sample.

[0070] Defining the loss function

[0071] ,in, Represents a simplified version of the loss function used in the training phase to evaluate and optimize model performance; Represents the probability distribution Extract samples from x ; Represents model parameterization, with parameters θ The error prediction function is defined as, x Represents a sample, represents the probability distribution, is a non-zero adjustment constant. The output of is used to calculate the loss function and is optimized during the training process. Identify the noise removed in each step, remove the noise subsequently, and finally get the image z0 after the noise is removed. Express expectations.

[0072] In another specific embodiment of the present invention, the present invention provides a method for automatically generating lip shape and expression from input speech, comprising: extracting corresponding audio features from audio data through an audio encoder (such as wav2vec); embedding speed information (such as speech rate) extracted from audio data using a speed encoder into feature representation to enhance the naturalness of generation; extracting brief description information of the image (for example, the picture shows a middle-aged man) and encoding it into a description feature vector. Initialize initial noise data representing a random potential representation of facial features; obtain the volume features (spatial position distribution of nose, eyes, etc.) and global description (overall information of the face, facial expression, gender, age group, etc.) of the character through an appearance encoder for the input video frame; extract the head posture and facial expression of the corresponding frame number through a motion encoder; align the audio features and other features in time to ensure that the audio features match the features of the video frame at the same time step. Volume features, global descriptions, head postures and facial expression features form a unified feature representation; the unified features, audio features, and description feature vectors are input into the Diffusion Transformer model, and the noise in the potential features is gradually reduced by iteratively applying the network structure; in each iteration, the self-attention mechanism is used to enhance the model's understanding of the internal correlation of the input features to better focus on important features; the reference attention mechanism is used to compare the generated features with the reference image features to ensure that the generated facial animation is consistent with the reference data in expression and details; the denoised features are used to generate the final video data through the video decoder; dynamic video clips with lip shape and expression are generated, and the target video data is output. In this embodiment, since the description information and unified feature representation are added to the model, the model's understanding of the appearance and gender characteristics of the characters is enhanced, so the appearance and gender characteristics of the output video can be redefined by modifying the description information to generate a new video version. The speed encoder in this embodiment is to count the number of words or characters in the speech per unit time from the audio feature sequence to obtain a feature vector representing the speaking speed, which is encoded as speed information. The appearance encoder in this embodiment extracts the volume features and global description of the video frame through the graph neural network and encodes them. The motion encoder in this embodiment extracts the head posture and facial expression of the corresponding frame number through the image convolutional neural network, and encodes them.

[0073] In some specific implementations, constructing the audio-video model further includes: inputting videos of different time lengths and aspect ratios into the Diffusion Transformer model for training. Training videos and images of different durations and aspect ratios can adapt the Diffusion Transformer model to various video input formats, facilitate inputting face speech videos of different languages ​​and regions into the Diffusion Transformer model, and improve training efficiency.

[0074] S3: Input the audio feature sequence into the audio-generated video model to obtain the lip-synced expression video.

[0075] In some specific implementations, the audio features are integrated and the unified feature representation code is encoded. and description information V d The generated features are input into the Diffusion Transformer model backbone network together. By iteratively applying the network structure, the noise in the latent features is gradually reduced to generate continuous video frames. During the denoising process, the reference attention mechanism is used to compare the generated features with the reference image features to ensure that the generated facial animation is consistent with the reference data in expression and details. In addition, the description attention mechanism compares the generated character information with the description features. V d Establish relationships and enhance the model's understanding of the person's appearance and gender characteristics. This method allows the appearance and gender characteristics of the output video to be redefined by modifying the description information, thereby generating a new video version. Through the above steps, the model can not only generate high-quality lip-synced expression videos, but also flexibly adjust the person's appearance and gender characteristics according to the description information, further improving the accuracy and naturalness of video generation.

[0076] In some specific implementations, the generated video is synchronized and verified to detect the accuracy and naturalness of the audio and video synchronization. The parameters and alignment strategy of the audio-video model are adjusted according to the verification results to further improve the quality of the audio-video synchronization.

[0077] In the above implementation manner of this embodiment, a corresponding lip-synced expression video can be generated according to the input voice, which can improve the user experience in various digital audio and video industry application scenarios such as online video platforms, virtual anchors, video production, virtual reality, augmented reality, game development, distance education and video conferencing.

[0078] This embodiment further improves the audio and video synchronization accuracy of the generated lip-synced expression video, can process the input voice in real time, and the generated lip-synced expression video is more natural.

[0079] This embodiment can be applied to voice input in different languages ​​and regions around the world, and generate lip-synced expression videos corresponding to similar voice characteristics, thereby enhancing the adaptability of synchronization technology, allowing it to work across languages ​​and cultural backgrounds and adapt to content requirements in different regions around the world.

[0080] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A method for automatically generating lip shapes and expressions from input speech, characterized in that: include: Extract features of the input speech to obtain an audio feature sequence; Constructing an audio-generated video model, including: inputting human face speech videos in different languages ​​and regions into a Diffusion Transformer model that uses neural dynamic time warping, attention-based time alignment, and audio and video time position coding synchronization, and training to obtain an audio-generated video model; the Diffusion Transformer model is a composite model in which a Diffusion denoising model is added to the Transformer framework; The face speech videos of different languages ​​and regions are input into the Diffusion Transformer model using neural dynamic time warping, attention-based time alignment, and audio and video temporal position encoding synchronization, and the audio-video model is trained, including: Feed speech videos of people speaking different languages ​​and regions into the Diffusion Transformer model to learn the mapping relationship between audio features and video frames. Using neural dynamic time warping and attention-based time alignment algorithms, the Diffusion Transformer model aligns the audio features and video features of the video: Dynamic time warping is performed on the neural network of the Diffusion Transformer model, and the audio features and video features are aligned for the first time by learning nonlinear transformations; Calculate the similarity matrix between the audio features and the video features. Based on the similarity matrix, the DiffusionTransformer model learns the alignment path of the audio features and the video features. Calculate the attention weights between the audio features and the video features. Based on the attention weights, the DiffusionTransformer model dynamically adjusts the degree of attention paid to the audio features and the video features at each time step. According to the alignment path and the degree of attention paid by the Diffusion Transformer model to the audio features and the video features, the Diffusion Transformer model performs a second alignment on the audio features and the video features; By adding the time position coding of audio and video to each layer of the Diffusion Transformer model, the audio and video are synchronized. The time position coding of audio indicates the position of the audio clip on the generated video timeline, and the time position coding of video indicates the position of the video frame on the generated video timeline. The audio feature sequence is input into the audio-generated video model to obtain the lip-synced expression video.

2. The method for automatically generating lip shapes and facial expressions from input speech according to claim 1, characterized in that: Extract the features of the input speech to obtain an audio feature sequence, including: Perform frequency domain filtering on the digital audio data of the input speech to obtain a denoised audio signal; The denoised audio signal is input into the wav2vec model consisting of an encoder network and a context network to extract the audio feature sequence.

3. The method for automatically generating lip shape and expression from input speech according to claim 2, characterized in that: The encoder network or context network is a convolutional neural network based on the Transformer architecture.

4. The method for automatically generating lip shape and expression from input speech according to claim 2, characterized in that: The denoised audio signal is input into the wav2vec model consisting of an encoder network and a context network to extract the audio feature sequence, including: Through the encoder network, the denoised audio signal is converted into a latent representation code in the latent space. The latent representation code at each time step includes the information of the current time point and the previous time point. Input the potential representation code into the context network, extract the context information of multiple time steps, and generate the context representation code; Based on the context representation encoding at multiple consecutive time steps, a sequence of audio features is determined.

5. The method for automatically generating lip shapes and facial expressions from input speech according to claim 1, characterized in that: The face speech videos of different languages ​​and regions are input into the Diffusion Transformer model to learn the mapping relationship between audio features and video frames, including: Compress the face speech videos of different languages ​​and regions and extract the spatiotemporal patches; The spatiotemporal patches and the speech information of each frame of video are input into the Diffusion Transformer model; Align each frame of video speech information with the spatiotemporal patch; Connecting audio features of nearby frames to determine audio features of a generated frame, wherein the nearby frames are video frames whose time distance from the generated frame does not exceed a preset value; Add an audio attention layer to the Diffusion Transformer model to inject the audio features of the generated frames into the generated video.

6. The method for automatically generating lip shape and expression from input speech according to claim 1, characterized in that: Building an audio-video model also includes: A multi-head self-attention algorithm is adopted to synchronize the temporal features, spatial features and audio features between video frames with the temporal features, spatial features and audio features of video frames as independent dimensions.

7. The method for automatically generating lip shape and expression from input speech according to claim 1, characterized in that: Building an audio-video model also includes: During the Diffusion Transformer model training process, sample noise is introduced into the generated video, and the sample noise in the generated video is removed, wherein the sample noise includes image noise and audio noise.

8. The method for automatically generating lip shapes and facial expressions from input speech according to any one of claims 1 to 7, characterized in that: Building an audio-video model also includes: Videos of different time lengths and aspect ratios are input into the Diffusion Transformer model for training.

Images