Lip alignment method, apparatus, device, and storage medium

Abstract

Embodiments of the present application disclose a lip alignment method and device, equipment and storage medium, which comprises: obtaining a first image comprising a target face and a first audio; generating first face reconstruction coefficients comprising first identity coefficients and first expression coefficients according to the first image; generating second expression coefficients corresponding to each audio frame in the first audio, and replacing the first expression coefficients with the second expression coefficients to obtain second face reconstruction coefficients; correcting the second face reconstruction coefficients according to a refining network model to obtain third face reconstruction coefficients comprising corrected second identity coefficients and third expression coefficients; and reconstructing second images corresponding to each audio frame according to the third face reconstruction coefficients, wherein the lip shape of the target face in the second images corresponds to the speech content of the corresponding audio frame. The above method can solve the technical problem that the lip shape of the mouth region image generated directly based on audio features in related technologies has a large difference from the real lip shape.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a lip alignment method, apparatus, device, and storage medium. Background Technology

[0002] A virtual digital human can be understood as a virtual character with a digitally created human appearance. A virtual digital human can be a cartoon character, a realistic character, or a real person. Lip alignment for virtual digital humans refers to using a voice message to drive a virtual digital human in a video or image to achieve a video that perfectly matches the voice message.

[0003] In related technologies, a mouth region image of a virtual digital human is generated based on the audio features of the speech during lip alignment. The lip shape in the mouth region image is the lip shape corresponding to the speech content. Then, the mouth region image is used to replace the mouth region of the virtual digital human in the video or image to generate a new image, thereby achieving a complete match between the lip shape of the virtual digital human's mouth region and the speech content.

[0004] However, in reality, the same speech content may have different lip shapes. For example, the lip shape corresponding to the same speech content may be different when the mouth is opened as wide as possible and when the mouth is closed as wide as possible. Similarly, the lip shape corresponding to the same speech content may be different when the nose is wrinkled and not wrinkled. In related technologies, the lip shape obtained based on the same speech content is approximately consistent, that is, the lip shape in the mouth region image corresponding to the same speech content is approximately the same. When using the mouth region image to drive the virtual digital human, the difference between the lip shape in the virtual digital human and the real lip shape when the speech content is pronounced will be large, affecting the visual experience. Summary of the Invention

[0005] This application provides a lip alignment method, apparatus, device, and storage medium to solve the technical problem in related technologies where the lip shape of the mouth region image differs significantly from the real lip shape when directly generating a virtual digital human's mouth region image based on audio features.

[0006] In a first aspect, one embodiment of this application provides a lip alignment method, including:

[0007] Acquire a first image and a first audio, wherein the first image displays a target face;

[0008] A first face reconstruction coefficient is generated based on the first image, the first face reconstruction coefficient including a first identity coefficient and a first expression coefficient;

[0009] The second expression coefficients corresponding to each audio frame when the first audio is played are generated based on the first audio, and the second expression coefficients are used to replace the first expression coefficients to obtain the second face reconstruction coefficients corresponding to each audio frame.

[0010] The second face reconstruction coefficient is corrected according to the refined network model to obtain the third face reconstruction coefficient, which includes the corrected second identity coefficient and third expression coefficient.

[0011] A second image corresponding to each audio frame is reconstructed based on the third face reconstruction coefficients corresponding to each audio frame, wherein the lip shape of the target face in the second image corresponds to the speech content of the corresponding audio frame in the first audio.

[0012] Secondly, one embodiment of this application also provides a lip alignment device, comprising:

[0013] The first acquisition unit is used to acquire a first image and a first audio, wherein the first image displays a target face;

[0014] The first generation unit is configured to generate a first face reconstruction coefficient based on the first image, wherein the first face reconstruction coefficient includes a first identity coefficient and a first expression coefficient.

[0015] The second generation unit is used to generate second expression coefficients corresponding to each audio frame when the first audio is emitted, and replace the first expression coefficients with the second expression coefficients to obtain the second face reconstruction coefficients corresponding to each audio frame.

[0016] The correction unit is used to correct the second face reconstruction coefficients according to the refined network model to obtain the third face reconstruction coefficients, the third face reconstruction coefficients including the corrected second identity coefficients and the third expression coefficients;

[0017] The reconstruction unit is used to reconstruct a second image corresponding to each audio frame based on the third face reconstruction coefficients corresponding to each audio frame, wherein the lip shape of the target face in the second image corresponds to the speech content of the corresponding audio frame in the first audio.

[0018] Thirdly, one embodiment of this application also provides a lip alignment device, comprising:

[0019] One or more processors;

[0020] Memory, used to store one or more programs;

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the lip alignment method as described in the first aspect.

[0022] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the lip alignment method as described in the first aspect.

[0023] In one embodiment of this application, a first image containing a target face and a first audio recording are acquired. A first face reconstruction coefficient is generated based on the first image. A second expression coefficient corresponding to each audio frame is obtained based on the first audio recording. Then, a second face reconstruction coefficient corresponding to each audio frame is obtained based on the second expression coefficient and the first face reconstruction coefficient. The second face reconstruction coefficient is corrected using a refined network model to obtain a third face reconstruction coefficient. A second image corresponding to each audio frame is obtained based on the third face reconstruction coefficient. Furthermore, the lip shape of the target face in the second image is aligned with the speech content of the corresponding audio frame. This method solves the problem in related technologies that rely on audio features... To address the technical problem of significant differences between the lip shape in the mouth region image and the real lip shape when directly generating the mouth region image of a virtual digital human, this paper first determines the second expression coefficients when the first audio is emitted and the first face reconstruction coefficients of the target face in the first image. Then, the second face reconstruction coefficients containing the second expression coefficients are obtained, that is, the expression when the first audio is emitted is added to the target face. After that, the second face reconstruction coefficients are corrected using a refined network model so that the second expression coefficients and the first identity coefficients of the target face are better integrated, thereby obtaining a second image that aligns the target face and the first audio, improving the alignment accuracy and effect. Attached Figure Description

[0024] Figure 1 A flowchart illustrating a lip alignment method provided in one embodiment of this application;

[0025] Figure 2 A flowchart illustrating a lip alignment method provided in one embodiment of this application;

[0026] Figure 3 A schematic diagram of a refined network model provided in one embodiment of this application;

[0027] Figure 4 This is a schematic diagram of the data processing flow of a lip alignment method provided in one embodiment of this application;

[0028] Figure 5 A schematic diagram of a data processing flow for a refined network model provided in one embodiment of this application;

[0029] Figure 6 A schematic diagram of a refined network model provided in one embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the structure of a lip alignment device provided in one embodiment of this application;

[0031] Figure 8 This is a schematic diagram of the structure of a lip alignment device provided in one embodiment of this application. Detailed Implementation

[0032] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the scope of the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application are shown in the drawings, not the entire structure.

[0033] One embodiment of this application provides a lip alignment method, which can be executed by a lip alignment device. The lip alignment device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. Currently, the lip alignment device can be an electronic device with data processing and computing capabilities, such as a laptop, desktop computer, tablet computer, mobile phone, or interactive whiteboard. In one embodiment, the lip alignment device has at least one application installed, through which the lip alignment method is implemented; in this case, the lip alignment device can also be the application itself.

[0034] Figure 1 A flowchart illustrating a lip alignment method according to one embodiment of this application. (See reference...) Figure 1 When a lip alignment device performs a lip alignment method, it includes the following steps:

[0035] Step 110: Obtain the first image and the first audio. The first image shows the target face.

[0036] The target face can be understood as the facial region of a virtual digital human, which can be a real person's face, a realistic human face, or a virtual cartoon face. In one embodiment, the target face is described as a real person's face.

[0037] Currently, the image containing the target face is referred to as the first image. The acquisition method of the first image is not currently limited. For example, the first image is an image captured locally by the lip alignment device, or the first image is an image acquired by the lip alignment device through a network or other transmission methods.

[0038] The first audio segment is a piece of audio data containing speech content. Speech content can be understood as spoken content, which can be human voice, voice-modified human voice, or other types of content. The acquisition method of the first audio segment is currently not limited. For example, the first audio segment may be audio data locally collected by the lip-syncing device, or it may be audio data acquired by the lip-syncing device through a network or other transmission methods.

[0039] In one embodiment, a first image is driven by a first audio to simulate the effect of a target face in the first image "speaking" the speech content of the first audio.

[0040] Step 120: Generate first face reconstruction coefficients based on the first image. The first face reconstruction coefficients include first identity coefficients and first expression coefficients.

[0041] Facial reconstruction coefficients are the coefficients required to reconstruct a face. Currently, the facial reconstruction coefficients generated based on the first image are denoted as the first facial reconstruction coefficients. In one embodiment, the facial reconstruction coefficients include identity coefficients and expression coefficients. Currently, the identity coefficients and expression coefficients in the first facial reconstruction coefficients are denoted as the first identity coefficients and the first expression coefficients, respectively.

[0042] Among them, the identity coefficient is related to facial identity. Facial identity is a specific description of facial features, such as key facial parts. Key facial parts include eyebrows, eyes, nose, mouth, and cheeks. For example, the key parts described by facial identity are large eyes, a high nose bridge, and thin lips. Therefore, for different virtual digital humans with different faces, the key parts described by their facial identities will differ. It is understandable that overlaying different expressions onto facial identities can produce faces with various expressions. Optionally, various identity bases that can appear on the face are pre-defined. Each identity base has the semantics of a key part, and each identity base can be considered a category in which that key part will appear. For example, when the key part is the mouth, the corresponding identity bases include upper thick lips, upper thin lips, upper small (short) lips, upper large (long) lips, lower thick lips, lower thin lips, lower small (short) lips, upper large (long) lips, etc. The number of identity bases and the construction rules are currently not limited. The identity coefficient can reflect the weight of each identity basis in the face (or confidence level). The identity coefficient can be in the form of a matrix vector, where each value in the identity coefficient corresponds to an identity basis. The larger the value, the greater the proportion of that identity basis in the face. Optionally, the identity coefficient can be applied to the identity basis and superimposed on a pre-constructed average face to obtain the desired face identity.

[0043] The expression coefficients are related to facial expressions. Optionally, various expression bases that can appear on the face can be pre-defined. Each expression base corresponds to an expression, which can represent the deformation required when the facial identity makes a corresponding expression. It is understandable that since facial expressions are very rich, if we want to represent all facial expressions, we need to construct a large number of expression bases, which will greatly increase the amount of data processing. Therefore, in one embodiment, expression bases of basic expressions can be constructed. The specific content of the basic expressions can be set according to the actual situation. Various facial expressions can be obtained by combining and superimposing the basic expressions onto the facial identity. For example, basic expressions for the eyes include: left eye closed, left eye wide open, right eye closed, and right eye wide open. Based on these four basic expressions and the eyes in the facial identity, various eye expressions can be obtained. For example, the expression of slightly squinting eyes can be obtained by linearly superimposing the left eye closed, right eye closed, and the eyes in the facial identity. The number of expression bases and the construction rules are not currently limited. The expression coefficient can reflect the weight of each expression base on the face (or can be understood as confidence). The expression coefficient can be in the form of a matrix vector, where each value in the expression coefficient corresponds to an expression base. The larger the value, the greater the proportion of that expression base on the face.

[0044] It is understood that the face reconstruction coefficients may also include other types of coefficients. In one embodiment, the face reconstruction coefficients may also include texture coefficients, pose coefficients, and illumination coefficients. Currently, the texture coefficients, pose coefficients, and illumination coefficients included in the first face reconstruction coefficients are referred to as the first texture coefficient, the first pose coefficient, and the first illumination coefficient, respectively.

[0045] The texture coefficients are related to the facial texture. Optionally, various texture bases that can appear on the face can be pre-defined. Each texture base represents a type of texture that can appear on the facial skin. The number of texture bases and the construction rules are not currently limited. The texture coefficients can reflect the weight of each texture base on the facial skin surface. The texture coefficients can be in the form of a matrix vector, where each value in the texture coefficient corresponds to a texture base. The larger the value, the greater the proportion of that texture base on the face. Optionally, applying the texture coefficients to the texture bases and superimposing them on the average texture can obtain the desired facial texture.

[0046] Pose coefficients are related to facial pose; optionally, facial pose can be used to describe facial posture. The face position can be reconstructed using pose coefficients; for example, if the face is in profile, the pose coefficients can be used to reconstruct a profile face. Generally, pose coefficients include camera rotation and translation coefficients, both of which are matrices.

[0047] The illumination coefficient is related to the ambient lighting of the face. When reconstructing a face, increasing the illumination coefficient can enhance the texture of the face.

[0048] Generally, the corresponding face can be reconstructed using the aforementioned face reconstruction coefficients. In one embodiment, the 3D face reconstructed using identity coefficients and expression coefficients includes 3D facial identity and facial expression. This is then mapped to a 2D space to obtain 2D facial identity and facial expression. Similarly, 3D facial texture can be reconstructed using texture coefficients, and then mapped to a 2D space to obtain 2D facial texture. Facial pose can be reconstructed using pose coefficients, and facial ambient lighting can be reconstructed using lighting coefficients. In practical applications, the specific types of coefficients included in the face reconstruction coefficients can be selected based on actual needs, and the corresponding face can be reconstructed. Currently, face reconstruction using identity coefficients, expression coefficients, pose coefficients, texture coefficients, and lighting coefficients is described as an example. In one embodiment, the formula... It can reconstruct a 3D facial identity and facial expression. Here, S represents the reconstructed 3D facial shape (i.e., the 3D facial identity and facial expression). B represents the pre-constructed average face. id B represents the identity basis. exp Let α represent the expression basis, β represent the identity coefficient, and β represent the expression coefficient. The parameters corresponding to these symbols can be represented in matrix-vector form. Optionally, an average face can be obtained based on the average 3D coordinates of a certain number (e.g., 100, 200, or more than 200) of 3D faces from a currently available database (such as the Basel Face Mode database). This average face can be considered a pre-constructed 3D face model. The identity basis is a set of identity bases, which can be constructed based on a publicly available database (such as the Basel Face Mode database) using Principal Component Analysis (PCA). Optionally, That is, there are 80 identity bases, each with a corresponding identity coefficient. Optionally, the identity bases only correspond to facial identities, excluding restrictions on the ears and neck. The expression base is a set of expression bases. Expression bases can also be constructed based on a public database using PCA, and explicit semantic information (such as closed eyes, frowning, etc.) can be added after construction. Optionally, semantic information can be added to the expression bases through semantic information transfer. That is, there are 64 expression bases, and each expression base has a corresponding expression coefficient. This is achieved through the formula... It can reconstruct a 3D facial texture. Here, T represents the reconstructed 3D facial texture. B represents the pre-built average texture. tδ represents the texture basis, and δ represents the texture coefficients. The parameters corresponding to each symbol can be represented in matrix-vector form. Optionally, the average texture can be obtained based on the average texture data of a certain number of 3D faces in a currently available database (such as the Basel Face Mode database), which can be considered as a pre-constructed 3D texture model. The texture basis is a set of texture bases, which can be constructed using PCA based on a currently available database (such as the Basel Face Mode database). Optionally, That is, there are 80 texture bases, each with corresponding texture coefficients. After obtaining S and T, the pose coefficients and lighting coefficients can be combined to render a two-dimensional face, which is the reconstructed face. In one embodiment, the pose coefficients are applied to a perspective camera projection model. The perspective camera projection model is a method of drawing or rendering on a two-dimensional plane to obtain a visual effect close to a real three-dimensional object (currently the reconstructed face). In this case, the pose coefficients are vectors composed of 12 parameters in the camera rotation matrix (also denoted as camera rotation coefficients) and translation matrix (also denoted as translation coefficients). The lighting coefficients are applied to a spherical harmonics (SH) lighting model to simulate the lighting environment. In this case, the lighting coefficients can also be considered as coefficients of the SH lighting model. Then, a differentiable renderer is used to reconstruct the two-dimensional face image.

[0049] In one embodiment, the first face reconstruction coefficient is determined by the first image. The specific method of determination is not currently limited. For example, the first image can be input into a face reconstruction network model, and the face reconstruction network model can output the first face reconstruction coefficient. The face reconstruction network model is a neural network model used to identify the face reconstruction coefficients required for face reconstruction. The specific structure and parameters of the face reconstruction network model are not currently limited. It is understood that the face reconstruction network model needs to be trained first to obtain stable performance (i.e., output relatively ideal face reconstruction coefficients). Then, the face reconstruction network model is applied. The structure, parameters, and training process of the face reconstruction network model are not currently limited; it only needs to satisfy the requirement of outputting the corresponding face reconstruction coefficients after inputting an image containing a face.

[0050] Step 130: Generate second expression coefficients corresponding to each audio frame when the first audio is played, and replace the first expression coefficients with the second expression coefficients to obtain the second face reconstruction coefficients corresponding to each audio frame.

[0051] In one embodiment, when the face "speaks" the first audio, each expression base has a corresponding expression coefficient. Currently, the expression coefficients obtained based on the first audio are recorded as the second expression coefficients. Optionally, the first audio consists of multiple frames of audio data. In this case, each audio frame in the first audio has a corresponding set of second expression coefficients, which includes the second expression coefficients corresponding to each expression base.

[0052] For example, the method for determining the second expression coefficient is not currently limited. For instance, the audio features of each audio frame in the first audio are first identified, and then the second expression coefficient is obtained based on these audio features. These audio features can reflect the characteristics of the sound; for example, Mel Frequency Cepstrum Coefficient (MFCC) features and Perceptual Linear Predictive (PLP) features are common audio features. In one embodiment, MFCC features are used as an example, where each audio frame in the first audio has a corresponding MFCC feature. Then, neural network technology is used to process the MFCC features to obtain the second expression coefficient. That is, a convolutional neural network model is pre-trained (the training method is not currently limited). The specific structure and parameters of this model are not currently limited. The convolutional neural network model can output the corresponding expression coefficient (currently the second expression coefficient) based on the input MFCC features. Alternatively, the second expression coefficient can be obtained directly from the first audio using neural network technology. In this case, the trained neural network model (the training method is not currently limited) can output the corresponding expression coefficient (currently the second expression coefficient) based on the input audio (currently the first audio).

[0053] After obtaining the second expression coefficient, the first expression coefficient in the first face reconstruction coefficient is replaced with the second expression coefficient. The resulting new face reconstruction coefficient is then recorded as the second face reconstruction coefficient. The second face reconstruction coefficient and the first face reconstruction coefficient are identical except for the specific value of the expression coefficient. In one embodiment, when the second expression coefficient corresponding to an audio frame is used to replace the first expression coefficient, the second face reconstruction coefficient corresponding to that audio frame is obtained. At this time, each audio frame has a corresponding second face reconstruction coefficient.

[0054] It should be noted that the first facial reconstruction coefficient and the second expression coefficient can be generated simultaneously or sequentially, and the specific execution order is not currently limited.

[0055] Step 140: Correct the second face reconstruction coefficients according to the refined network model to obtain the third face reconstruction coefficients, which include the second identity coefficients and the third expression coefficients.

[0056] For example, the facial expression reconstructed using the second facial reconstruction coefficients corresponds to the expression when the audio frame is spoken. However, the face reconstructed using the second facial reconstruction coefficients merely adds the expression from the corresponding audio frame to the target face in the first image, failing to integrate it well with the target face's identity. Therefore, in one embodiment, a refinement module is used to better integrate the expression corresponding to the first audio with the target face. The refinement module can better integrate the second expression coefficients and the first identity coefficients, resulting in better alignment between the mouth shape of the target face and the first audio when the first audio is superimposed on the target face. The refinement module is a trained neural network model, and its specific structure and parameters are not currently limited. During training, the refinement module incorporates optimizations for the mouth to improve the alignment between the lip shape and the first audio.

[0057] The refined network model can obtain a second identity coefficient and a third expression coefficient with better fusion effect based on the second expression coefficient and the first identity coefficient. Currently, the second identity coefficient and the third expression coefficient can be considered as coefficients obtained after optimizing and correcting the first identity coefficient and the second expression coefficient. In one embodiment, the second identity coefficient and the third expression coefficient are used to replace the first identity coefficient and the second expression coefficient in the second face reconstruction coefficient to update the second face reconstruction coefficient, and the updated second face reconstruction coefficient is recorded as the third face reconstruction coefficient. At this time, the third face reconstruction coefficient can be considered as the reconstruction coefficient obtained after correcting the second face reconstruction coefficient.

[0058] In one embodiment, when using the refined network model, a 3D target face is first reconstructed based on the second facial reconstruction coefficients. The expression of this target face is obtained through the second expression coefficients. The 3D target face obtained before rendering the 2D target face during reconstruction is the 3D target face used by the current refined network model. After obtaining the 3D target face, the positions of facial keypoints on the target face are determined (represented by 3D coordinates). Facial keypoints, also known as key facial features, refer to points located in key facial areas, and each keypoint represents semantic information of the face. The selection of facial keypoints can be set according to actual conditions. For example, currently, there are 68 facial keypoints: 5 keypoints each for the left and right eyebrows, 6 keypoints each for the left and right eyes, 20 keypoints for the mouth, 17 keypoints for the cheeks, and 9 keypoints for the nose. Optionally, the reconstructed facial keypoints (i.e., the 3D coordinates of the facial keypoints), the second expression coefficient, and the first identity coefficient are input together into the refined network model. This allows the refined network model to optimize the second expression coefficient and the first identity coefficient based on the facial keypoints, outputting a third expression coefficient and a second identity coefficient. Specifically, during the processing of the refined network model, new facial keypoints are first obtained based on the facial keypoints, the second expression coefficient, and the first identity coefficient. These new facial keypoints can be considered corrections to the existing facial keypoints. These new facial keypoints are also 3D facial keypoints, and the 3D face corresponding to the new facial keypoints can better integrate the second expression coefficient and the first identity coefficient. Then, the corresponding third expression coefficient and second identity coefficient are obtained based on the new facial keypoints. Currently, for ease of distinction, the new facial keypoints are denoted as the second 3D facial keypoints, and the facial keypoints input into the refined network model are denoted as the first 3D facial keypoints.

[0059] In one embodiment, to facilitate understanding of the optimization process of the refined network model, the training process of the refined network model is briefly described. Currently, the content input to the refined network model during training is denoted as the first identity coefficient, the second expression coefficient, and the first 3D facial keypoints used during training, respectively, and the content output by the refined network model is denoted as the second identity coefficient and the third expression coefficient used during training, respectively. It is understood that all of the above data can be obtained based on the first image and the first audio used for training. For example, during training, two-dimensional facial keypoints (i.e., using two-dimensional coordinates to represent the position of facial keypoints) are introduced for supervised training. These two-dimensional facial keypoints can be annotated manually or semi-manually. The lip shape of the face corresponding to the two-dimensional facial keypoints is aligned with the first audio (corresponding to the second expression coefficient in the training process). The two-dimensional facial keypoints supervise the refined network model's prediction of the second 3D facial keypoints, thereby improving the accuracy of the second 3D facial keypoints. During training, the alignment of lip shape with the first audio is improved by constraining keypoint pairs composed of facial keypoints in the upper and lower lips. Each keypoint pair contains two facial keypoints located on the upper and lower lips respectively, and their positions in the upper and lower lips correspond. For example, a keypoint located on the upper middle side of the upper lip and a keypoint located on the lower middle side of the lower lip constitute a keypoint pair (i.e., symmetrical position). When using keypoint pairs, the keypoint pairs in the two-dimensional facial keypoints supervise the corresponding keypoint pairs in the second three-dimensional facial keypoints to improve the alignment effect of the second three-dimensional facial keypoints. Based on this, during training, the first identity coefficient, second expression coefficient, and first 3D facial keypoints used in the training process are first input into the refined network model. Then, the second 3D facial keypoints generated during the refined network model's processing, along with the output second identity coefficient and third expression coefficient, are obtained. Next, loss functions related to facial identity are constructed based on the first and second identity coefficients; loss functions related to facial expression are constructed based on the second and third expression coefficients; loss functions related to 3D facial keypoints are constructed based on the first and second 3D facial keypoints; loss functions related to 2D facial keypoints are constructed based on the 2D and 2D facial keypoints; and loss functions related to lip shape are constructed based on the mouth-related keypoint pairs in the 2D and 2D facial keypoints. Then, the model parameters of the refined network model are adjusted based on each loss function. This process is repeated to train the refined network model until each loss function converges. The convergence condition for the loss function can be that the training has reached a certain number of iterations or that the specific values ​​of the loss function obtained from consecutive iterations are within a certain range. Once training is complete, the refined network model can output more accurate second identity coefficients and third expression coefficients.

[0060] It is understandable that when the second face reconstruction coefficient includes texture coefficient, pose coefficient, and illumination coefficient, the third face reconstruction coefficient also includes texture coefficient, pose coefficient, and illumination coefficient.

[0061] After refining the network model to output the second identity coefficient and the third expression coefficient to obtain the third face reconstruction coefficient, step 150 is executed.

[0062] Step 150: Reconstruct the second image corresponding to each audio frame based on the third face reconstruction coefficients corresponding to each audio frame. The lip shape of the target face in the second image corresponds to the speech content of the corresponding audio frame in the first audio.

[0063] For example, a corresponding two-dimensional target face can be reconstructed based on the third face reconstruction coefficients. The lip shape of the two-dimensional target face is aligned with the speech content of the corresponding audio frame in the first audio. Currently, the reconstructed image containing the two-dimensional target face is denoted as the second image. In one embodiment, referring to the reconstruction method described in step 130, a two-dimensional target face is reconstructed based on the third face reconstruction coefficients to obtain the second image. It should be noted that, referring to the reconstruction method described in step 130, the third face reconstruction coefficients can only reconstruct the target face and cannot reconstruct the background (non-target face part) of the first image. Therefore, after reconstructing the target face, it can be merged with the background in the first image to obtain the second image. In one embodiment, referring to the reconstruction method described in step 130, after reconstructing the target face based on the third face reconstruction coefficients, the image containing the reconstructed target face is input into a generative adversarial network model for processing to output an image with a more realistic style and containing the target face. Then, this image is merged with the background in the first image to obtain the second image. The Generative Adversarial Network (GAN) consists of a generator network and a discriminator network. Its output can mimic the real samples used for training as closely as possible. Currently, the GAN model can output a more realistic target face based on the input target face. Optionally, the training set used in the training process of the GAN model employs a masking strategy. By masking, the area below the nose in each image required for training can be extracted separately to better optimize the area around the mouth, ensuring that the area around the mouth in the target face output by the GAN model is more realistic.

[0064] For example, based on the above method, each audio frame in the first audio can have a corresponding second image. The second images are arranged in the time order of each audio frame to obtain a video data. In the video data, the lip shape of the target face is aligned with the first audio, which can simulate the effect of the target face "speaking" the first audio.

[0065] The above-described method, which involves acquiring a first image containing the target face and a first audio recording, generating a first face reconstruction coefficient based on the first image, obtaining a second expression coefficient corresponding to each audio frame based on the first audio recording, and then obtaining a second face reconstruction coefficient corresponding to each audio frame based on the second expression coefficient and the first face reconstruction coefficient. A refined network model is then used to correct the second face reconstruction coefficient to obtain a third face reconstruction coefficient. A second image corresponding to each audio frame is obtained based on the third face reconstruction coefficient, and the lip shape of the target face in the second image is aligned with the speech content of the corresponding audio frame. This method solves the technical problem in related technologies where the lip shape in the mouth region image differs significantly from the real lip shape when directly generating the mouth region image of a virtual digital human based on audio features. First, the second expression coefficients when the first audio recording is played and the first face reconstruction coefficient of the target face in the first image are determined. Then, a second face reconstruction coefficient containing the second expression coefficient is obtained, that is, the expression when the first audio recording is played is added to the target face. Next, a refined network model is used to correct the second face reconstruction coefficient, so that the second expression coefficient and the first identity coefficient of the target face are better integrated, thereby obtaining a second image where the target face and the first audio recording are aligned, improving the alignment accuracy and effect.

[0066] Figure 2 This is a flowchart illustrating a lip alignment method according to one embodiment of this application. This embodiment, based on the above embodiments, elaborates on the process of correcting the second face reconstruction coefficients according to a refined network model and the process of reconstructing the second image based on the third face reconstruction coefficients corresponding to each audio frame. (See reference...) Figure 2 The lip alignment method includes the following steps:

[0067] Step 210: Obtain the first image and the first audio. The first image shows the target face.

[0068] Step 220: Generate first face reconstruction coefficients based on the first image. The first face reconstruction coefficients include first identity coefficients and first expression coefficients.

[0069] Step 230: Extract features from the first audio to obtain the audio features corresponding to each audio frame in the first audio.

[0070] In one embodiment, the audio features used are MFCC features. MFCC features are commonly used in audio processing techniques. MFCC features retain semantically relevant content in the first audio file while filtering out irrelevant information such as background noise. The extraction method for MFCC features is an already implemented technique and will not be elaborated upon here. Currently, each audio frame in the first audio file has a corresponding MFCC feature.

[0071] Step 240: Obtain the second expression coefficients corresponding to each audio frame when the first audio is emitted based on the audio features, and replace the first expression coefficients with the second expression coefficients to obtain the second face reconstruction coefficients corresponding to each audio frame.

[0072] In one embodiment, a convolutional neural network is used to process MFCC features to obtain the corresponding second expression coefficients. Currently, the convolutional neural network used is denoted as the expression recognition network model. In this case, obtaining the second expression coefficients corresponding to each audio frame when the first audio is emitted, based on the audio features, specifically includes: inputting the audio features into the expression recognition network model so that the expression recognition network model obtains the second expression coefficients of the corresponding audio frames based on the audio features; the loss function used by the expression recognition network model during training includes: l exp =||β-β R ||2+λβ1, where β and β R Let represent the second and first expression coefficients used in the training process, respectively; λ represent the adjustment parameter; ||||1 represents the L1 norm; and ||||2 represents the L2 norm.

[0073] For example, the facial expression recognition network model uses a convolutional neural network, and its specific structure is not currently limited. The facial expression recognition network model undergoes a training process before deployment. The training method used for training the facial expression network model is not currently limited. For example, a supervised training method can be used, where audio features with known facial expression coefficients are used as the training set. The facial expression recognition network model processes the audio features to output corresponding second facial expression coefficients. Then, based on the known facial expression coefficients and the output second facial expression coefficients, the model parameters of the facial expression recognition network model are adjusted to achieve the purpose of training the facial expression recognition network model, making the facial expression recognition network model output more accurate second facial expression coefficients. In one embodiment, since the second facial expression coefficients output by the facial expression recognition network model are used to replace the first facial expression coefficients, and the first and second facial expression coefficients are extracted from the first image and the first audio respectively and have no obvious correlation, in order to improve the accuracy of the fusion of the first audio and the target face, a loss function is added when training the facial expression recognition network model: l exp =||β-β R ||2+λβ1, where the value of λ can be set according to the actual situation. During training, the second expression coefficient β output by the expression recognition network model replaces the first expression coefficient β. R Substituting into the loss function above, we obtain l exp The specific value of (the increased loss function), then, through l expThe specific values ​​and the loss values ​​of the original loss function of the expression recognition network model are used to update the model parameters of the expression recognition network model in reverse, so that the second expression coefficients output by the expression recognition network model are associated with the expression of the two-dimensional target face, thereby improving the accuracy of subsequent fusion.

[0074] Step 250: Reconstruct the first three-dimensional facial key points based on the second facial reconstruction coefficient.

[0075] For example, the first identity coefficient and the second expression coefficient from the second face reconstruction coefficients are substituted into the formula. The reconstructed 3D facial identity and facial expression are then used to substitute the texture coefficients from the second facial reconstruction coefficients into the formula. A 3D facial texture is reconstructed, and then, by combining pose and illumination coefficients, a 3D target face can be reconstructed. It can be understood that the second facial reconstruction coefficients corresponding to each audio frame can reconstruct a 3D target face. Then, based on the reconstructed target face, the 3D coordinates of facial key points are obtained. Currently, the facial key points obtained based on the second facial reconstruction coefficients are denoted as the first 3D facial key points. In one embodiment, there are 68 first 3D facial key points, each containing its corresponding 3D coordinates.

[0076] Step 260: Input the first three-dimensional facial key points, the first identity coefficient, and the second expression coefficient into the refined network model so that the refined network model outputs the corrected second identity coefficient and the third expression coefficient.

[0077] For example, the refined network model obtains a second identity coefficient and a third expression coefficient with better fusion and lip alignment based on the first three-dimensional facial key points, the first identity coefficient, and the second expression coefficient of the input, and then obtains a third facial reconstruction coefficient based on the second identity coefficient and the third expression coefficient.

[0078] In one embodiment, the refined network model includes a fine-tuning sub-model and a reconstruction sub-model. Step 260 may include: inputting a first three-dimensional facial key points, a first identity coefficient, and a second expression coefficient into the fine-tuning sub-model so that the fine-tuning sub-model outputs the fine-tuned second three-dimensional facial key points; and inputting the second three-dimensional facial key points into the reconstruction sub-model so that the reconstruction sub-model outputs the reconstructed second identity coefficient and third expression coefficient.

[0079] In one embodiment, a finetune model (also referred to as Finetune) is used to extract features of a first identity coefficient, a second expression coefficient, and a first three-dimensional facial keypoint, and to obtain a corrected second three-dimensional facial keypoint based on the extracted features. The finetune model employs a multilayer perceptron (MLP). Figure 3 This is a schematic diagram of a refined network model provided in one embodiment of this application. (Reference) Figure 3 Taking 68 facial key points as an example, the first three-dimensional facial key point L c (68×3, which can be understood as the 3D coordinates corresponding to 68 facial key points) After being input into the fine-tuning sub-model, it first passes through MLP(64,64), where MLP(64,64) represents two fully connected layers with 64 and 64 output channels respectively. After processing by MLP(64,64), it outputs 68×64 features. Then, it passes through MLP(128,256,1024), where MLP(128,256,1024) represents three fully connected layers with 128, 256, and 1024 output channels respectively. After processing by MLP(128,256,1024), it outputs 68×1024 features. Finally, it passes through a max-pooling layer. The first identity coefficient α R The second expression coefficient β is passed through its corresponding fully connected layer (FC) and then concatenated with the features output by the max pooling layer. This involves fusing the features of the first 3D facial keypoints, the first identity coefficient, and the second expression coefficient. The fused features are then processed by an MLP (512,256,128,3), where MLP (512,256,128,3) represents four fully connected layers with output channels of 512, 256, 128, and 3, respectively. Based on the fused features, MLP (512,256,128,3) outputs the corrected second 3D facial keypoint L. r (68×3).

[0080] The reconstructed sub-model is used to obtain the corrected second identity coefficient and third expression coefficient based on the second 3D facial key points. The reconstructed sub-model employs a multilayer perceptron (MLP), referencing... Figure 3 The second 3D facial key points L output by the fine-tuning sub-model r After the (68×3) input reconstruction sub-model is processed by MLP(128,256,512), where MLP(128,256,512) represents three fully connected layers with output channels of 128, 256 and 512 respectively. After processing by MLP(128,256,512), the output features are 68×512. The output features are then processed by a max pooling layer and output to two fully connected layers (FC) to obtain the second identity coefficient and the third expression coefficient respectively.

[0081] It should be noted that Figure 3 Each MLP in the MLP is followed by a batch normalization layer and an activation function layer. Currently, the activation function layer uses ReLU. This is understandable. Figure 3The neural network shown is merely an exemplary description of the refined network model. In practical applications, the refined network model can also adopt other structures, which are not limited here.

[0082] In one embodiment, the loss function used during the training of the refined network model includes:

[0083] l α =||α R -α * ||1

[0084] l β =||β-β * ||1

[0085]

[0086]

[0087]

[0088] Where, α R and α * Let β and β represent the first and second identity coefficients used during training, respectively. * Let L represent the second and third facial expression coefficients used during training, n represent the total number of facial keypoints, i represent the i-th facial keypoint, and L represent the third facial keypoint. r and L c Let k represent the first 3D facial keypoints and the second 3D facial keypoints used during training, respectively. i Let M represent the i-th 2D facial keypoint used for supervision. i Let represent the i-th second 3D facial keypoint used during training, E represent the set of keypoint pairs formed by corresponding facial keypoints in the upper and lower lips, (i,j) represent the keypoint pair formed by the i-th and j-th facial keypoints in E, s, ∏ and t represent the scale factor, rotation matrix and translation vector, respectively, and ||||1 represents the L1 norm.

[0089] Substitute the first identity coefficient input to the refined network model and the second identity coefficient output by the refined network model into l respectively. α From the corresponding loss function, we can obtain l α The specific value, so as to be obtained through l α Restrict facial recognition.

[0090] Substitute the second expression coefficients input to the refined network model and the third expression coefficients output by the refined network model into l respectively. β From the corresponding loss function, we can obtain l β The specific value, so as to be obtained through lβ Constrain facial expressions. Understandably, in practical applications, the first expression coefficient can also be used instead of l. β The second expression coefficient in the calculation is used to obtain l. β The specific value.

[0091] Substitute the first 3D facial key points input from the refined network model and the second 3D facial key points output from the refined network model into l respectively. l1 From the corresponding loss function, we can obtain l l1 The specific value, so as to be obtained through l l1 Constrain the key points of the 3D face.

[0092] When training the refined network model, two-dimensional facial keypoints are also used, either manually or semi-manually annotated (which can be achieved with the help of certain keypoint recognition tools or programs). The lip shape of the face corresponding to the two-dimensional facial keypoints is consistent with the lip shape of the audio frame corresponding to the currently used second facial reconstruction coefficients when the speech is produced. The two-dimensional facial keypoints and the second three-dimensional facial keypoints output by the refined network model are then substituted into l. l2 From the corresponding loss function, we can obtain l l2 The specific value, so as to be obtained through l l2 To improve the accuracy of the second three-dimensional facial key points, it should be noted that the second three-dimensional facial key points can be mapped to two-dimensional space through s, π, and t to obtain the corresponding facial key points in two-dimensional space.

[0093] When training the refined network model, keypoint pairs are constructed based on facial keypoints located around the mouth. Each keypoint pair contains the i-th and j-th facial keypoints located in the upper and lower lips, respectively, and their positions correspond. For example, a keypoint located at the upper center of the upper lip and the lower center of the lower lip form one keypoint pair; a keypoint located at the lower center of the upper lip and the upper center of the lower lip form another; and so on. Then, the keypoint pairs from the two-dimensional and the corresponding keypoint pairs from the second and third-dimensional facial keypoints are substituted into the l... m From the corresponding loss function, we can obtain l m The specific value, so as to be obtained through l m Improve lip alignment, especially when the mouth is open.

[0094] For example, after training a refined network model based on the above loss function, the refined network model can be deployed to obtain the second identity coefficient and the third expression coefficient.

[0095] Step 270: Render the third image corresponding to each audio frame based on the third face reconstruction coefficients corresponding to each audio frame.

[0096] For example, the second identity coefficient and the third expression coefficient from the third face reconstruction coefficients are substituted into the formula. The reconstructed 3D facial identity and facial expression are then used to substitute the texture coefficients from the third facial reconstruction coefficients into the formula. A 3D facial texture is reconstructed. After obtaining S and T, pose and lighting coefficients are combined, and a 2D face is rendered, which is the reconstructed target face. In one embodiment, the image containing the reconstructed target face is currently referred to as the third image. It is understood that the third image only contains the reconstructed target face and does not contain any additional background (the background can be transparent or a set solid color). Each audio frame in the first audio has a corresponding third image.

[0097] Step 280: Input the third image into the generative adversarial network model so that the generative adversarial network model outputs a fourth image optimized for the lip region.

[0098] In one embodiment, pix2pix is ​​used as the generative adversarial network model. pix2pix essentially performs style transfer tasks. The pix2pix network structure includes a generator and a discriminator. The generator generates images based on the input image and learns to fool the discriminator, which learns to classify fake images (synthesized by the generator) and real images. In one embodiment, when training pix2pix, the first image used during training is denoted as I. s I s It can also be considered as the original image, and the third image generated based on the first image and the first audio used during training is denoted as I. g , will I g and I s Data pairs are formed, and multiple data pairs together constitute the training set required for training pix2pix. Where i = 1, 2, 3...N, and N is an integer. In one embodiment, the optimization objective function for pix2pix is:

[0099]

[0100] L(G,D)=L rec (G)+λL adv (G,D)

[0101] Among them, L rec =||I s -G(I g )||1,L adv =logD(I s)+log(1-D(G(I g In this context, G represents the generator, and D represents the discriminator. * This represents a generative adversarial network model, where λ is an adjustment parameter.

[0102] Based on the above objective function, the optimal generative adversarial network (GAN) model G is found through iterative optimization (i.e., iteratively training the GAN model using multiple datasets). * The iterative process needs to be broken down into sequential iterative optimizations of the generator G and the discriminator D. The objective function to be optimized for the generator G is... The objective function that discriminator D needs to optimize is: The generator G and discriminator D can be iteratively optimized sequentially using two objective functions.

[0103] It is understandable that the optimization process (i.e. the training process) of pix2pix is ​​an already implemented technique and will not be described separately at this time.

[0104] In one embodiment, to optimize the mouth region and improve alignment, a masking strategy is used to construct the training set for the generative adversarial network (GAN) model. In this case, a first mask is applied to each data pair in the training set. The first mask extracts the area below the nose (including the mouth region). The specific value used for the first mask is not currently limited. After applying the first mask to the first and third images in each data pair, the training set for the GAN model is obtained. Therefore, the dataset used for training the GAN model consists of multiple data pairs. Each data pair includes a first image and a corresponding third image used during training. Both the first image and the corresponding third image are masked, and the first mask is used to extract the area below the nose in the image.

[0105] After the generative adversarial network model is trained, it is deployed and applied. During the application process, a new image can be generated based on the third image that is similar in style to the first image (especially the area below the nose). This new image is referred to as the fourth image. Compared to the third image, the mouth effect in the fourth image is more realistic.

[0106] It is understandable that each audio frame in the first audio has a corresponding fourth image.

[0107] Step 290: Merge the fourth image and the first image to obtain the second image corresponding to the corresponding audio frame.

[0108] For example, the fourth image is fused with the first image, that is, the target face in the fourth image is superimposed with the background area of ​​the first image to obtain the second image.

[0109] In one embodiment, to make the fused second image more natural and smooth, the fourth image and the first image are fused based on the second mask. In this case, this step may include steps 291-293:

[0110] Step 291: Obtain the second mask, which has the same size as the fourth image.

[0111] For example, the second mask is used to extract the facial region of the target face in the fourth image, and the specific value used for the second mask is not currently limited. The generation rule for the second mask is not currently limited; for example, the facial region in the fourth image is identified through a segmentation network, and then the corresponding second mask is generated. Optionally, after obtaining the second mask, a Gaussian blur algorithm is used to process the edges of the second mask to reduce the edge hierarchy of the second mask, which facilitates subsequent image fusion.

[0112] Step 292: Use the second mask to extract the face region of the fourth image.

[0113] For example, the second mask is applied to the fourth image to extract the face region (the area where the target face is located) in the fourth image.

[0114] Step 293: Fuse the face region with the first image to obtain the second image corresponding to the corresponding audio frame. The lip shape of the target face in the second image corresponds to the speech content of the corresponding audio frame in the first audio.

[0115] For example, the extracted face region is fused with the background region of the first image (the region where the target face is not located) to obtain the second image.

[0116] The lip alignment method provided in this embodiment will be described exemplarily below. Figure 4 This is a schematic diagram of the data processing flow for a lip alignment method provided in one embodiment of this application. (Reference) Figure 4 Includes the target face ( Figure 4The first image (of a face) is processed by the Face Reconstruction module to obtain the first face reconstruction coefficients, where the Face Reconstruction module uses a face reconstruction network model. The MFCC features of each audio frame in the first audio are input into the Audio2Expression module to obtain the second expression coefficients, where the Audio2Expression module uses an expression recognition network model. Then, the first expression coefficients in the first face reconstruction coefficients are replaced with the second expression coefficients to obtain the second face reconstruction coefficients. Subsequently, the Refinement Module obtains the corrected second identity coefficients and third expression coefficients, and finally the third face reconstruction coefficients, where the Refinement Module uses a refined network model. Figure 5 This application provides a schematic diagram of a data processing flow for a refined network model, as illustrated in one embodiment. Figure 5 The second face reconstruction coefficients (β, α) can be obtained through the Refinement Module. R ρ R (pose coefficient), δ R (Texture coefficient) and γ R (Illumination factor) Corrected third face reconstruction factor (β) * α * ρ R δ R and γ R Next, based on the third face reconstruction coefficients, a third image is obtained using the Differential Render module, which is used to perform face reconstruction. Then, the third image is input into the pix2pix GAN module (i.e., a generative adversarial network model) to obtain a fourth image with optimized lip shape. Finally, a corresponding second image is obtained based on the fourth image and the first image, where the lip shape of the target face matches the corresponding audio frame in the first audio.

[0117] The above describes a process where, by first determining the second expression coefficients when the first audio is played and the first face reconstruction coefficients of the target face in the first image, a second face reconstruction coefficient containing the second expression coefficients is obtained. This involves adding the expression when the first audio is played to the target face. Then, a refined network model is used to correct the second face reconstruction coefficients. During the training of the refined network model, loss functions that improve the accuracy of the second three-dimensional facial keypoints and optimize the lip shape effect are used, making the third face reconstruction coefficients output by the refined network model more accurate, thereby improving alignment accuracy and effect. Furthermore, a generative adversarial model makes the style of the reconstructed target face more realistic. The training set used by the generative adversarial model applies a first mask, which can better optimize the area around the mouth, ensuring a realistic style. Finally, when fusing the fourth image and the first image, a second mask is used to ensure a more natural and smooth second image.

[0118] In one embodiment, the refined network model may also employ other structures, such as incorporating Arcface features to improve the reconstruction effect of 3D facial key points. In this case, when reconstructing the first 3D facial key points based on the second facial reconstruction coefficients, the method further includes: obtaining facial recognition features based on the second facial reconstruction coefficients, whereby the facial recognition features are the facial features reconstructed based on the second facial reconstruction coefficients. Correspondingly, inputting the first 3D facial key points, the first identity coefficient, and the second expression coefficient into the refined network model to output corrected second identity coefficients and third expression coefficients includes: inputting the first 3D facial key points, facial recognition features, the first identity coefficient, and the second expression coefficient into the refined network model to output corrected second identity coefficients and third expression coefficients.

[0119] Figure 6 This is a schematic diagram of a refined network model provided in one embodiment of this application, with reference to... Figure 6 ,and Figure 3 Compared to the refined network model shown, Figure 6 The refined network model shown also inputs face recognition features, obtained through ArcFace, a commonly used face recognition algorithm. After reconstructing the target face based on the second face reconstruction coefficients, ArcFace can be used to obtain the features of the target face. Currently, the recognized features are denoted as face recognition features. Then, the face recognition features, along with the first identity coefficient and the second expression coefficient, are input into the refined network model. The fine-tuning sub-model fuses the face recognition features through a fully connected layer with the features of the first 3D facial keypoints, the first identity coefficient, and the second expression coefficient, and obtains the second 3D facial keypoint L based on the fusion result.r Second 3D facial key points L r The accuracy is higher, and the second identity coefficient and third expression coefficient obtained based on the second three-dimensional facial key points are more accurate, which improves the lip alignment effect of the subsequently generated second image.

[0120] Figure 7 This is a schematic diagram of a lip alignment device provided in one embodiment of this application. (Reference) Figure 7 The lip alignment device includes: a first acquisition unit 301, a first generation unit 302, a second generation unit 303, a correction unit 304, and a reconstruction unit 305.

[0121] The system includes a first acquisition unit 301, which acquires a first image and a first audio, wherein the first image displays a target face; a first generation unit 302, which generates a first face reconstruction coefficient based on the first image, wherein the first face reconstruction coefficient includes a first identity coefficient and a first expression coefficient; a second generation unit 303, which generates a second expression coefficient corresponding to each audio frame when the first audio is played, and replaces the first expression coefficient with the second expression coefficient to obtain a second face reconstruction coefficient corresponding to each audio frame; a correction unit 304, which corrects the second face reconstruction coefficient based on a refined network model to obtain a third face reconstruction coefficient, wherein the third face reconstruction coefficient includes a corrected second identity coefficient and a third expression coefficient; and a reconstruction unit 305, which reconstructs a second image corresponding to each audio frame based on the third face reconstruction coefficient corresponding to each audio frame, wherein the lip shape of the target face in the second image corresponds to the speech content of the corresponding audio frame in the first audio.

[0122] In one embodiment of this application, the correction unit 304 includes: a key point reconstruction subunit, used to reconstruct a first three-dimensional facial key point according to a second facial reconstruction coefficient; and a coefficient correction subunit, used to input the first three-dimensional facial key point, a first identity coefficient, and a second expression coefficient into a refined network model, so that the refined network model outputs the corrected second identity coefficient and third expression coefficient.

[0123] In one embodiment of this application, the refined network model includes a fine-tuning sub-model and a reconstruction sub-model; the coefficient correction sub-unit includes: a key point fine-tuning sub-unit, used to input a first three-dimensional facial key point, a first identity coefficient, and a second expression coefficient into the fine-tuning sub-model, so that the fine-tuning sub-model outputs the fine-tuned second three-dimensional facial key point; and a coefficient reconstruction sub-unit, used to input the second three-dimensional facial key point into the reconstruction sub-model, so that the reconstruction sub-model outputs the reconstructed second identity coefficient and third expression coefficient.

[0124] In one embodiment of this application, the key point reconstruction subunit is further configured to: obtain facial recognition features based on the second facial reconstruction coefficients, wherein the facial recognition features are facial features reconstructed based on the second facial reconstruction coefficients; the coefficient correction subunit is specifically configured to input the first three-dimensional facial key points, facial recognition features, first identity coefficients and second expression coefficients into the refined network model, so that the refined network model outputs the corrected second identity coefficients and third expression coefficients.

[0125] In one embodiment of this application, the loss function used during the training of the refined network model includes:

[0126] l α =||α R -α * ||1

[0127] l β =||β-β * ||1

[0128]

[0129]

[0130]

[0131] Where, α R and α * Let β and β represent the first and second identity coefficients used during training, respectively. * Let L represent the second and third facial expression coefficients used during training, n represent the total number of facial keypoints, i represent the i-th facial keypoint, and L represent the third facial keypoint. r and L c Let k represent the first 3D facial keypoints and the second 3D facial keypoints used during training, respectively. i Let M represent the i-th 2D facial keypoint used for supervision. i Let represent the i-th second 3D facial keypoint used during training, E represent the set of keypoint pairs formed by corresponding facial keypoints in the upper and lower lips, (i,j) represent the keypoint pair formed by the i-th and j-th facial keypoints in E, s, ∏ and t represent the scale factor, rotation matrix and translation vector, respectively, and ||||1 represents the L1 norm.

[0132] In one embodiment of this application, the second generation unit 302 includes: a feature extraction subunit, used to extract features from the first audio to obtain audio features corresponding to each audio frame in the first audio; an expression coefficient determination subunit, used to obtain second expression coefficients corresponding to each audio frame when the first audio is played based on the audio features; and a coefficient replacement subunit, used to replace the first expression coefficients with the second expression coefficients to obtain second face reconstruction coefficients corresponding to each audio frame.

[0133] In one embodiment of this application, the expression coefficient determination subunit is specifically used to: input audio features into the expression recognition network model, so that the expression recognition network model obtains the second expression coefficient of the corresponding audio frame based on the audio features, wherein the loss function used by the expression recognition network model during training includes:

[0134] l exp =||β-β R ||2+λ||β||1

[0135] Where β and β R Let represent the second and first expression coefficients used during training, respectively; λ represent the adjustment parameter; ||||1 represents the L1 norm; and ||||2 represents the L2 norm.

[0136] In one embodiment of this application, the reconstruction unit 305 includes: an image rendering subunit, used to render a third image corresponding to each audio frame according to the third face reconstruction coefficients corresponding to each audio frame; an adversarial subunit, used to input the third image into a generative adversarial network model so that the generative adversarial network model outputs a fourth image optimized for the lip region; and an image fusion subunit, used to fuse the fourth image and the first image to obtain a second image corresponding to the corresponding audio frame, wherein the lip shape of the target face in the second image corresponds to the speech content of the corresponding audio frame in the first audio.

[0137] In one embodiment of this application, the dataset used for training the generative adversarial network model consists of multiple data pairs. Each data pair includes a first image and a corresponding third image used during training. Both the first image and the corresponding third image are masked. The first mask is used to extract the area below the nose in the image.

[0138] In one embodiment of this application, the image fusion subunit includes: a mask acquisition subunit for acquiring a second mask, the second mask having the same size as the fourth image; a face extraction subunit for extracting the face region of the fourth image using the second mask; and a region fusion subunit for fusing the face region and the first image to obtain a second image corresponding to the corresponding audio frame.

[0139] The lip alignment device provided above can be used to perform the lip alignment method provided in any of the above embodiments, and has the corresponding functions and beneficial effects.

[0140] It is worth noting that in the embodiments of the above-mentioned lip alignment device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.

[0141] Figure 8 This is a schematic diagram of a lip alignment device provided in one embodiment of this application. Figure 8 As shown, the lip alignment device includes a processor 40 and a memory 41. The number of processors 40 in the lip alignment device can be one or more. Figure 8 Taking a processor 40 as an example, in the lip alignment device, the processor 40 and the memory 41 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0142] The memory 41, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the lip alignment method in the embodiments of this application (e.g., the first acquisition unit 301, the first generation unit 302, the second generation unit 303, the correction unit 304, and the reconstruction unit 305 in the lip alignment device). The processor 40 executes various functional applications and data processing of the lip alignment device by running the software programs, instructions, and modules stored in the memory 41, thereby implementing the aforementioned lip alignment method.

[0143] The memory 41 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the lip alignment device. Furthermore, the memory 41 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 41 may further include memory remotely located relative to the processor 40, which can be connected to the lip alignment device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0144] The lip alignment device may also include an input device for receiving input numeric or character information and generating key signal inputs related to user settings and function control of the lip alignment device. It may also include an image acquisition device, an audio acquisition device, etc. The lip alignment device may also include an output device, which may include an audio playback device. The lip alignment device may also include a display screen. This display screen may or may not have touch functionality. The lip alignment device may also include a communication device to enable data communication with other devices.

[0145] The aforementioned lip alignment device includes a lip alignment apparatus, which can be used to perform any lip alignment method and has corresponding functions and beneficial effects.

[0146] Furthermore, embodiments of this application also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform relevant operations in the lip alignment method provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0147] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0148] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0150] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0151] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0152] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A lip alignment method, characterized in that, include: Acquire a first image and a first audio, wherein the first image displays a target face; A first face reconstruction coefficient is generated based on the first image, the first face reconstruction coefficient including a first identity coefficient and a first expression coefficient; The second expression coefficients corresponding to each audio frame when the first audio is played are generated based on the first audio, and the second expression coefficients are used to replace the first expression coefficients to obtain the second face reconstruction coefficients corresponding to each audio frame. The second face reconstruction coefficient is corrected according to the refined network model to obtain the third face reconstruction coefficient, which includes the corrected second identity coefficient and third expression coefficient. The second image corresponding to each audio frame is reconstructed based on the third face reconstruction coefficients corresponding to each audio frame, and the lip shape of the target face in the second image corresponds to the speech content of the corresponding audio frame in the first audio. The step of correcting the second face reconstruction coefficients according to the refined network model to obtain the third face reconstruction coefficients includes: The first three-dimensional facial key points are reconstructed based on the second facial reconstruction coefficient; The first three-dimensional facial key points, the first identity coefficient, and the second expression coefficient are input into the refined network model so that the refined network model outputs the corrected second identity coefficient and third expression coefficient. The refined network model corrects the first three-dimensional facial key points during the processing.

2. The lip alignment method according to claim 1, characterized in that, The refined network model includes a fine-tuning sub-model and a reconstruction sub-model; The step of inputting the first three-dimensional facial key points, the first identity coefficient, and the second expression coefficient into the refined network model, so that the refined network model outputs the corrected second identity coefficient and third expression coefficient, includes: The first three-dimensional facial key points, the first identity coefficient, and the second expression coefficient are input into the fine-tuning sub-model so that the fine-tuning sub-model outputs the fine-tuned second three-dimensional facial key points; The second three-dimensional facial key points are input into the reconstruction sub-model so that the reconstruction sub-model outputs the reconstructed second identity coefficient and third expression coefficient.

3. The lip alignment method according to claim 1, characterized in that, When reconstructing the first three-dimensional facial key points based on the second facial reconstruction coefficient, the method further includes: The facial recognition features are obtained based on the second facial reconstruction coefficients, and the facial recognition features are the features of the face reconstructed based on the second facial reconstruction coefficients; The step of inputting the first three-dimensional facial key points, the first identity coefficient, and the second expression coefficient into the refined network model, so that the refined network model outputs the corrected second identity coefficient and third expression coefficient, includes: The first three-dimensional facial key points, the facial recognition features, the first identity coefficient, and the second expression coefficient are input into the refined network model so that the refined network model outputs the corrected second identity coefficient and third expression coefficient.

4. The lip alignment method according to claim 1 or 2, characterized in that, The loss function used during the training of the refined network model includes: in, and These represent the first identity coefficient and the second identity coefficient used during training, respectively. and These represent the second and third expression coefficients used during training, respectively. n This indicates the total number of facial landmarks. i Indicates the first i Key facial features and These represent the first and second 3D facial key points used during training, respectively. Indicates the first i Two-dimensional facial key points used for supervision, Indicates the first term used during training. i A second 3D facial key point, E This represents the set of keypoint pairs formed by corresponding facial keypoints in the upper and lower lips. i , j )express E The Middle i Key facial features and the first j A key point pair composed of facial key points, s , and t Let || ...

5. The lip alignment method according to claim 1, characterized in that, The step of generating the second expression coefficients corresponding to each audio frame when the first audio is emitted includes: Feature extraction is performed on the first audio to obtain the audio features corresponding to each audio frame in the first audio. The second expression coefficients corresponding to each audio frame when the first audio is emitted are obtained based on the audio features.

6. The lip alignment method according to claim 5, characterized in that, The step of obtaining the second expression coefficients corresponding to each audio frame when the first audio is emitted based on the audio features includes: The audio features are input into the facial expression recognition network model, so that the facial expression recognition network model obtains the second facial expression coefficient of the corresponding audio frame based on the audio features; The loss function used in the training process of the facial expression recognition network model includes: in, and These represent the second and first expression coefficients used during training, respectively. The parameter represents the adjustment parameter, || ||1 represents the L1 norm, and || ||2 represents the L2 norm.

7. The lip alignment method according to claim 1, characterized in that, The step of reconstructing the second image corresponding to each audio frame based on the third face reconstruction coefficients corresponding to each audio frame includes: The third image corresponding to each audio frame is rendered based on the third face reconstruction coefficients corresponding to each audio frame; The third image is input into a generative adversarial network model so that the generative adversarial network model outputs a fourth image optimized for the lip region; The fourth image and the first image are fused together to obtain the second image corresponding to the corresponding audio frame.

8. The lip alignment method according to claim 7, characterized in that, The dataset used for training the generative adversarial network model consists of multiple data pairs. Each data pair includes a first image and a corresponding third image used during training. Both the first image and the corresponding third image are masked. The first mask is used to extract the area below the nose in the image.

9. The lip alignment method according to claim 7, characterized in that, The process of fusing the fourth image and the first image to obtain the second image corresponding to the corresponding audio frame includes: Obtain a second mask, which has the same size as the fourth image; The face region of the fourth image is extracted using a second mask; The facial region and the first image are fused together to obtain the second image corresponding to the corresponding audio frame.

10. A lip alignment device, characterized in that, include: The first acquisition unit is used to acquire a first image and a first audio, wherein the first image displays a target face; The first generation unit is configured to generate a first face reconstruction coefficient based on the first image, wherein the first face reconstruction coefficient includes a first identity coefficient and a first expression coefficient. The second generation unit is used to generate second expression coefficients corresponding to each audio frame when the first audio is emitted, and replace the first expression coefficients with the second expression coefficients to obtain the second face reconstruction coefficients corresponding to each audio frame. The correction unit is used to correct the second face reconstruction coefficients according to the refined network model to obtain the third face reconstruction coefficients, the third face reconstruction coefficients including the corrected second identity coefficients and the third expression coefficients; The reconstruction unit is used to reconstruct a second image corresponding to each audio frame based on the third face reconstruction coefficients corresponding to each audio frame, wherein the lip shape of the target face in the second image corresponds to the speech content of the corresponding audio frame in the first audio. The correction unit includes: a key point reconstruction subunit, used to reconstruct the first three-dimensional facial key points according to the second facial reconstruction coefficient; The coefficient correction subunit is used to input the first three-dimensional facial key points, the first identity coefficient, and the second expression coefficient into the refined network model, so that the refined network model outputs the corrected second identity coefficient and the third expression coefficient. The refined network model corrects the first three-dimensional facial key points during the processing.

11. A lip alignment device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the lip alignment method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the lip alignment method as described in any one of claims 1-9.

Images