Face generation method and device

Abstract

The present application discloses a facial generation method and device, which belongs to the field of digital humans. The facial generation method includes: obtaining target audio and target style feature sequences; the target style feature sequence is a facial feature sequence corresponding to the target style object when any audio is played; based on the target audio and the target style feature sequence, the target lip shape feature is predicted; the target lip shape feature is a lip shape feature that matches the target audio under the lip shape style corresponding to the target style object; based on the target lip shape feature and the target style feature sequence, an overall facial feature sequence is generated. The facial generation method of the present application can generate different lip shape features of different styles with large style differences, which is conducive to the realization of personalized style.

Description

Technical Field

[0001] The present application belongs to the field of digital humans, and in particular, relates to a face generation method and device. Background Art

[0002] Digital human technology is constantly developing, and digital humans are gradually replacing real people in scenarios such as introductions and interactions. In the field of digital human generation, related technologies often use large-scale oral data sets to train audio-driven lip models to make the models more generalizable. However, this method will cause the model to have consistent lip styles when reasoning on different images, which is not conducive to the realization of personalized styles. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a facial generation method and device, which can generate different lip features with different styles and large style differences, which is conducive to the realization of personalized style.

[0004] In a first aspect, the present application provides a face generation method, the method comprising:

[0005] Obtaining a target audio and a target style feature sequence; the target style feature sequence is a facial feature sequence corresponding to the target style object when any audio is played orally;

[0006] Based on the target audio and the target style feature sequence, a target lip shape feature is predicted; the target lip shape feature is a lip shape feature that matches the target audio under the lip shape style corresponding to the target style object;

[0007] Based on the target lip shape feature and the target style feature sequence, an overall facial feature sequence is generated.

[0008] According to the facial generation method of the present application, by performing style transfer on the target style features, the target lip shape features of the target style object when broadcasting the target audio are obtained. On the basis of generating lip shape features corresponding to each audio frame included in the target audio based on the target audio, the lip shape style of the target style object to be generated can be further combined to generate target lip shape features matching the lip shape style. For the same target audio, different lip shape features with different styles and large style differences can be generated, which significantly improves the authenticity of the generated facial features and is conducive to the realization of personalized style.

[0009] According to an embodiment of the present application, generating an overall facial feature sequence based on the target lip shape feature and the target style feature sequence includes:

[0010] The other facial features extracted from the initial facial features corresponding to the target style feature sequence are fused with the target lip shape features to obtain the overall facial feature sequence; the other facial features are the features in the initial facial features except the lip features.

[0011] According to an embodiment of the present application, the other facial features extracted from the initial facial features corresponding to the target style feature sequence are fused with the target lip shape features to obtain the overall facial feature sequence, including:

[0012] Masking the lip region in the initial facial feature to obtain a first facial feature;

[0013] The target mouth shape feature and the first facial feature are fused to obtain the overall facial feature sequence.

[0014] According to an embodiment of the present application, the step of fusing the target mouth shape feature and the first facial feature to obtain the overall facial feature sequence includes:

[0015] The target mouth shape feature and the first facial feature are input into a rendering module to obtain an overall facial feature sequence output by the rendering module; wherein,

[0016] The rendering module is trained based on a first target loss function, using sample lip shape features and sample facial mask feature sequences as samples and a sample overall facial feature sequence corresponding to the sample lip shape features and the sample facial mask feature sequence as sample labels.

[0017] According to an embodiment of the present application, predicting a target lip shape feature based on the target audio and the target style feature sequence includes:

[0018] The target audio and the target style feature sequence are input into the lip style transfer module to obtain the target lip features output by the lip style transfer module; wherein,

[0019] The lip-shape style transfer module is trained based on a second objective loss function, where the second objective loss function includes at least one of a lip-shape feature loss function and a lip-shape key point loss function.

[0020] According to one embodiment of the present application, the lip style transfer module is trained by the following steps:

[0021] Obtain sample audio and a spoken data set under a plurality of different sample style features corresponding to the sample audio;

[0022] Building a training sample based on the sample audio and a spoken data set corresponding to a target sample style feature among a plurality of different sample style features to obtain a plurality of training samples;

[0023] Based on the multiple training samples, the lip style transfer module is trained.

[0024] In a second aspect, the present application provides a face generation device, the device comprising:

[0025] The first processing module is used to obtain a target audio and a target style feature sequence; the target style feature sequence is a facial feature sequence corresponding to the target style object when any audio is played orally;

[0026] A second processing module is used to obtain a target lip shape feature based on the target audio and the target style feature sequence; the target lip shape feature is a lip shape feature that matches the target audio under the lip shape style corresponding to the target style object;

[0027] The third processing module is used to generate an overall facial feature sequence based on the target lip shape feature and the target style feature sequence.

[0028] According to the facial generation device of the present application, by performing style transfer on the target style features, the target lip shape features of the target style object when broadcasting the target audio are obtained. Based on the lip shape features corresponding to each audio frame included in the target audio generated based on the target audio, the lip shape style of the target style object to be generated can be further combined to generate target lip shape features matching the lip shape style. For the same target audio, different lip shape features with different styles and large style differences can be generated, which significantly improves the authenticity of the generated facial features and is conducive to the realization of personalized style.

[0029] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the face generation method as described in the first aspect above is implemented.

[0030] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the face generation method as described in the first aspect above is implemented.

[0031] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the face generation method as described in the first aspect above.

[0032] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0033] By performing style transfer on the target style features, we can obtain the target lip shape features of the target style object when it plays the target audio. On the basis of generating lip shape features corresponding to each audio frame included in the target audio based on the target audio, we can further combine the lip shape style of the target style object to be generated to generate target lip shape features that match the lip shape style. For the same target audio, different lip shape features with different styles and large style differences can be generated, which significantly improves the authenticity of the generated facial features and is conducive to the realization of personalized style.

[0034] Furthermore, by masking the lip area in the initial facial features to obtain other facial features, and then fusing the masked image with the target lip shape features, the operation is simple and convenient, and can avoid problems such as serious distortion caused by face-changing at a large angle.

[0035] Furthermore, by training the lip-shape style transfer module through the lip-shape feature loss function and the lip-shape key point loss function, the processing accuracy of the lip-shape style transfer module can be improved, the distortion in the lip-shape synthesis process can be effectively reduced, and the opening and closing of the lip shape can be ensured to be synchronized with the target audio and the lip shape continuity is high.

[0036] Furthermore, by training the rendering module with the reconstruction loss function and the perceptual loss function, the rendering smoothness of the rendering module can be improved, and the distortion in the lip synthesis process can be effectively reduced, so that in the final generated overall facial features, the opening and closing of the mouth is synchronized with the target audio, the lip continuity is high, and it has high realism.

[0037] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0039] Figure 1 This is one of the flowcharts of the face generation method provided in the embodiment of the present application;

[0040] Figure 2 This is the second flow chart of the face generation method provided in the embodiment of the present application;

[0041] Figure 3 This is the third flow chart of the face generation method provided in the embodiment of the present application;

[0042] Figure 4 is a schematic diagram of the structure of a face generation device provided in an embodiment of the present application;

[0043] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0045] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0046] The face generation method, face generation device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0047] The face generation method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0048] The terminal includes but is not limited to a portable communication device such as a mobile phone or a tablet computer. It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer.

[0049] The facial generation method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the facial generation method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The facial generation method provided in the embodiment of the present application is described below using an electronic device as an example of an execution subject.

[0050] like Figure 1 As shown, the face generation method includes: step 110, step 120 and step 130.

[0051] Step 110: Obtain target audio and target style feature sequence; the target style feature sequence is a facial feature sequence corresponding to the target style object when any audio is played orally;

[0052] In this step, the target audio is an audio file that needs to be synthesized by a digital human, such as one sentence or multiple sentences.

[0053] The target audio includes a plurality of audio frames.

[0054] The target style object is the style feature that the final synthesized digital human needs to present.

[0055] The target style object can be different people, animals or different cartoon characters, etc., which is not limited in this application.

[0056] The target style feature sequence includes a lip shape feature sequence corresponding to the target style object when the target style object plays a certain audio; each lip shape feature in the lip shape feature sequence corresponds one-to-one to each audio frame included in the audio.

[0057] It can be understood that the audio corresponding to the target style feature sequence and the target audio are different audio files.

[0058] The target style features corresponding to the target style feature sequence may include style features of the digital human's facial features and style features when speaking, including: the size and shape of the facial features, the texture of the skin, the degree of opening of the lips when speaking, and the corresponding unique lip shape style when speaking specific content.

[0059] Step 120: predicting a target lip shape feature based on the target audio and the target style feature sequence; the target lip shape feature is a lip shape feature that matches the target audio under the lip shape style corresponding to the target style object;

[0060] In this step, the target lip shape feature is the lip shape feature that matches the target audio under the lip shape style corresponding to the target style feature sequence.

[0061] Each audio frame included in the target audio corresponds to at least one lip shape feature, thereby obtaining a target lip shape feature sequence.

[0062] Lip style may include but is not limited to: lip shape, lip size, lip skin texture, teeth, and the degree of opening and closing of the mouth when speaking.

[0063] It is understandable that different stylistic features may correspond to different lip shape styles. For example, different people may have different mouth opening degrees when pronouncing the phoneme "O".

[0064] By migrating the style features of the target style object when speaking other audios, the target lip shape features of the target style object when speaking the target audio can be predicted.

[0065] For example, in the actual implementation process, a style transfer model can be used to learn and process the target audio and target style feature sequence to predict the target lip shape features.

[0066] Among them, the style transfer model can include any realizable artificial intelligence model, which is not limited in this application.

[0067] The style transfer model is pre-trained.

[0068] In some embodiments, step 120 may include:

[0069] Input the target audio and target style feature sequence into the lip style transfer module to obtain the target lip features output by the lip style transfer module; wherein,

[0070] The lip-shape style transfer module is trained based on a second objective loss function, where the second objective loss function includes at least one of a lip-shape feature loss function and a lip-shape key point loss function.

[0071] In this embodiment, the lip style transfer module may be a deep learning model or any other achievable model, which is not limited in this application.

[0072] The second objective loss function is a loss function used to train the lip-type style transfer module, so that the output of the lip-type style transfer module is closer to the true value.

[0073] In the actual implementation process, the target audio and target style feature sequence are input into the lip-shaped style transfer module, and the lip-shaped style transfer module can directly output the target lip-shaped features.

[0074] In some embodiments, the lip-sync style transfer module may include: an audio content encoder Ea, a style encoder Es, and a lip-sync style transfer decoder Ds.

[0075] The audio content encoder Ea is used to encode the input target audio, and the audio content encoder Ea can use a pre-trained Hubert encoder.

[0076] The style encoder Es and the lip-sync style transfer decoder Ds can be trained accordingly based on sample data in actual scenarios.

[0077] The style encoder Es is used to encode the style features.

[0078] In some embodiments, the lip-shape style transfer module may be trained based on a second objective loss function, and the second objective loss function may include at least one of a lip-shape feature loss function and a lip-shape key point loss function.

[0079] In some embodiments, the lip shape feature loss function can be expressed as:

[0080] L feat =||D s (E s (x i ),E a (y))-F e ||2

[0081] Among them, L feat is the lip shape feature loss function; D s is the lip-sync style transfer decoder; D s () represents the lip shape features reconstructed using the lip shape style transfer decoder; E s is the style encoder; E s () represents the features extracted using the style encoder; E a F is the audio content encoder; e is the sample mouth shape feature; x i are aligned video frames with different lip shapes (i.e., facial features); y is the sample audio input to the audio content encoder.

[0082] In some embodiments, the lip key point loss function can be expressed as:

[0083] L lmk =||K(D s (E s (x i ),E a (y)))-K(F e )||2

[0084] Among them, L lmk is the lip shape key point loss function; D s is the lip-sync style transfer decoder; E s is the style encoder; E a F is the audio content encoder; e is the sample mouth shape feature; K(*) is used to render the sample mouth shape feature into a mouth shape key point function; x i is the i-th sample style feature of the input style encoder; y is the sample audio of the input audio content encoder.

[0085] In some embodiments, Deep3DMM technology may be used to extract lip shape features from a sample video, and the extracted lip shape features may be rendered to obtain a lip shape key point function.

[0086] In some embodiments, the lip style transfer module may be trained based on a second objective loss function, and the second objective loss function may be:

[0087] L total =L feat +λL lmk

[0088] Among them, L total is the second objective loss function; L feat is the lip shape feature loss function; L lmk is the lip key point loss function; λ is the second target weight.

[0089] In this embodiment, the second objective loss function may include a lip feature loss function and a lip key point loss function.

[0090] The second target weight may be obtained based on experimental verification, or may be determined based on historical experience. For example, the value range of the second target weight may be set to be between 0.01 and 0.04.

[0091] During the training process, the lip shape feature loss function and the lip shape key point loss function can be integrated to perform overall training on the lip shape style transfer module.

[0092] According to the facial generation method provided in the embodiment of the present application, the lip style transfer module is trained by using the lip feature loss function and the lip key point loss function, so as to improve the processing accuracy of the lip style transfer module, effectively reduce the distortion in the lip synthesis process, and ensure that the opening and closing of the lip shape is synchronized with the target audio and the lip shape continuity is high.

[0093] The following is an explanation of the training method of the lip-sync style transfer module.

[0094] like Figure 2 As shown, in some embodiments, the lip style transfer module can be trained by the following steps:

[0095] Obtain sample audio and a spoken data set with multiple different sample style features corresponding to the sample audio;

[0096] Building training samples based on the sample audio and the spoken data set corresponding to the target sample style features among the multiple different sample style features to obtain multiple training samples;

[0097] Based on multiple training samples, train the lip style transfer module.

[0098] In this embodiment, the sample audio may be any audio file, and the sample audio may be multiple audio files.

[0099] The oral data set includes: the facial features corresponding to a certain sample style feature when the sample audio is spoken, and the sample lip shape features corresponding to each audio frame included in the sample audio.

[0100] In the actual implementation process, the oral data set can be obtained by extracting the sample overall facial feature sequence corresponding to a certain sample style feature when the sample audio is spoken, wherein the number of image frames included in the sample overall facial feature sequence is the same as the number of audio frames included in the sample audio.

[0101] The target sample style feature may be any style feature among a variety of different sample style features.

[0102] Constructing training samples based on sample audio and spoken data corresponding to target sample style features among a plurality of different sample style features can include: establishing positive samples based on target audio frames in the sample audio and video frames corresponding to the target audio frames, and establishing negative samples based on target audio frames in the sample audio and the other video frames.

[0103] It is understandable that in the actual implementation process, different image spoken video data can be obtained, and by separating the audio and video of each image spoken video data, the sample audio and spoken data set corresponding to the image spoken video data can be obtained.

[0104] The following is an explanation using a certain image oral video data as an example.

[0105] In the actual implementation process, audio data aligned with the video data included in the oral video data of the image and aligned face pictures of other different lip shapes under the image can be obtained as input of the lip style transfer module to train the lip style transfer module.

[0106] Step 130: Generate an overall facial feature sequence based on the target lip shape feature and the target style feature sequence.

[0107] In this step, the target lip shape feature may be expressed as a feature based on a time series change, and the target lip shape feature changes accordingly with the change of the audio frame included in the target audio.

[0108] The overall facial feature sequence is a sequence that includes the facial features of the style corresponding to the target style feature sequence, and the lip shape can change accordingly based on the change of the audio, such as Figure 3 The final face image is shown below.

[0109] Continue to refer Figure 3In the actual implementation process, for an audio file to be synthesized, such as a speech, the speech can be input into the audio content encoder Ea for encoding to obtain an encoded target audio frame sequence; select the desired target style feature sequence, such as obtaining any segment such as Figure 3 The target image lip style sequence is shown, and the target image lip style sequence is input into the style encoder Es for encoding to obtain the lip style sequence of the target image when speaking any audio segment.

[0110] Then, the target audio frame sequence and the lip shape style sequence are input into the lip shape style transfer decoder Ds for processing, so as to transfer the lip shape style of the target image to the lip shape features corresponding to the target audio frame sequence, thereby obtaining the target lip shape features.

[0111] By fusing the target lip shape features and the target style feature sequence corresponding to the target image, the overall facial features can be generated.

[0112] It is understandable that the same sentence may have different effects when spoken by different people.

[0113] During the research and development process, the inventors discovered that in the related technology, the audio-driven lip shape model is mainly trained on a large-scale oral broadcast data set. This method easily leads to the average face problem of the model. That is, when the model is inferring, the lip opening and closing amplitude and lip and tooth similarity between different images are extremely high, which is not conducive to the realization of personalized style, thus affecting the user experience.

[0114] In the present application, style transfer is introduced on the basis of lip shape prediction. On the basis of generating lip shape features corresponding to each audio frame included in the target audio based on the target audio, the lip shape style of the image to be generated is further combined to generate target lip shape features matching the image style. For the same target audio, different lip shape features with different styles and large style differences can be generated, which is conducive to the realization of personalized style.

[0115] In addition, it can also make the lip and teeth synthesis more similar to the original image, eliminating the need for image fine-tuning, thereby improving the realism of the generated digital human.

[0116] According to the facial generation method provided in the embodiment of the present application, by performing style transfer on the target style features, the target lip shape features of the target style object when broadcasting the target audio are obtained. On the basis of generating lip shape features corresponding to each audio frame included in the target audio based on the target audio, the lip shape style of the target style object to be generated can be further combined to generate target lip shape features matching the lip shape style. For the same target audio, different lip shape features with different styles and large style differences can be generated, which significantly improves the authenticity of the generated facial features and is conducive to the realization of personalized style.

[0117] In some embodiments, step 130 may include:

[0118] The other facial features extracted from the initial facial features corresponding to the target style feature sequence are fused with the target lip shape features to obtain the overall facial features; the other facial features are the features in the initial facial features except the lip features.

[0119] In this embodiment, other facial features are features of the initial facial features except for the lip features, such as eyes, nose, ears and other features.

[0120] After obtaining other facial features, the extracted facial features are fused with the lip shape features corresponding to the predicted target audio to obtain the overall facial features.

[0121] In some embodiments, other facial features extracted from the initial facial features corresponding to the target style feature sequence are fused with the target lip shape features to obtain the overall facial features, which may include:

[0122] Masking the lip region in the initial facial feature to obtain a first facial feature;

[0123] The target mouth shape feature and the first facial feature are fused to obtain the overall facial feature.

[0124] In this embodiment, the first facial feature is the feature of the initial facial features except the lip feature.

[0125] The lip area in the initial facial features is masked to obtain other facial features, and then the masked image is merged with the target lip shape features. The operation is simple and convenient, and can avoid the serious distortion caused by large-angle face swapping caused by the existing technology of performing an overall face swap operation on the face in the video after generating the video.

[0126] In actual application, there is no need to retrain the model for new audio data and image data. The model can adaptively transfer the lip style and lip shape features according to the new image features to reconstruct the facial image and obtain the overall facial features.

[0127] In some embodiments, the reconstructed face image can be expressed as:

[0128]

[0129] Among them, D s is the lip-sync style transfer decoder; E s is the style encoder; E a E is the audio content encoder; v() indicates the use of identity encoder to extract identity features; I s is a sequence of masked facial features (ie, the first facial feature).

[0130] Among them, E v The input of () is a sequence of masked facial features, E v () is used to blend and connect the input value with other areas such as the background (such as the hair and background of the target style object) and perform posture simulation.

[0131] In the actual implementation process, other facial features can be integrated with the target lip shape features through a neural network model or other realizable models to obtain the overall facial features. Please do not train it here.

[0132] In some embodiments, the target mouth shape feature and the first facial feature are fused to obtain an overall facial feature, including:

[0133] Input the target mouth shape feature and the first facial feature into the rendering module, and obtain the overall facial feature output by the rendering module; wherein,

[0134] The rendering module is trained based on the first objective loss function, with sample lip shape features and sample facial mask feature sequences as samples and sample overall facial feature sequences corresponding to the sample lip shape features and sample facial mask feature sequences as sample labels.

[0135] In this embodiment, the sample lip shape features can be obtained by extracting the lip shape features in the sample video using the Deep3DMM technology.

[0136] The sample facial mask feature sequence can be obtained by masking the lip region in the sample overall facial feature sequence.

[0137] The rendering module may include: an identity encoder Ev and a rendering decoder G.

[0138] Among them, both the identity encoder Ev and the rendering decoder G need to be trained.

[0139] In some embodiments, the first objective loss function may be expressed as:

[0140] L render =L recon +βL vgg

[0141] Among them, L render is the first objective loss function; L recon is the reconstruction loss function; L vgg is the perceptual loss function; β is the first target weight.

[0142] In this embodiment, the first target weight may be obtained based on experimental verification, or may be determined based on historical experience. For example, the value range of the first target weight may be set to between 0 and 1.

[0143] In some embodiments, the reconstruction loss function may be expressed as:

[0144]

[0145] Among them, L recon is the reconstruction loss function; I is the facial features reconstructed by the rendering decoder G; i is the real facial feature of the i-th frame of the original video (i.e., the i-th frame in the overall facial feature sequence of the sample); M is the total number of frames of the original video, which is the same as the number of frames of the input sample audio; i is the index of each frame in the original video.

[0146] In some embodiments, the perceptual loss function may be expressed as:

[0147]

[0148] Among them, L vgg is the perceptual loss function; is the facial features reconstructed by the rendering decoder G; I i is the real facial feature of the i-th frame of the original video (i.e., the i-th frame in the overall facial feature sequence of the sample); Vn() represents the n-th layer feature extractor of the VGG19 pre-trained model; M is the total number of frames of the original video, which is the same as the number of audio frames of the input sample; N is the total number of feature extractors in the VGG19 pre-trained model; i is the index of each frame in the original video.

[0149] During the training process, the reconstruction of the face image based on the rendering module can be expressed as:

[0150]

[0151] Among them, F e is the sample mouth shape feature; E v () indicates the use of identity encoder to extract identity features; I s is a sequence of masked facial features; G(*) represents the reconstruction of the face using the rendering decoder.

[0152] According to the facial generation method provided in the embodiment of the present application, by training the rendering module with the reconstruction loss function and the perceptual loss function, the rendering smoothness of the rendering module can be improved, and the distortion in the lip synthesis process can be effectively reduced, so that in the final generated overall facial features, the opening and closing of the lip shape is synchronized with the target audio and the lip shape continuity is high, and it has a high degree of realism.

[0153] The method provided in the embodiment of the present application can realize the adaptive transfer of lip style and lip shape features when only audio and partial lip shape pictures of the image are provided; the universal audio-driven lip shape model has different lip shape styles for different images while having high generalization, thereby improving the realism and style diversity of the lip shapes generated by the virtual digital human; in addition, only one training is required, and in subsequent applications, for new style features, there is no need to re-train, and the style sequence only needs to be input into the pre-trained model to obtain the style sequence and the overall facial features under the audio, which is simple and convenient to operate.

[0154] The face generation method provided in the embodiment of the present application can be executed by a face generation device. In the embodiment of the present application, the face generation device provided in the embodiment of the present application is described by taking the face generation method executed by the face generation device as an example.

[0155] The embodiment of the present application also provides a face generating device.

[0156] like Figure 4 As shown, the face generating device includes: a first processing module 410 , a second processing module 420 and a third processing module 430 .

[0157] The first processing module 410 is used to obtain a target audio and a target style feature sequence; the target style feature sequence is a facial feature sequence corresponding to the target style object when any audio is played orally;

[0158] The second processing module 420 is used to predict the target lip shape feature based on the target audio and the target style feature sequence; the target lip shape feature is the lip shape feature that matches the target audio under the lip shape style corresponding to the target style object;

[0159] The third processing module 430 is used to generate an overall facial feature sequence based on the target lip shape feature and the target style feature sequence.

[0160] According to the facial generation device provided in the embodiment of the present application, by performing style transfer on the target style features, the target lip shape features of the target style object when broadcasting the target audio are obtained. On the basis of generating lip shape features corresponding to each audio frame included in the target audio based on the target audio, the lip shape style of the target style object to be generated can be further combined to generate target lip shape features matching the lip shape style. For the same target audio, different lip shape features with different styles and large style differences can be generated, which significantly improves the authenticity of the generated facial features and is conducive to the realization of personalized style.

[0161] In some embodiments, the third processing module 430 may also be used to:

[0162] The other facial features extracted from the initial facial features corresponding to the target style feature sequence are fused with the target lip shape features to obtain an overall facial feature sequence; the other facial features are the features in the initial facial features except the lip features.

[0163] In some embodiments, the third processing module 430 may also be used to:

[0164] Masking the lip region in the initial facial feature to obtain a first facial feature;

[0165] The target mouth shape features and the first facial features are fused to obtain the overall facial feature sequence.

[0166] In some embodiments, the third processing module 430 may also be used to:

[0167] Input the target mouth shape feature and the first facial feature into the rendering module, and obtain the overall facial feature sequence output by the rendering module; wherein,

[0168] The rendering module is trained based on the first objective loss function, with sample lip shape features and sample facial mask feature sequences as samples and sample overall facial feature sequences corresponding to the sample lip shape features and sample facial mask feature sequences as sample labels.

[0169] In some embodiments, the second processing module 420 may also be used to:

[0170] Input the target audio and target style feature sequence into the lip style transfer module to obtain the target lip features output by the lip style transfer module; wherein,

[0171] The lip-shape style transfer module is trained based on a second objective loss function, where the second objective loss function includes at least one of a lip-shape feature loss function and a lip-shape key point loss function.

[0172] In some embodiments, the apparatus may further include a fourth processing module, configured to:

[0173] Obtain sample audio and a spoken data set with multiple different sample style features corresponding to the sample audio;

[0174] Building training samples based on the sample audio and the spoken data set corresponding to the target sample style features among the multiple different sample style features to obtain multiple training samples;

[0175] Based on multiple training samples, train the lip style transfer module.

[0176] The face generation device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0177] The face generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0178] The face generation device provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0179] In some embodiments, Figure 5 As shown, the embodiment of the present application also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, each process of the above-mentioned face generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0180] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0181] The embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned face generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0182] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0183] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned face generation method when executed by a processor.

[0184] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0185] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned face generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0186] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0187] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0189] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0190] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0191] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A face generation method, characterized in that: include: Obtain target audio and target style feature sequence; The target style feature sequence is a facial feature sequence corresponding to the target style object when any audio is played orally; Based on the target audio and the target style feature sequence, predict a target lip shape feature; The target lip shape feature is a lip shape feature that matches the target audio under the lip shape style corresponding to the target style object; Generate an overall facial feature sequence based on the target lip shape feature and the target style feature sequence; The step of predicting a target lip shape feature based on the target audio and the target style feature sequence comprises: Inputting the target audio and the target style feature sequence into a lip-shaped style transfer module, and obtaining the target lip-shaped features output by the lip-shaped style transfer module; The lip style transfer module includes: an audio content encoder, a style encoder and a lip style transfer decoder; The audio content encoder is used to encode the input target audio, the style encoder is used to encode the style features, and the lip style transfer decoder is used to generate target lip features based on the encoded audio features and style features; The reconstructed face image is expressed as: in, Decoder for lip-sync style transfer; is the style encoder; is an audio content encoder; It means that the identity features are extracted using the identity encoder; is a sequence of masked facial features, are aligned video frames with different lip shapes, and y is the sample audio input to the audio content encoder; in, The input is a sequence of masked facial features. Used to blend and connect input values ​​with other areas including the background, and perform posture simulation.

2. The face generation method according to claim 1, characterized in that: The step of generating an overall facial feature sequence based on the target lip shape feature and the target style feature sequence comprises: The other facial features extracted from the initial facial features corresponding to the target style feature sequence are fused with the target lip shape features to obtain the overall facial feature sequence; the other facial features are the features in the initial facial features except the lip features.

3. The face generation method according to claim 2, characterized in that: The other facial features extracted from the initial facial features corresponding to the target style feature sequence are fused with the target lip shape features to obtain the overall facial feature sequence, including: Masking the lip region in the initial facial feature to obtain a first facial feature; The target mouth shape feature and the first facial feature are fused to obtain the overall facial feature sequence.

4. The face generation method according to claim 3, characterized in that: The step of fusing the target mouth shape feature and the first facial feature to obtain the overall facial feature sequence includes: The target mouth shape feature and the first facial feature are input into a rendering module to obtain an overall facial feature sequence output by the rendering module; wherein, The rendering module is trained based on a first target loss function, using sample lip shape features and sample facial mask feature sequences as samples and a sample overall facial feature sequence corresponding to the sample lip shape features and the sample facial mask feature sequence as sample labels.

5. The face generation method according to any one of claims 1 to 4, characterized in that: The lip-shape style transfer module is trained based on a second objective loss function, where the second objective loss function includes at least one of a lip-shape feature loss function and a lip-shape key point loss function.

6. The face generation method according to claim 5, characterized in that: The lip style transfer module is trained by the following steps: Obtain sample audio and a spoken data set under a plurality of different sample style features corresponding to the sample audio; Building a training sample based on the sample audio and a spoken data set corresponding to a target sample style feature among a plurality of different sample style features to obtain a plurality of training samples; Based on the multiple training samples, the lip style transfer module is trained.

7. A facial generation device, characterized in that: include: A first processing module, used to obtain a target audio and a target style feature sequence; The target style feature sequence is a facial feature sequence corresponding to the target style object when any audio is played orally; A second processing module, configured to predict a target lip shape feature based on the target audio and the target style feature sequence; The target lip shape feature is a lip shape feature that matches the target audio under the lip shape style corresponding to the target style object; A third processing module, configured to generate an overall facial feature sequence based on the target lip shape feature and the target style feature sequence; The step of predicting a target lip shape feature based on the target audio and the target style feature sequence comprises: Inputting the target audio and the target style feature sequence into a lip-shaped style transfer module, and obtaining the target lip-shaped features output by the lip-shaped style transfer module; The lip style transfer module includes: an audio content encoder, a style encoder and a lip style transfer decoder; The audio content encoder is used to encode the input target audio, the style encoder is used to encode the style features, and the lip style transfer decoder is used to generate target lip features based on the encoded audio features and style features; The reconstructed face image is expressed as: in, Decoder for lip-sync style transfer; is the style encoder; is an audio content encoder; It means that the identity features are extracted using the identity encoder; is a sequence of masked facial features, are aligned video frames with different lip shapes, and y is the sample audio input to the audio content encoder; in, The input is a sequence of masked facial features. Used to blend and connect input values ​​with other areas including the background, and perform posture simulation.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the face generation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the face generation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the face generation method according to any one of claims 1 to 6 is implemented.

Images