Stylized digital human video generation method, electronic equipment and storage medium

Abstract

The invention provides a stylized digital human video generation method, electronic equipment and a storage medium. The method belongs to the technical field of image processing, and comprises the following steps: receiving a stylized digital human video generation instruction, the stylized digital human video generation instruction comprising a user photo, a target stylized type and a dubbing file; converting the user photo into a target stylized image according to the target stylized type; the target stylized image and the dubbing file are input into a pre-trained mouth shape driving model, the pre-trained mouth shape driving model extracts identity features of the target stylized image and audio features of the dubbing file, and a stylized digital human video is generated according to the identity features and the audio features; and obtaining a stylized digital human video output by the pre-trained mouth shape driving model, wherein the mouth shape driving of the stylized digital human video is synchronized with the dubbing file. According to the invention, stylized digital human videos which are more personalized, real-time and high in quality can be provided.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for generating a stylized digital human video, an electronic device and a storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, image processing and video generation technology have made significant progress. This progress has spawned digital population-driven video technology, which converts audio signals into corresponding lip animations and applies them to digital human images to generate highly realistic and interactive video content. However, with the diversification of user needs and the complexity of application scenarios, traditional digital population-driven technology that pursues accurate restoration rather than stylized expression is difficult to apply to specific scenarios that require giving digital humans a unique style.

[0003] In order to integrate stylized elements into digital human-driven videos, the existing technologies are mainly divided into two categories. The first method is to prepare stylized digital human images in advance for users to choose, and then use non-real-time lip-syncing drive technology to convert the user's voice or text input into digital human lip-syncing animation. Since the digital human images in this way are prepared in advance, they are less flexible and cannot fully meet the personalized needs of each user, and the real-time performance of video generation is poor. The second method requires users to dress themselves in the corresponding style, then record a video, and then convert the recorded video into a digital human video through image processing technology. Although this method can provide highly personalized results, its implementation process is relatively cumbersome and requires users to invest a lot of time and energy in dressing up and recording. This high-threshold operation process limits its universality and cannot meet the needs of all users.

[0004] Therefore, in order to realize the integration of stylized elements into digital population-driven videos, it is urgent to further develop and improve relevant algorithms and techniques to overcome the limitations of existing technologies. Summary of the invention

[0005] The purpose of this application is to provide a stylized digital human video generation method, electronic device and storage medium to solve the above problems.

[0006] To achieve the above objectives, in a first aspect, the present application proposes a method for generating a stylized digital human video, the method comprising: Receiving a stylized digital human video generation instruction, wherein the stylized digital human video generation instruction includes a user photo, a target stylization type, and a dubbing file; According to the target stylization type, converting the user photo into a target stylized image; Inputting the target stylized image and the dubbing file into a pre-trained lip-sync driven model, wherein the pre-trained lip-sync driven model extracts identity features of the target stylized image and audio features of the dubbing file, and generates a stylized digital human video according to the identity features and the audio features; The stylized digital human video output by the pre-trained lip-driven model is obtained, and the lip-driven stylized digital human video is synchronized with the dubbing file.

[0007] In some embodiments, the lip-sync drive model includes a generator network and a discriminator network, the discriminator network includes a lip synchronization discriminator, and before the target stylized image and the dubbing file are input into the pre-trained lip-sync drive model, the method further includes: Obtain a training set, wherein the training set includes digital population-type driving videos of various stylized types; By using the digital mouth-driven videos of the stylized types, the lip sync discriminator is trained to evaluate the synchronization between the lip sync and the audio; After the lip synchronization discriminator is trained, a reconstructed target frame is extracted from the training set, and the generator network is trained to generate a reconstructed image frame based on the reconstructed target frame of the masked lip shape part by masking the lip shape part of the reconstructed target frame, and a reconstruction loss is calculated based on the reconstructed target frame and the reconstructed image frame; Based on the trained lip synchronization discriminator, calculating the synchronization loss between the lip shape of the reconstructed image frame and the lip shape of the reconstructed target frame; Feedback is performed on the generator network based on the reconstruction loss and the synchronization loss until the training is completed.

[0008] In some embodiments, the discriminator network further includes a visual quality discriminator, and after masking the lip shape portion of the reconstructed target frame and training the generator network to generate a reconstructed image frame based on the reconstructed target frame of the masked lip shape portion, further includes: The image quality of the reconstructed image frame is supervised by the visual quality discriminator, and an adversarial loss is calculated, where the adversarial loss is used to measure the adversarial performance of the reconstructed image frame against the discriminator network in adversarial training; The step of performing feedback updating on the generator network based on the reconstruction loss and the synchronization loss until the training is completed comprises: The generator network is feedback updated by minimizing the weighted sum of the reconstruction loss, the synchronization loss and the adversarial loss until the training is completed.

[0009] In some implementations, the step of training the lip sync discriminator to evaluate the synchronization between lip sync and audio by driving the videos with digital populations of various stylized types includes: Extracting audio information of the digital population-driven videos of each stylized type, and cutting the audio information into audio blocks; Pairing the audio blocks with image frames in the digital population-driven videos of the respective stylized types to form matching pairs of the audio blocks and corresponding image frames and non-matching pairs of the audio blocks and non-corresponding image frames; The lip sync discriminator is trained to evaluate the synchronization between the lip sync and the audio through the matching pairs and the mismatching pairs.

[0010] In some implementations, the step of training the lip sync discriminator to assess the synchronization between lip sync and audio by using the matching pairs includes: Performing feature extraction on the audio blocks and image frames in the matching pair and the non-matching pair to generate an audio feature vector and an image feature vector; Calculating the cosine similarity between the audio feature vectors and the image feature vectors of the matching pair and the non-matching pair respectively; Based on the cosine similarity of the matching pair and the unmatching pair, the lip synchronization discriminator is feedback updated by minimizing the matching loss until the training is completed.

[0011] In some embodiments, extracting a reconstructed target frame from the training set, masking the lip shape portion of the reconstructed target frame, training the generator network to generate a reconstructed image frame based on the reconstructed target frame of the masked lip shape portion, and calculating the reconstruction loss based on the reconstructed target frame and the reconstructed image frame, comprises: Based on the training set, a reference frame, a reconstructed target frame, and an audio clip of a time dimension corresponding to the reconstructed target frame are obtained, and the reference frame, the reconstructed target frame, and the audio clip are used as inputs of the generator network, wherein the reference frame contains complete facial features, and the mouth shape part of the reconstructed target frame is masked; Obtain a reconstructed image frame generated by the generator network based on the reference frame, the reconstructed target frame and the audio clip; Calculating a reconstruction loss based on the reconstructed target frame and the reconstructed image frame in which the lip shape part is not masked; By minimizing the reconstruction loss, the generator network is fed back and updated until the training is completed.

[0012] In some implementations, the generator network includes an identity encoder, a speech encoder, and a face decoder, and based on the training set, obtaining a reference frame, a reconstructed target frame, and an audio clip of a time dimension corresponding to the reconstructed target frame, and using the reference frame, the reconstructed target frame, and the audio clip as inputs of the generator network, includes: Based on the training set, a reference frame, a reconstructed target frame, and an audio segment of a time dimension corresponding to the reconstructed target frame are obtained; splicing the reference frame and the reconstructed target frame together according to the channel dimension as input of the identity encoder to obtain the identity feature output by the identity encoder; and Using the audio segment as input of the speech encoder to obtain speech features output by the speech encoder; The identity feature and the voice feature are concatenated as input to the facial decoder to obtain a reconstructed image frame generated by the facial decoder.

[0013] In some embodiments, before obtaining a training set, wherein the training set includes digital population-type driving videos of various stylized types, the process includes: Obtain a collection of lip-sync-driven videos containing various stylized types of digital humans; Selecting lip-sync-driven videos that meet a preset audio and video synchronization standard from the lip-sync-driven video set to construct an initial training set; Based on the stylized labels of the lip-sync-driven videos in the initial training set, classifying the lip-sync-driven videos with the same stylized labels into the same dataset; Based on the classified data set, a training set is formed.

[0014] In a second aspect, the present application further proposes an electronic device, comprising: one or more processors; A memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to perform the stylized digital human video generation method as described above.

[0015] In a third aspect, the present application further proposes a storage medium, wherein the storage medium stores executable instructions, and when the instructions are executed by a processor, the processor executes the stylized digital human video generation method as described above.

[0016] Compared with the prior art, the beneficial effects of this application include: On the one hand, the present application can generate a digital human image of a target stylized type based on the received user photos, avoiding the limitations of making digital human images in advance, and improving flexibility and user satisfaction. On the other hand, the present application uses a pre-trained lip-driven model to generate a synchronized lip-driven video based on a dubbing file in real time. Compared with traditional non-real-time lip-driven technology, the real-time performance of video generation is greatly improved. On the third hand, to generate a stylized digital human video based on the technical solution of the present application, the user only needs to provide photos, dubbing files and select the target stylized type. This greatly simplifies the user's operating process and lowers the threshold for participation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope of the present application.

[0018] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for generating stylized digital human videos of the present application; Figure 2 A partial flow chart of an embodiment of the method for generating stylized digital human video of the present application; Figure 3 A partial flow chart of an embodiment of the method for generating stylized digital human video of the present application; Figure 4 A schematic diagram of a flow chart of an embodiment of a method for generating stylized digital human videos of the present application; Figure 5 A partial flow chart of an embodiment of the method for generating stylized digital human video of the present application; Figure 6 A schematic diagram of a flow chart of an embodiment of a method for generating stylized digital human videos of the present application; Figure 7 It is a structural schematic diagram of an electronic device involved in the method for generating a stylized digital human video in any embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] All terms (including technical and scientific terms) used in this application have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used here should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0021] For example, the terms "first", "second", etc. used in this application may be used in this document to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element.

[0022] For another example, the terms “include”, “comprising”, etc. used in this application indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0023] As mentioned above, in order to integrate stylized elements into digital population-driven videos, the existing technology is mainly divided into two categories of methods. The first method is to prepare stylized digital human images in advance for users to choose, and then use non-real-time lip-syncing driving technology to convert the user's voice or text input into digital human lip-syncing animation. Since the digital human images in this way are prepared in advance, they are less flexible and cannot fully meet the personalized needs of each user, and the real-time performance of video generation is poor. The second method requires users to dress themselves up in the corresponding style, then record a video, and then convert the recorded video into a digital human video through image processing technology. Although this method can provide highly personalized results, its implementation process is relatively cumbersome and requires users to invest a lot of time and energy in dressing up and recording. This high-threshold operation process limits its universality and cannot meet the needs of all users. Therefore, in order to integrate stylized elements into digital population-driven videos, it is urgent to further develop and improve relevant algorithms and technologies to overcome the limitations of the existing technology. To this end, the present application proposes a stylized digital human video generation method, electronic device and storage medium, which not only overcomes the limitations of the existing technology, but also can provide more personalized, real-time and high-quality stylized digital human videos.

[0024] Figure 1 This is a flow chart of an embodiment of stylized digital human video generation in this application. Figure 1 As shown, the stylized digital human video generation of this embodiment includes the following steps: Step S10, receiving a stylized digital human video generation instruction, wherein the stylized digital human video generation instruction includes a user photo, a target stylization type, and a dubbing file.

[0025] In this embodiment, the application scenarios of the stylized digital human video generation method cover a variety of intelligent terminals, including but not limited to smart phones, personal computers, robots and other electronic devices with data processing, network communication and program running functions. The specific online implementation form can be a website, mobile application (APP), WeChat applet, H5 page and API interface. Users access the Internet through these terminals and interact with the interactive interface that provides stylized digital human video generation services. This may involve opening a website, starting an APP, entering a small program or loading an H5 page in a browser.

[0026] In some implementations, the electronic device receives a stylized digital human video generation instruction triggered by a user based on an interactive interface, wherein the stylized digital human video generation instruction includes a photo upload instruction, a dubbing file upload instruction, and a target stylization type selection instruction.

[0027] The electronic device obtains the user photo uploaded by the user terminal associated with the photo upload instruction based on the photo upload instruction, wherein the user photo refers to the portrait picture uploaded by the user, and is used to generate a stylized digital human image that matches the user's portrait features.

[0028] The electronic device obtains the dubbing file uploaded by the user terminal associated with the dubbing file upload instruction based on the dubbing file upload instruction, and the dubbing file includes text content or voice content. If the dubbing file obtained by the electronic device is text content, a voice style selection interface is published through the interactive interface, a voice style selection instruction is received based on the voice style selection interface, and a voice file is generated according to the voice parameters associated with the voice style selection instruction and the text content, wherein the voice parameters include speech speed, intonation, gender, timbre and language, etc.

[0029] The electronic device determines the stylization type selected by the user as the target stylization type based on the target stylization type selection instruction. In this embodiment, the stylization types include cartoon style, realistic style, retro style, etc., wherein the realistic style includes but is not limited to professional style, casual style, dress style, cowboy style, lady style, sports style, etc.

[0030] Step S20: converting the user photo into a target stylized image according to the target stylization type.

[0031] In this embodiment, the electronic device obtains a style image associated with the target stylization type according to the target stylization type. The content features of the user photo and the style features of the style image are extracted through a pre-trained convolutional neural network. The content features are used to retain the structure and content of the user photo, and the style features are used to capture the texture and color information of the style image. The target stylized image is generated according to the content features and the style image.

[0032] In some embodiments, an initial stylized image is randomly generated based on the content features and the style image. A content loss of the initial stylized image and the user photo is calculated, and the content loss is used to measure the difference between the content features of the generated initial stylized image and the content features of the user photo. A style loss of the initial stylized image and the style image is calculated, and the style loss is used to measure the difference between the style features of the generated initial stylized image and the style features of the style image.

[0033] Using optimization algorithms such as gradient descent, the pixel values ​​of the generated stylized image are gradually adjusted by minimizing the combination of content loss and style loss. In each iteration, the content features and style features of the generated stylized image are calculated and compared with the content features of the user's photo and the style features of the style image to update the generated stylized image.

[0034] When it is detected that the combination of content loss and style loss tends to be stable, the iteration is confirmed to be finished, and the final generated target stylized image is obtained. The target stylized image is a high-quality image that retains the content features of the user's photo and has the style features of the style image.

[0035] It is understandable that before extracting features from user photos, the user photos may be preprocessed, and the preprocessing may include operations such as resizing and normalization, so as to facilitate subsequent stylization processing.

[0036] Step S30, inputting the target stylized image and the dubbing file into a pre-trained lip-driven model, wherein the pre-trained lip-driven model extracts the identity features of the target stylized image and the audio features of the dubbing file, and generates a stylized digital human video according to the identity features and the audio features.

[0037] In this embodiment, the electronic device inputs the received voice content, or the voice file generated based on the received text content, together with the target stylized image, into the pre-trained lip-driven model. After receiving the voice content or the voice file, the pre-trained lip-driven model extracts the audio features of the voice content or the voice file, and the audio features are features related to pronunciation, such as syllables, tones, etc. After receiving the target stylized image, the pre-trained lip-driven model extracts the identity features of the target stylized image, and the identity features refer to the facial features of the digital human in the target stylized image, and the facial features include features of key parts such as facial contours, eyes, nose, and mouth. A lip animation is generated based on the audio features and the facial features. A stylized digital human video is generated based on the lip animation and the target stylized image.

[0038] Step S40, obtaining a stylized digital human video output by a pre-trained lip-driven model, wherein the lip-driven stylized digital human video is synchronized with the dubbing file.

[0039] In this embodiment, the stylized digital human video output by the pre-trained lip-driven model is obtained, and the lip-driven stylized digital human video is synchronized with the voice content or voice file. The stylized digital human video is stored in a server or cloud storage. A download path and / or preview path of the stylized digital human video is created, and the download path and / or preview path of the stylized digital human video is sent to an interactive platform for access by a user terminal.

[0040] In this embodiment, on the one hand, the present application can generate a digital human image of a target stylized type based on the received user photo, avoiding the limitation of making a digital human image in advance, and improving flexibility and user satisfaction. On the other hand, the present application uses a pre-trained lip-driven model to generate a synchronized lip-driven video based on a dubbing file in real time. Compared with traditional non-real-time lip-driven technology, the real-time performance of video generation is greatly improved. On the third hand, based on the technical solution of the present application, a stylized digital human video is generated, and the user only needs to provide photos, dubbing files and select the target stylized type. This greatly simplifies the user's operation process and lowers the threshold for participation.

[0041] In one embodiment, the lip-sync drive model includes a generator network and a discriminator network, wherein the discriminator network includes a lip-sync discriminator, such as Figure 2 As shown, before step S30, the following steps are also included: Step A10: obtaining a training set, wherein the training set includes digital population-type driving videos of various stylized types.

[0042] In this embodiment, the digital human mouth-driven videos of various stylized types included in the training set are videos obtained by lip-driving the stylized digital human in the stylized digital human image based on a pre-made stylized digital human image through a non-real-time lip-driven technology (such as sadtalker, etc.).

[0043] In some embodiments, facial feature point detection is performed on pre-made stylized digital human images; based on the distribution of feature points, it is determined whether each pre-made stylized digital human image has an abnormal feature point distribution area, wherein the abnormal feature point distribution area includes an abnormal mouth opening (too large), abnormal eye closure (incomplete closure), abnormal jaw (distortion), etc.; stylized digital human images with abnormal feature point distribution areas are eliminated to obtain a stylized digital human image sample set, which is used for the subsequent production of a digital population type driven video set.

[0044] In some embodiments, Figure 3 As shown, before step A10, the following steps are included: Step A00, obtaining a lip-driven video set containing various stylized types of digital humans; It is important to understand that, compared to real people or highly realistic digital humans, the proportions and sizes of stylized digital humans’ facial features are beautified accordingly. If the existing digital human mouth shape driving technology is directly used for real-time lip-syncing driving, the problem of not being able to correctly drive the lip shape will occur due to the adjustment of the stylized digital human’s facial features.

[0045] This embodiment uses the existing digital human lip-driven technology to produce a lip-driven video set containing various stylized digital humans. Since the videos in the lip-driven video set are all pre-produced and the production process can be manually intervened, there are no videos with abnormal lip-driven in the lip-driven video set in this embodiment.

[0046] Step A01, selecting lip-driven videos that meet a preset audio and video synchronization standard from the lip-driven video set to construct an initial training set; The preset audio and video synchronization standard in this embodiment includes at least one of the following: delay threshold, which refers to the maximum allowable delay between audio and video; synchronization accuracy, which refers to the degree of synchronization between audio and video, and can be measured by calculating the synchronization rate or cross-correlation of audio and video; lip synchronization accuracy, which refers to the degree of synchronization between the digital human's lip shape changes and the pronunciation of the audio.

[0047] Step A02, based on the stylized labels of the lip-sync-driven videos in the initial training set, classifying the lip-sync-driven videos with the same stylized labels into the same data set; In this embodiment, the initial training set is traversed to read the stylized label of each lip-sync-driven video; a dictionary is created, in which the key is the stylized label and the value is a list of videos of the corresponding stylized type; and according to the stylized label of each lip-sync-driven video, each lip-sync-driven video is added to the data set of the corresponding stylized type (i.e., under the corresponding key of the dictionary).

[0048] Step A03: forming a training set based on the classified data set.

[0049] In this embodiment, a structured training set is formed based on the classified data set, which can not only be used uniformly for the subsequent training of the lip-driven model to improve the training efficiency, enhance the generalization ability of the model and simplify the evaluation process, but also each data set in the training set can be used separately for the training of the lip-driven model, thereby achieving targeted training, multi-style support and avoiding style entanglement.

[0050] Step A20: training the lip sync discriminator to evaluate the synchronization between lip sync and audio by using the digital population-driven videos of each stylized type.

[0051] In this embodiment, the lip-sync discriminator is trained to evaluate the synchronization between lip-sync and audio through the digital population driving videos of each stylized type. This training method can help the lip-sync discriminator better understand and identify the synchronization between digital populations of different styles and audio, thereby improving its evaluation accuracy and generalization ability.

[0052] In some embodiments, Figure 4 As shown, the step A20 includes: Step A21, extracting audio information of each stylized type of digital population-driven video, and cutting the audio information into audio blocks.

[0053] In this embodiment, the extracted audio information is cut into audio blocks of fixed length, and each audio block corresponds to an image frame in the digital population-driven video.

[0054] In some implementations, the frame rate and audio sampling rate of the digital population-driven video are obtained, and the cutting length is determined according to the frame rate and the audio sampling rate. The audio information is cut into a plurality of audio blocks of the cutting length.

[0055] For example, assuming that the frame rate of the digital population-driven video is 30fps and the audio sampling rate is 44.1kHz, the length of the audio block corresponding to one frame of the digital population-driven video is 33ms (ie, 33 * 44100 / 1000 = 1455 sampling points).

[0056] For example, assuming that the frame rate of the digital population-driven video is 25fps and the audio sampling rate is 16kHz, the length of the audio block corresponding to one frame of the digital population-driven video is 16ms (ie, 16 * 1000 / 16000 = 256 sampling points).

[0057] Step A22, pairing the audio blocks with image frames in the digital population-driven videos of each stylized type to form matching pairs of the audio blocks and corresponding image frames and non-matching pairs of the audio blocks and non-corresponding image frames.

[0058] Step A23: training the lip sync discriminator to evaluate the synchronization between lip sync and audio through the matching pairs and the mismatching pairs.

[0059] In this embodiment, a matching pair refers to an audio block and a corresponding image frame, which are synchronized in content. Through the matching pair, the lip sync discriminator can learn the correct correspondence between the audio block and the image frame. An unmatched pair refers to an audio block and an uncorresponding image frame, which are not synchronized in content. Through the unmatched pair, the model can learn the wrong correspondence between the audio block and the image frame.

[0060] In this embodiment, the matching pairs are used as positive samples, and the mismatching pairs are used as negative samples. By training the positive samples and negative samples simultaneously, the lip sync discriminator can learn to distinguish between synchronization and asynchrony, thereby improving its discrimination ability.

[0061] In some embodiments, Figure 5 As shown, the step A23 also includes: Step A24: extract features from the audio blocks and image frames in the matching pair and the non-matching pair to generate audio feature vectors and image feature vectors.

[0062] In this embodiment, feature extraction is performed on the audio blocks in the matching pairs and the non-matching pairs to generate audio feature vectors. The audio feature extraction methods that can be used include MFCC (Mel Frequency Cepstral Coefficient), Spectrogram, Chroma features, etc. And, feature extraction is performed on the image frames in the matching pairs and the non-matching pairs to generate image feature vectors. The image feature extraction methods that can be used include using pre-trained convolutional neural networks (CNN) such as VGG, ResNet, etc., or combining facial key point detection technology.

[0063] Step A25, respectively calculating the cosine similarity between the audio feature vectors and the image feature vectors of the matching pair and the non-matching pair.

[0064] The cosine similarity between the audio feature vector and the image feature vector of the matching pair and the non-matching pair is calculated respectively. Cosine similarity is a method to measure the similarity between two vectors. The closer the value is to 1, the higher the similarity.

[0065] Step A26, based on the cosine similarity of the matching pair and the unmatching pair, the lip synchronization discriminator is updated by feedback by minimizing the matching loss until the training is completed.

[0066] In this embodiment, the cosine similarity of the matching pair and the non-matching pair is mapped to a probability interval of [0, 1] and converted into a matching probability value. The matching loss is calculated based on the matching probability values ​​of the matching pair and the non-matching pair. The matching loss refers to a loss function used to measure the error made by the lip-sync discriminator when judging whether the audio block and the image frame match. By minimizing the matching loss, the lip-sync discriminator can be fed back and updated, thereby optimizing the performance of the lip-sync discriminator, so that the lip-sync discriminator can more accurately distinguish between matching pairs and non-matching pairs.

[0067] Step A30, after training the lip synchronization discriminator, extract the reconstructed target frame from the training set, mask the lip shape part of the reconstructed target frame, train the generator network to generate a reconstructed image frame based on the reconstructed target frame with the masked lip shape part, and calculate the reconstruction loss based on the reconstructed target frame and the reconstructed image frame.

[0068] In this embodiment, the reconstructed target frame with the masked lip shape is input into the generator network, and the generator network is trained to generate a reconstructed image frame that is as similar as possible to the reconstructed target frame, especially the lip shape. In order to evaluate the performance of the generator network, the reconstruction loss is calculated based on the reconstructed target frame and the reconstructed image frame, and the reconstruction loss is used to measure the difference between the reconstructed target frame and the generated reconstructed image frame.

[0069] In some embodiments, Figure 6 As shown, the step A30 includes: Step A31, based on the training set, obtain a reference frame, a reconstructed target frame, and an audio clip of the time dimension corresponding to the reconstructed target frame, and use the reference frame, the reconstructed target frame, and the audio clip as inputs of the generator network, wherein the reference frame contains complete facial features and the mouth shape part of the reconstructed target frame is masked.

[0070] Step A32, obtaining a reconstructed image frame generated by the generator network based on the reference frame, the reconstructed target frame and the audio clip.

[0071] In this embodiment, the reference frame is an image frame containing complete facial features, which is used to provide the structure and features of the lip shape part. The lip shape part of the reconstructed target frame has been masked, which is used to provide the image structure and features other than the lip shape part. The audio clip on the time dimension corresponding to the reconstructed target frame includes lip shape motion information.

[0072] The reference frame, the reconstructed target frame and the audio clip are input into the generator network together. The generator network generates a reconstructed image frame based on the structure and features of the lip part provided by the reference frame, the image structure and features other than the lip part provided by the reconstructed target frame, and the lip motion information.

[0073] Step A33, calculating the reconstruction loss based on the reconstructed target frame and the reconstructed image frame where the lip part is not masked.

[0074] Step A34, by minimizing the reconstruction loss, feedback update is performed on the generator network until the training is completed.

[0075] Based on the difference in pixel values ​​between the reconstructed target frame and the reconstructed image frame where the lip part is not masked, a reconstruction loss is calculated, and the reconstruction loss is used to adjust the parameters of the generator network to obtain better performance in the next iteration.

[0076] Step A40: Calculate the synchronization loss between the lip shape of the reconstructed image frame and the lip shape of the reconstructed target frame based on the trained lip synchronization discriminator.

[0077] In this embodiment, the reconstructed image frames generated by the generator network are continuous frames. Based on the training samples associated with the continuous frames and the reconstructed target frames, the synchronization loss is calculated. The synchronization loss takes into account the contextual information of the reconstructed image frames and the reconstructed target frames, and can more accurately measure the lip sync quality of the generated reconstructed image frames. The training samples are video clips with a fixed number of frames cut from the digital population-driven video in the training set. The fixed number of frames is the same as the number of the generated continuous frames.

[0078] Step A50, performing feedback update on the generator network based on the reconstruction loss and the synchronization loss until the training is completed.

[0079] A comprehensive loss function is calculated based on the reconstruction loss and the synchronization loss, and the comprehensive loss function includes hyperparameters for balancing the importance of the reconstruction loss and the synchronization loss. According to the comprehensive loss function, the parameters of the generator network are adjusted to obtain better performance in the next iteration until the comprehensive loss function is maintained at a minimum value and the training is completed.

[0080] In this embodiment, by creating a training set of digital human-driven videos of various stylized types, extracting features, training the generator network, calculating synchronization loss and performing feedback updates, the lip-sync-driven model can learn the characteristics of different styles and generate digital human videos with corresponding styles. At the same time, by calculating synchronization loss and performing feedback updates, it can be ensured that the generated digital human video maintains good lip sync with the original input data, thereby improving the realism and naturalness of the generated digital human video.

[0081] In one embodiment, the generator network includes an identity encoder, a speech encoder and a face decoder, and the step A31 includes: Based on the training set, a reference frame, a reconstructed target frame, and an audio clip of the time dimension corresponding to the reconstructed target frame are obtained; the reference frame and the reconstructed target frame are spliced ​​together according to the channel dimension as the input of the identity encoder to obtain the identity feature output by the identity encoder; and the audio clip is used as the input of the speech encoder to obtain the speech feature output by the speech encoder; the identity feature and the speech feature are spliced ​​together as the input of the facial decoder to obtain the reconstructed image frame generated by the facial decoder.

[0082] In this embodiment, the identity encoder, speech encoder and face decoder are all part of the neural network modules in the generator network. By separating the identity features and speech features and processing them by the identity encoder and speech encoder respectively, the feature information in the reconstructed target frame and audio clip can be better captured. This helps improve the ability of the face decoder to generate realistic and synchronized reconstructed image frames. In one embodiment, the discriminator network further includes a visual quality discriminator, and after step A30, further includes: The image quality of the reconstructed image frame is supervised by the visual quality discriminator, and an adversarial loss is calculated, where the adversarial loss is used to measure the adversarial performance of the reconstructed image frame against the discriminator network in adversarial training.

[0083] In this embodiment, compared with the lip shape discriminator used to determine the synchronization between lip shape and audio, the visual quality discriminator is used to determine the image quality of the generated reconstructed image frame to avoid blur or artifacts in the reconstructed image frame. The adversarial loss is used to measure the adversarial performance of the reconstructed image frame with the discriminator network in adversarial training, helping the generator network to learn a more accurate image reconstruction method and improve the visual quality of the generated reconstructed image frame.

[0084] The larger the adversarial loss, the easier it is for the visual quality discriminator to recognize the difference between the reconstructed image frame and the reconstructed target frame, the worse the performance of the generator, and the lower the quality of the generated reconstructed image frame. The smaller the adversarial loss, the less likely it is for the visual quality discriminator to recognize the difference between the reconstructed image frame and the reconstructed target frame, the better the performance of the generator, and the higher the quality of the generated reconstructed image frame. That is, the adversarial loss is inversely proportional to the image quality of the reconstructed image frame.

[0085] The step A50 comprises: The generator network is feedback updated by minimizing the weighted sum of the reconstruction loss, the synchronization loss and the adversarial loss until the training is completed.

[0086] In this embodiment, each iteration of the generator network will update the parameters of the generator network according to the gradients of the reconstruction loss, the synchronization loss, and the adversarial loss. This process will continue until certain stopping conditions are met, such as reaching a preset number of training rounds or the loss function converges. By balancing different types of losses, the generator network can generate high-quality and realistic high-resolution images while retaining image details.

[0087] In one embodiment, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the steps in the above-mentioned method embodiments.

[0088] In one embodiment, an electronic device is also provided, including one or more processors; a memory, in which one or more programs are stored, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the steps in the above-mentioned method embodiments.

[0089] In one embodiment, Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device for implementing an embodiment of the present application. The electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0090] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0091] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, including a computer-readable medium carrying instructions, in such an embodiment, the instructions can be downloaded and installed from a network through a communication part 709, and / or installed from a removable medium 711. When the instructions are executed by a central processing unit (CPU) 701, the various method steps described in the present application are executed.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0093] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the above claims, any one of the claimed embodiments may be used in any combination. The information disclosed in this background technology section is intended only to deepen the understanding of the overall background technology of the present application and should not be regarded as an admission or in any form of implication that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for generating stylized digital human videos, characterized in that: The method comprises: Receiving a stylized digital human video generation instruction, wherein the stylized digital human video generation instruction includes a user photo, a target stylization type, and a dubbing file; According to the target stylization type, converting the user photo into a target stylized image; Inputting the target stylized image and the dubbing file into a pre-trained lip-sync driven model, wherein the pre-trained lip-sync driven model extracts identity features of the target stylized image and audio features of the dubbing file, and generates a stylized digital human video according to the identity features and the audio features; The stylized digital human video output by the pre-trained lip-driven model is obtained, and the lip-driven stylized digital human video is synchronized with the dubbing file.

2. The method for generating stylized digital human videos according to claim 1, characterized in that: The lip-sync driven model includes a generator network and a discriminator network, the discriminator network includes a lip-sync discriminator, and before the target stylized image and the dubbing file are input into the pre-trained lip-sync driven model, the method further includes: Obtain a training set, wherein the training set includes digital population-type driving videos of various stylized types; By using the digital mouth-driven videos of the stylized types, the lip sync discriminator is trained to evaluate the synchronization between the lip sync and the audio; After the lip synchronization discriminator is trained, a reconstructed target frame is extracted from the training set, and the generator network is trained to generate a reconstructed image frame based on the reconstructed target frame of the masked lip shape part by masking the lip shape part of the reconstructed target frame, and a reconstruction loss is calculated based on the reconstructed target frame and the reconstructed image frame; Based on the trained lip synchronization discriminator, calculating the synchronization loss between the lip shape of the reconstructed image frame and the lip shape of the reconstructed target frame; Feedback is performed on the generator network based on the reconstruction loss and the synchronization loss until the training is completed.

3. The method for generating stylized digital human video according to claim 2, characterized in that: The discriminator network further includes a visual quality discriminator, and after masking the lip shape part of the reconstructed target frame and training the generator network to generate a reconstructed image frame based on the reconstructed target frame of the masked lip shape part, further includes: The image quality of the reconstructed image frame is supervised by the visual quality discriminator, and an adversarial loss is calculated, where the adversarial loss is used to measure the adversarial performance of the reconstructed image frame against the discriminator network in adversarial training; The step of performing feedback updating on the generator network based on the reconstruction loss and the synchronization loss until the training is completed comprises: The generator network is feedback updated by minimizing the weighted sum of the reconstruction loss, the synchronization loss and the adversarial loss until the training is completed.

4. The method for generating stylized digital human video according to claim 2, characterized in that: The step of training the lip sync discriminator to evaluate the synchronization between lip sync and audio by driving the digital population-type videos of each stylized type includes: Extracting audio information of the digital population-driven videos of each stylized type, and cutting the audio information into audio blocks; Pairing the audio blocks with image frames in the digital population-driven videos of the respective stylized types to form matching pairs of the audio blocks and corresponding image frames and non-matching pairs of the audio blocks and non-corresponding image frames; The lip sync discriminator is trained to evaluate the synchronization between the lip sync and the audio through the matching pairs and the mismatching pairs.

5. The method for generating stylized digital human video according to claim 4, characterized in that: The step of training the lip synchronization discriminator to evaluate the synchronization between the lip shape and the audio through the matching pair includes: Performing feature extraction on the audio blocks and image frames in the matching pair and the non-matching pair to generate an audio feature vector and an image feature vector; Calculating the cosine similarity between the audio feature vectors and the image feature vectors of the matching pair and the non-matching pair respectively; Based on the cosine similarity of the matching pair and the unmatching pair, the lip synchronization discriminator is feedback updated by minimizing the matching loss until the training is completed.

6. The method for generating stylized digital human video according to claim 2, characterized in that: The step of extracting a reconstructed target frame from the training set, masking the lip shape part of the reconstructed target frame, training the generator network to generate a reconstructed image frame based on the reconstructed target frame of the masked lip shape part, and calculating the reconstruction loss based on the reconstructed target frame and the reconstructed image frame includes: Based on the training set, a reference frame, a reconstructed target frame, and an audio clip of a time dimension corresponding to the reconstructed target frame are obtained, and the reference frame, the reconstructed target frame, and the audio clip are used as inputs of the generator network, wherein the reference frame contains complete facial features, and the mouth shape part of the reconstructed target frame is masked; Obtain a reconstructed image frame generated by the generator network based on the reference frame, the reconstructed target frame and the audio clip; Calculating a reconstruction loss based on the reconstructed target frame and the reconstructed image frame in which the lip shape part is not masked; By minimizing the reconstruction loss, the generator network is fed back and updated until the training is completed.

7. The method for generating stylized digital human video according to claim 6, characterized in that: The generator network includes an identity encoder, a speech encoder and a face decoder. Based on the training set, a reference frame, a reconstructed target frame and an audio clip of a time dimension corresponding to the reconstructed target frame are obtained, and the reference frame, the reconstructed target frame and the audio clip are used as inputs of the generator network, including: Based on the training set, a reference frame, a reconstructed target frame, and an audio segment of a time dimension corresponding to the reconstructed target frame are obtained; splicing the reference frame and the reconstructed target frame together according to the channel dimension as input of the identity encoder to obtain the identity feature output by the identity encoder; and Using the audio segment as input of the speech encoder to obtain speech features output by the speech encoder; The identity feature and the voice feature are concatenated as input to the facial decoder to obtain a reconstructed image frame generated by the facial decoder.

8. The method for generating stylized digital human video according to claim 2, characterized in that: Before obtaining a training set, wherein the training set includes digital population-driven videos of various stylized types, the method includes: Obtain a collection of lip-sync-driven videos containing various stylized types of digital humans; Selecting lip-sync-driven videos that meet a preset audio and video synchronization standard from the lip-sync-driven video set to construct an initial training set; Based on the stylized labels of the lip-sync-driven videos in the initial training set, classifying the lip-sync-driven videos with the same stylized labels into the same dataset; Based on the classified data set, a training set is formed.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the stylized digital human video generation method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium stores executable instructions, which, when executed by a processor, enable the processor to execute the method for generating a stylized digital human video according to any one of claims 1 to 8.

Images