Video generation method, apparatus, and storage medium

Abstract

The application relates to a video generation method, device and storage medium. The method comprises the following steps: obtaining a target person image of a target person, target audio set by the target person and a preset driving video, the face of a person in the driving video is dynamic; dynamic face features of the person in the driving video are migrated to the target person image to obtain a target person dynamic video, the face of the target person in the target person dynamic video is dynamic; an audio-lip synchronous video is generated according to the target audio and the target person dynamic video, dynamic face features in the audio-lip synchronous video are synchronous with the target audio; and the dynamic face features in the audio-lip synchronous video and picture quality are enhanced based on the target person image to obtain a target person audio-lip synchronous video. According to the embodiment of the application, a single target person image can be driven by any audio and any video, and a target person audio-lip synchronous video with high picture quality and high audio-lip synchronization can be effectively generated.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video generation method, apparatus and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, virtual digital human technology that simulates the appearance, speech, and behavior of real people is a prerequisite for the realization of the "metaverse" concept. Since the quality of digital virtual human technology depends on the user experience, high anthropomorphism and high visualization accuracy are basic requirements for virtual digital human technology.

[0003] However, traditional virtual digital human technology requires recording a large amount of video data of a real person to train the virtual human generation model in order to generate a virtual character based on that real person. This is necessary to ensure that the virtual character generated by the trained model has a good anthropomorphic effect. Furthermore, virtual characters are usually presented to users in the form of videos, and the quality of the videos depends on the pixel size of each frame. The virtual human generation model itself is very complex, and feature learning of high-definition images will significantly increase the computational cost. However, the computing resources of the device are limited, resulting in low image quality, lack of realism, and poor generalization ability of the virtual digital human videos output by the virtual human generation model. Summary of the Invention

[0004] In view of this, a video generation method, apparatus and storage medium are proposed.

[0005] In a first aspect, embodiments of this application provide a video generation method, the method comprising: acquiring a target person image, a target audio set by the target person, and a preset driving video, wherein the face of the person in the driving video is dynamic; transferring the dynamic facial features of the person in the driving video to the target person image to obtain a target person dynamic video, wherein the face of the target person in the target person dynamic video is dynamic, and the dynamic facial features include at least one of the following: expression, action, and lip shape; generating a lip-sync video based on the target audio and the target person dynamic video, wherein the dynamic facial features in the lip-sync video are synchronized with the target audio; and enhancing the dynamic facial features and image quality in the lip-sync video based on the target person image to obtain a target person lip-sync video.

[0006] According to the embodiments of this application, a single target person image can be driven by any audio and any video to generate a target person's lip-sync video. That is, by using a single target person image, target audio, and driving video, dynamic facial feature transfer, lip-sync, feature enhancement, and image quality enhancement are performed sequentially to generate a target person's lip-sync video with high image quality and high lip-sync, which is equivalent to generating a virtual digital human video of the target person. It can decouple the person image, audio, and video from the model, so that there is no need to collect a large number of customized sample videos of the target person to train a specific model to generate a virtual digital human video. It can generate a virtual dynamic image of the target person. The transfer and generalization ability is strong. In particular, through the process of facial dynamic feature enhancement and image quality enhancement, the virtual dynamic image in the target person's lip-sync video can be made more realistic and the image quality is higher, that is, the clarity and accuracy are higher.

[0007] According to the first aspect, in a first possible implementation of the video generation method, the step of transferring the dynamic facial features of a person in the driving video to the target person image to obtain a dynamic video of the target person includes: performing keypoint detection on the driving video and the target person image respectively to obtain dynamic facial keypoints of the person in the driving video and static facial keypoints of the target person in the target person image; determining a first keypoint mapping relationship between the driving video and the target person image based on the dynamic facial keypoints of the person in the driving video and the static facial keypoints of the target person in the target person image; and transferring the dynamic facial features of the person in the driving video to the target person image based on the first keypoint mapping relationship to obtain the dynamic video of the target person.

[0008] According to the embodiments of this application, the dynamic facial features of a person in a driving video can be efficiently and accurately transferred to a target person image, so that the dynamic facial features of the target person in the target person dynamic video are the same as the dynamic facial features of the person in the driving video. This is equivalent to generating a dynamic target person dynamic video using a static target person image.

[0009] According to a first possible implementation of the first aspect, the step of transferring the dynamic facial features of the person in the driving video to the target person image based on the first keypoint mapping relationship to obtain the target person dynamic video includes: transforming the static facial keypoints of the target person in the target person image into first target dynamic facial keypoints based on the first keypoint mapping relationship, wherein the dynamic facial features represented by the first target dynamic facial keypoints are the same as the dynamic facial features of the person in the driving video; and generating the target person dynamic video based on the first target dynamic facial keypoints and the target person image.

[0010] According to the embodiments of this application, by transforming the static facial key points in the target person image into dynamic first target dynamic facial key points based on the first key point mapping relationship, it is equivalent to generating the dynamic facial features of the target person. Then, based on the first target dynamic facial key points and the target person image, a target person dynamic video is generated, which is equivalent to combining the dynamic facial features of the target person with the target person image to generate the target person dynamic video. This can effectively make the dynamic facial features of the target person in the generated target person dynamic video the same as the dynamic facial features of the person in the driving video.

[0011] According to the first aspect, in a second possible implementation of the video generation method, the step of enhancing the dynamic facial features and image quality of the lip-sync video based on the target person image to obtain a target person lip-sync video includes: enhancing the dynamic facial features in the lip-sync video based on the target person image to obtain a facial feature enhancement video with enhanced dynamic facial features; and transferring the dynamic facial features in the facial feature enhancement video to the target person image to obtain a target person lip-sync video with enhanced image quality.

[0012] According to the embodiments of this application, by first enhancing the dynamic facial features in the lip-sync video and then transferring the enhanced dynamic facial features to the target person image, a lip-sync video of the target person with higher image quality and stronger dynamic facial features can be obtained, which means that the image quality of the lip-sync video of the target person is clearer and the lip-sync effect is more realistic.

[0013] According to the second possible implementation of the first aspect, the step of transferring the dynamic facial features in the facial feature enhancement video to the target person image to obtain a target person lip-synced video with enhanced image quality includes: performing keypoint detection on the facial feature enhancement video and the target person image respectively to obtain dynamic facial keypoints in the facial feature enhancement video and static facial keypoints of the target person in the target person image; determining a second keypoint mapping relationship between the facial feature enhancement video and the target person image based on the dynamic facial keypoints in the facial feature enhancement video and the static facial keypoints of the target person in the target person image; and transferring the dynamic facial features in the facial feature enhancement video to the target person image based on the second keypoint mapping relationship to obtain the target person lip-synced video.

[0014] According to the embodiments of this application, by transferring the enhanced dynamic facial features to the target person image, a target person lip-sync video with higher image quality and stronger dynamic facial features is obtained, which means that the image quality of the target person lip-sync video is clearer and the lip-sync effect is more realistic.

[0015] According to the second possible implementation of the first aspect, the step of transferring the dynamic facial features in the facial feature enhancement video to the target person image based on the second keypoint mapping relationship to obtain the target person's lip-sync video includes: transforming the static facial keypoints of the target person in the target person image into second target dynamic facial keypoints based on the second keypoint mapping relationship, wherein the dynamic facial features represented by the second target dynamic facial keypoints are the same as the dynamic facial features in the facial feature enhancement video; and generating the target person's lip-sync video based on the second target dynamic facial keypoints and the target person image.

[0016] According to the embodiments of this application, by transforming the static facial key points in the target person image into dynamic second target dynamic facial key points based on the second key point mapping relationship, it is equivalent to generating dynamic facial features of the target person. Then, based on the second target dynamic facial key points and the target person image, a lip-sync video of the target person is generated. This is equivalent to combining the dynamic facial features of the target person with the target person image to generate a lip-sync video of the target person. This can effectively make the dynamic facial features of the target person in the generated lip-sync video the same as the enhanced dynamic facial features in the facial feature enhancement video and synchronized with the target audio, thereby making the dynamic facial features in the lip-sync video of the target person more realistic.

[0017] According to the first aspect, in a third possible implementation of the video generation method, the method generates a lip-sync video of the target person based on the target person image, the target audio, and the driving video using a pre-trained model; the pre-trained model includes: a person dynamic video generation model, a lip-sync model, and a facial feature enhancement model; wherein, the step of transferring the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video includes: using the person dynamic video generation model to transfer the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video; the step of transferring the dynamic facial features of the person in the driving video to the target person image based on the target audio and the target person dynamic video... The method for generating a lip-sync video includes: generating a lip-sync video using the lip-sync model based on the target audio and the target person's dynamic video; enhancing the dynamic facial features and image quality of the lip-sync video based on the target person's image to obtain a target person's lip-sync video, including: enhancing the dynamic facial features of the lip-sync video using the facial feature enhancement model based on the target person's image to obtain a facial feature enhancement video with enhanced dynamic facial features; and transferring the dynamic facial features from the facial feature enhancement video to the target person's image using the person's dynamic video generation model to obtain a target person's lip-sync video with enhanced image quality.

[0018] According to the embodiments of this application, the image, audio and video of the person can be decoupled from the model. In this way, it is not necessary to collect a large number of customized sample videos of the target person to train a specific model to generate virtual digital human videos. The virtual dynamic image of the target person can be generated, and the transfer and generalization ability is strong. In particular, through the process of enhancing facial dynamic features and enhancing image quality, the lip-sync video of the target person can be made more realistic and the image quality is higher, that is, the clarity and accuracy are higher.

[0019] Secondly, embodiments of this application provide a video generation apparatus, comprising: an acquisition module for acquiring a target person image, target audio set by the target person, and a preset driving video, wherein the face of the person in the driving video is dynamic; a migration module for migrating the dynamic facial features of the person in the driving video to the target person image to obtain a target person dynamic video, wherein the face of the target person in the target person dynamic video is dynamic, and the dynamic facial features include at least one of the following: expression, action, and lip shape; a synchronization module for generating a lip-sync video based on the target audio and the target person dynamic video, wherein the dynamic facial features in the lip-sync video are synchronized with the target audio; and an enhancement module for enhancing the dynamic facial features and image quality in the lip-sync video based on the target person image to obtain a target person lip-sync video.

[0020] According to the second aspect, in a first possible implementation of the video generation apparatus, the step of transferring the dynamic facial features of a person in the driving video to the target person image to obtain a dynamic video of the target person includes: performing keypoint detection on the driving video and the target person image respectively to obtain dynamic facial keypoints of the person in the driving video and static facial keypoints of the target person in the target person image; determining a first keypoint mapping relationship between the driving video and the target person image based on the dynamic facial keypoints of the person in the driving video and the static facial keypoints of the target person in the target person image; and transferring the dynamic facial features of the person in the driving video to the target person image based on the first keypoint mapping relationship to obtain the dynamic video of the target person.

[0021] According to a first possible implementation of the second aspect, the step of transferring the dynamic facial features of the person in the driving video to the target person image based on the first keypoint mapping relationship to obtain the target person dynamic video includes: transforming the static facial keypoints of the target person in the target person image into first target dynamic facial keypoints based on the first keypoint mapping relationship, wherein the dynamic facial features represented by the first target dynamic facial keypoints are the same as the dynamic facial features of the person in the driving video; and generating the target person dynamic video based on the first target dynamic facial keypoints and the target person image.

[0022] According to the second aspect, in a second possible implementation of the video generation device, the step of enhancing the dynamic facial features and image quality of the lip-sync video based on the target person image to obtain a target person lip-sync video includes: enhancing the dynamic facial features in the lip-sync video based on the target person image to obtain a facial feature enhancement video with enhanced dynamic facial features; and transferring the dynamic facial features in the facial feature enhancement video to the target person image to obtain a target person lip-sync video with enhanced image quality.

[0023] According to a second possible implementation of the second aspect, the step of transferring the dynamic facial features in the facial feature enhancement video to the target person image to obtain a target person lip-synced video with enhanced image quality includes: performing keypoint detection on the facial feature enhancement video and the target person image respectively to obtain dynamic facial keypoints in the facial feature enhancement video and static facial keypoints of the target person in the target person image; determining a second keypoint mapping relationship between the facial feature enhancement video and the target person image based on the dynamic facial keypoints in the facial feature enhancement video and the static facial keypoints of the target person in the target person image; and transferring the dynamic facial features in the facial feature enhancement video to the target person image based on the second keypoint mapping relationship to obtain the target person lip-synced video.

[0024] According to a second possible implementation of the second aspect, the step of transferring the dynamic facial features in the facial feature enhancement video to the target person image based on the second keypoint mapping relationship to obtain the target person's lip-sync video includes: transforming the static facial keypoints of the target person in the target person image into second target dynamic facial keypoints based on the second keypoint mapping relationship, wherein the dynamic facial features represented by the second target dynamic facial keypoints are the same as the dynamic facial features in the facial feature enhancement video; and generating the target person's lip-sync video based on the second target dynamic facial keypoints and the target person image.

[0025] According to the second aspect, in a third possible implementation of the video generation device, the device generates a lip-sync video of the target person based on the target person image, the target audio, and the driving video using a pre-trained model; the pre-trained model includes: a person dynamic video generation model, a lip-sync model, and a facial feature enhancement model; wherein, the step of transferring the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video includes: using the person dynamic video generation model to transfer the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video; the step of transferring the dynamic facial features of the person in the driving video to the target person image based on the target audio and the target person dynamic video... The method for generating a lip-sync video includes: generating a lip-sync video using the lip-sync model based on the target audio and the target person's dynamic video; enhancing the dynamic facial features and image quality of the lip-sync video based on the target person's image to obtain a target person's lip-sync video, including: enhancing the dynamic facial features of the lip-sync video using the facial feature enhancement model based on the target person's image to obtain a facial feature enhancement video with enhanced dynamic facial features; and transferring the dynamic facial features from the facial feature enhancement video to the target person's image using the person's dynamic video generation model to obtain a target person's lip-sync video with enhanced image quality.

[0026] Thirdly, embodiments of this application provide a video generation apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement one or more of the video generation methods described in the first aspect or various possible implementations of the first aspect.

[0027] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement one or more of the video generation methods described in the first aspect or various possible implementations of the first aspect.

[0028] Fifthly, embodiments of this application provide a terminal device that can execute one or more of the video generation methods described in the first aspect or various possible implementations of the first aspect.

[0029] Sixthly, embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in an electronic device, a processor in the electronic device executes one or more of the video generation methods described in the first aspect or various possible implementations of the first aspect.

[0030] These and other aspects of this application will become more apparent in the description of the following embodiments(s). Attached Figure Description

[0031] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0032] Figure 1 A schematic diagram illustrating an application scenario according to an embodiment of this application is shown.

[0033] Figure 2 A flowchart illustrating a video generation method according to an embodiment of this application is shown.

[0034] Figure 3 This diagram illustrates a method for generating a dynamic video of a target person according to an embodiment of this application.

[0035] Figure 4 This diagram illustrates a method for generating lip-sync video according to an embodiment of this application.

[0036] Figure 5 This diagram illustrates a method for generating a facial feature enhancement video according to an embodiment of this application.

[0037] Figure 6 This diagram illustrates a method for generating a lip-synced video of a target character according to an embodiment of this application.

[0038] Figure 7 A schematic diagram of a video generation method according to an embodiment of this application is shown.

[0039] Figure 8 A schematic diagram of a video generation system according to an embodiment of this application is shown.

[0040] Figure 9 A structural diagram of a video generation apparatus according to an embodiment of this application is shown.

[0041] Figure 10 A structural diagram of an electronic device 1300 according to an embodiment of this application is shown. Detailed Implementation

[0042] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0043] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0044] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0045] To better understand the solutions of the embodiments of this application, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced below.

[0046] As mentioned above, with the development of artificial intelligence technology, virtual digital human technology that simulates the appearance, speech, and behavior of real people is a prerequisite for the realization of the "metaverse" concept. Since the quality of digital virtual human technology depends on the user experience, high anthropomorphism and high visualization accuracy are basic requirements for virtual digital human technology.

[0047] In related technologies, the AudioDVP model can be used to generate virtual digital human videos. This model is a dedicated virtual human generation model that uses sound to freely drive input portrait videos to synthesize target human videos. To conjure believable and realistic facial expressions from the input audio, this method employs a parameterized 3D face model represented by geometry, facial expressions, and lighting. The face in the original video is parameterized, and the mapping from audio features to model parameters is learned. Then, the input source audio is represented as a high-dimensional feature to predict the facial expression parameters of the 3D face model. The expression parameters calculated from the original video are then replaced with the expression parameters predicted from the audio, and the face is re-rendered. Finally, a neural face renderer generates a realistic portrait video from the re-rendered synthetic face sequence. However, this model requires a large amount of training data, has a long timeframe, high shooting costs, and strict requirements on posture, expression, movement, and background. Furthermore, due to limited training data, the model's generalization ability is poor, resulting in poor lip-sync performance in the generated portrait video.

[0048] Alternatively, the Wav2Lip model can be used. This is a supervised learning-based virtual human generation model for lip-syncing. Such general-purpose virtual human generation models only require a video of the person and a target audio clip to merge the audio and video, achieving good matching between the person's lip movements and the audio. This model primarily uses an additional lip-sync discriminator (such as the LipGAN model) to determine whether the lip movements are synchronized with the audio, thus constraining the lip-sync discriminator, which incorporates contextual information detection, to learn the ability to convert any audio clip into a corresponding lip movement, achieving synchronization between any audio clip and the lip movement. However, this model mainly generates the lip region, resulting in a disharmony between the lips and the overall image, and the generated video quality is low, with a maximum resolution of only 128×128 pixels, limiting its practical value.

[0049] In view of this, this application provides a video generation method. The video generation method of this application embodiment can drive a single target person image with arbitrary audio and arbitrary video to generate a target person's lip-sync video, which is equivalent to generating a virtual digital human video of the target person. It can decouple the person image, audio and video from the model, so that it can generate a virtual dynamic image of the target person without collecting a large amount of customized video datasets to train the model. It has strong transfer and generalization ability. At the same time, by combining the facial dynamic feature enhancement process, the virtual dynamic image in the target person's lip-sync video is more realistic and the picture quality is higher, that is, the clarity and accuracy are higher.

[0050] The video generation method of this application embodiment can be applied to video generation scenarios that generate highly human-like and high-quality virtual digital human videos for users. It can be applied to virtual digital human tool platforms, making it convenient for users to use the tool platform to generate their own virtual avatars or to carry out secondary development based on the platform.

[0051] Figure 1This paper illustrates an application scenario for a video generation method. In this scenario, the video generation system 100 can be directly deployed on a terminal device or server, and can also be applied to a virtual digital human tool platform already deployed on the terminal device or server. When a user wants to generate their own virtual digital human video, they can input a target person image and a target audio clip into the video generation system 100. The target person image can be the user's own real image, and the target audio can be the audio spoken by the person in the virtual digital human video that the user expects. The video generation system 100 can output a lip-synced video of the target person with high image quality based on the target person image and the target audio, combined with the built-in driving video. The dynamic facial features of the target person, such as facial expressions and lip shapes, in the lip-synced video are synchronized with the target audio. Users can directly use the lip-synced video of the target person as their desired virtual digital human video, or they can perform secondary development on the lip-synced video of the target person, such as adjusting the background, filters, style, etc., and use the secondary developed video as their desired virtual digital human video. This application embodiment does not limit this.

[0052] The video generation method of this application can be deployed on various terminal devices through software or hardware modifications. The terminal devices involved in this application can be devices with wireless connectivity, meaning they can connect to other terminal devices via Wi-Fi, Bluetooth, or other wireless methods. The terminal devices of this application can also have wired communication capabilities. The terminal devices of this application can be touchscreen, non-touchscreen, or screenless. Touchscreen devices can be controlled by clicking or swiping on the display screen using fingers or styluses. Non-touchscreen devices can connect to input devices such as mice, keyboards, or touch panels to control the terminal device. Screenless devices can be, for example, screenless Bluetooth speakers. For instance, a user can input a target person's image and target audio into the video generation system 100 by touching corresponding controls on the terminal device. The video generation system 100 deployed on the terminal device can then output a lip-synced video of the target person and display it to the user on the terminal device.

[0053] For example, the terminal device of this application may be a smartphone, netbook, tablet computer, laptop computer, wearable electronic device (such as smart bracelet, smartwatch, etc.), TV, virtual reality device, speaker, e-ink, etc.

[0054] The video generation method of this application embodiment can also be deployed on a server, which can be located in the cloud or locally, and can be a physical device or a virtual device, such as a virtual machine or container, with wireless communication capabilities. These wireless communication capabilities can be configured in the server's chip (system) or other components. This can refer to a device with wireless connectivity, meaning it can connect to other servers or terminal devices via wireless connection methods such as Wi-Fi or Bluetooth. The server in this application can also have wired communication capabilities. For example, the server in this application can be located in the cloud, communicate with terminal devices, receive target person images and target audio sent by the terminal devices, and use the video generation system 100 deployed on the server to generate a lip-synced video of the target person, which is then returned to the terminal device.

[0055] The following is passed Figures 2 to 7 The video generation method provided in the embodiments of this application will be described in detail.

[0056] Figure 2 A flowchart illustrating a video generation method according to an embodiment of this application is shown. This method can be applied to the aforementioned video generation system 100, and can be executed by the aforementioned terminal device or server, such as... Figure 2 As shown, the method includes:

[0057] Step S201: Acquire the target person's image, the target person's set audio, and the preset driving video. The person's face in the driving video is dynamic.

[0058] The target person is a real person, and the image of the target person can be a high-resolution facial image or a high-resolution full-body image of the target person captured by an image capturing device (such as a camera), or it can be a high-resolution facial image or a high-resolution full-body image of the target person downloaded by the user from the network to the terminal device or uploaded from the terminal device to the server. It should be understood that this application embodiment does not limit the image content in the target person image or the method of acquiring the target person image. Optionally, the user-provided target person image can also undergo preprocessing such as adaptive cropping and resizing to meet the requirements of subsequent processing steps for image data.

[0059] The target audio can be audio of any person recorded using an audio recording device, or it can be the audio of the target person, or it can be any audio downloaded by the user from the network to the terminal device or uploaded from the terminal device to the server. This target audio can serve as the driving audio, without needing to consider the actions, expressions, postures, or clothing of any person. It should be understood that this application embodiment does not limit the audio content or the method of acquiring the target audio.

[0060] The driving video can be a high-quality video of any person pre-recorded by a video recording device. The person in the driving video can have specific expressions, actions, postures, and clothing, but the person's face must be dynamic so that the face of the target person in the dynamic video of the target person generated using the driving video is also dynamic. It should be understood that the embodiments of this application do not limit the video content in the driving video or the method of acquiring the driving video.

[0061] Step S202: Transfer the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video. The face of the target person in the target person dynamic video is dynamic, and the dynamic facial features include at least one of the following: expression, action, and lip shape.

[0062] It should be understood that when a person is speaking, their facial expressions, movements, and lip shapes can change dynamically. Among these, dynamic facial features can be understood as dynamic facial characteristics, namely dynamic expressions, movements (such as swaying from side to side), and dynamic lip shapes (such as the shape of a person's lips when speaking).

[0063] In one possible implementation, the dynamic facial features of a person in a driving video are transferred to a target person image to obtain a dynamic video of the target person. This includes: performing keypoint detection on both the driving video and the target person image to obtain dynamic facial keypoints of the person in the driving video and static facial keypoints of the target person in the target person image; determining a first keypoint mapping relationship between the driving video and the target person image based on the dynamic facial keypoints of the person in the driving video and the static facial keypoints of the target person in the target person image; and transferring the dynamic facial features of the person in the driving video to the target person image based on the first keypoint mapping relationship to obtain the dynamic video of the target person. This method can efficiently and accurately transfer the dynamic facial features of the person in the driving video to the target person image, ensuring that the dynamic facial features of the target person in the obtained dynamic video are identical to those in the driving video. Essentially, it is equivalent to generating a dynamic video of the target person using a static image of the target person.

[0064] This application employs known keypoint detection techniques to detect keypoints in both the driving video and the target person image, obtaining dynamic facial keypoints in the driving video and static facial keypoints in the target person image. This is not limited in the specific implementation of this application. It should be understood that after obtaining the dynamic facial keypoints in the driving video and the static facial keypoints in the target person image, the dynamic and static facial keypoints can be matched to obtain paired static and dynamic facial keypoints belonging to the same location (e.g., both being keypoints at the left corner of the mouth). Based on these paired static and dynamic facial keypoints, a first keypoint mapping relationship can be determined between the driving video and the target person image. Then, based on this first keypoint mapping relationship, the dynamic facial features of the person in the driving video are transferred to the target person image to obtain a dynamic video of the target person. This is equivalent to transforming the target person image into a dynamic video of the target person with dynamic facial features consistent with the driving video based on the first keypoint mapping relationship.

[0065] Optionally, based on the first keypoint mapping relationship, the dynamic facial features of the person in the driving video are transferred to the target person image to obtain a dynamic video of the target person. This includes: transforming the static facial keypoints of the target person in the target person image into first target dynamic facial keypoints based on the first keypoint mapping relationship, wherein the dynamic facial features represented by the first target dynamic facial keypoints are the same as the dynamic facial features of the person in the driving video; and generating a dynamic video of the target person based on the first target dynamic facial keypoints and the target person image. Specifically, transforming the static facial keypoints in the target person image into dynamic first target dynamic facial keypoints based on the first keypoint mapping relationship is equivalent to generating the dynamic facial features of the target person. Then, generating a dynamic video of the target person based on the first target dynamic facial keypoints and the target person image is equivalent to combining the dynamic facial features of the target person with the target person image to generate the dynamic video of the target person. This method effectively ensures that the dynamic facial features of the target person in the generated dynamic video of the target person are the same as the dynamic facial features of the person in the driving video.

[0066] In practical applications, a pre-trained human dynamic video generation model (such as a motion capture and video generation model) can be used to transfer the dynamic facial features of a person in a driving video to a target person image, thus obtaining a dynamic video of the target person. Specifically, this human dynamic video generation model can perform keypoint detection on both the driving video and the target person image to obtain the dynamic facial keypoints of the person in the driving video and the static facial keypoints of the target person in the target person image, and determine the first keypoint mapping relationship between the driving video and the target person image. Then, based on this first mapping relationship, the dynamic facial features such as expressions, movements, and lip shapes of the person in the driving video can be transferred to the target person image to obtain a dynamic video of the target person. Optionally, the human dynamic video generation model can also use occlusion masks to block out uninteresting image regions (such as background regions) in the target person image, so that the dynamic video of the target person mainly retains the person area.

[0067] Figure 3 This diagram illustrates a method for generating a dynamic video of a target person according to an embodiment of this application, such as... Figure 3 As shown, the driving video and the target person image are input into the person dynamic video generation model, and the output is the target person dynamic video. The person dynamic video generation model includes a keypoint detection module, an action generation module, and a video generation module. The keypoint detection module performs keypoint detection on both the driving video and the target person image to obtain the dynamic facial keypoints of the person in the driving video and the static facial keypoints of the target person in the target person image. Based on these dynamic and static facial keypoints, a first keypoint mapping relationship is determined between the driving video and the target person image. The action generation module transforms the static facial keypoints of the target person in the target person image into first target dynamic facial keypoints based on the first keypoint mapping relationship. The video generation module generates the target person dynamic video based on the first target dynamic facial keypoints and the target person image.

[0068] It should be understood that the above-mentioned character dynamic video generation model can adopt deep learning models or machine learning models known in the art, and those skilled in the art can also independently develop the above-mentioned character dynamic video generation model; and those skilled in the art can use model training methods known in the art to train the above-mentioned character dynamic video generation model. The embodiments of this application do not limit the type, structure and training method of the character dynamic video generation model.

[0069] Step S203: Generate a lip-sync video based on the target audio and the target person's dynamic video. The dynamic facial features in the lip-sync video are synchronized with the target audio.

[0070] In practical applications, a pre-trained lip-sync model can be used to generate lip-sync videos based on the target audio and the target person's dynamic video. Figure 4 This diagram illustrates a method for generating lip-sync video according to an embodiment of this application, as shown below. Figure 4 As shown, the target audio and the target person's dynamic video are input into the lip-sync model, and the lip-sync video is output.

[0071] The lip-sync model can employ any known audio-driven face generation model with high-frequency lip-sync. In practical applications, the lip-sync video generated by this model can be low-resolution; that is, it is not necessary to generate high-resolution lip-sync video, only to ensure the high-frequency lip-sync effect. The dynamic facial features in the lip-sync video are synchronized with the target audio. This can be understood as the dynamic facial features of the target person, such as expressions, movements, and lip shapes, matching the target audio.

[0072] It should be understood that the lip-sync model can adopt deep learning models or machine learning models known in the art, and those skilled in the art can also independently develop the above-mentioned lip-sync model; and those skilled in the art can use model training methods known in the art to train the above-mentioned lip-sync model. The embodiments of this application do not limit the type, structure and training method of the lip-sync model.

[0073] Step S204: Based on the target person image, enhance the dynamic facial features and image quality in the lip-sync video to obtain the lip-sync video of the target person.

[0074] Enhancing the dynamic facial features in lip-sync videos can be understood as enhancing the dynamic facial features of the target person, such as facial expressions, movements, and lip shapes, thereby making the lip-syncing effect of key facial features in the video more realistic. As mentioned above, lip-sync videos may be low-resolution; therefore, enhancing the image quality of the lip-sync video is equivalent to enhancing its clarity, thus making the lip-sync video of the target person appear higher definition.

[0075] In one possible implementation, based on the target person's image, the dynamic facial features and image quality of the lip-sync video are enhanced to obtain the target person's lip-sync video. This includes: enhancing the dynamic facial features in the lip-sync video based on the target person's image to obtain a facial feature-enhanced video; and transferring the dynamic facial features from the facial feature-enhanced video to the target person's image to obtain a target person's lip-sync video with enhanced image quality. This method can be understood as first enhancing the dynamic facial features in the lip-sync video, and then transferring the enhanced dynamic facial features to the target person's image, thereby obtaining a target person's lip-sync video with higher image quality and stronger dynamic facial features, resulting in a clearer image quality and more realistic lip-sync effect.

[0076] In practical applications, a pre-trained facial feature enhancement model can be used to enhance the dynamic facial features in a lip-sync video based on the target person's image, resulting in a facial feature enhancement video with dynamic facial feature enhancement. Figure 5 This illustration shows a schematic diagram of generating a facial feature enhancement video according to an embodiment of this application, such as... Figure 5 As shown, the target person image and the lip-sync video are input into the facial feature enhancement model, and the facial feature enhancement video is output.

[0077] The facial feature enhancement model can be any known model in the art capable of enhancing image features by referencing the original high-definition image features, as long as it can enhance the dynamic facial features in a lip-synced video based on the target person's image. It should be understood that the facial feature enhancement model can employ known deep learning models or machine learning models, etc., and those skilled in the art can also independently develop the aforementioned facial feature enhancement model; furthermore, those skilled in the art can train the aforementioned facial feature enhancement model using known model training methods. This application does not limit the type, structure, or training method of the facial feature enhancement model.

[0078] In one possible implementation, dynamic facial features from a facial feature enhancement video are transferred to a target person image to obtain a target person lip-sync video with enhanced image quality. This includes: performing keypoint detection on both the facial feature enhancement video and the target person image to obtain dynamic facial keypoints in the facial feature enhancement video and static facial keypoints of the target person in the target person image; determining a second keypoint mapping relationship between the facial feature enhancement video and the target person image based on the dynamic facial keypoints in the facial feature enhancement video and the static facial keypoints of the target person in the target person image; and transferring the dynamic facial features from the facial feature enhancement video to the target person image based on the second keypoint mapping relationship to obtain the target person lip-sync video. This method, by transferring the enhanced dynamic facial features to the target person image, yields a target person lip-sync video with higher image quality and stronger dynamic facial features, resulting in a clearer image quality and more realistic lip-sync effect.

[0079] Among them, key point detection technology known in the art can be used to perform key point detection on the facial feature enhancement video and the target person image respectively, to obtain the dynamic facial key points in the facial feature enhancement video and the static facial key points of the target person in the target person image. This application embodiment does not limit this. It should be understood that after obtaining the dynamic facial key points of the person in the facial feature enhancement video and the static facial key points of the target person in the target person image, the dynamic facial key points can be matched with the static facial key points to obtain paired static and dynamic facial key points belonging to the same location (e.g., both are key points at the left corner of the mouth). Then, based on the paired static and dynamic facial key points, a second key point mapping relationship can be determined between the facial feature enhancement video and the target person image. Based on this second key point mapping relationship, the dynamic facial features of the person in the driving video are transferred to the target person image to obtain the target person dynamic video. This is equivalent to transforming the target person image into a target person lip-sync video with dynamic facial features consistent with the facial feature enhancement video based on the second key point mapping relationship. Since the dynamic facial features in the facial feature enhancement video are synchronized with the target audio, the dynamic facial features in the target lip-sync video obtained using the dynamic facial features in the facial feature enhancement video are also synchronized with the target audio.

[0080] Optionally, based on the second keypoint mapping relationship, the dynamic facial features in the facial feature enhancement video are transferred to the target person image to obtain the target person's lip-sync video. This includes: transforming the static facial keypoints of the target person in the target person image into second target dynamic facial keypoints based on the second keypoint mapping relationship. The dynamic facial features represented by the second target dynamic facial keypoints are the same as the dynamic facial features in the facial feature enhancement video. Then, based on the second target dynamic facial keypoints and the target person image, the target person's lip-sync video is generated. Specifically, transforming the static facial keypoints in the target person image into dynamic second target dynamic facial keypoints based on the second keypoint mapping relationship is equivalent to generating the target person's dynamic facial features. Then, generating the target person's lip-sync video based on the second target dynamic facial keypoints and the target person image is equivalent to combining the target person's dynamic facial features with the target person image to generate the target person's lip-sync video. This method effectively makes the dynamic facial features of the target person in the generated lip-sync video identical to the enhanced dynamic facial features in the facial feature enhancement video and synchronized with the target audio, thus making the dynamic facial features in the lip-sync video more realistic.

[0081] As described above, a pre-trained character dynamic video generation model (e.g., a motion capture and video generation model) can be used to transfer the dynamic facial features of the person in the driving video to the target person image, resulting in a target person dynamic video. Therefore, the aforementioned character dynamic video generation model can be further used to transfer the dynamic facial features in the facial feature enhancement video to the target person image, resulting in a target person lip-synced video with enhanced image quality. Specifically, the character dynamic video generation model can perform keypoint detection on both the facial feature enhancement video and the target person image, obtaining the dynamic facial keypoints of the person in the facial feature enhancement video and the static facial keypoints of the target person in the target person image, and determining a second keypoint mapping relationship between the facial feature enhancement video and the target person image. Based on this second mapping relationship, the dynamic facial features such as expressions, movements, and lip shapes of the person in the facial feature enhancement video can then be transferred to the target person image, resulting in a target person lip-synced video. Optionally, the character dynamic video generation model can also use occlusion masks to mask uninteresting image regions (such as background regions) in the target person image, thereby preserving mainly the person region in the target person lip-synced video.

[0082] Figure 6 This diagram illustrates a method for generating a lip-sync video of a target character according to an embodiment of this application. Figure 6As shown, a facial feature enhancement video and a target person image are input into a character dynamic video generation model, which outputs a lip-synced video of the target person. The character dynamic video generation model includes a keypoint detection module, an action generation module, and a video generation module. The keypoint detection module performs keypoint detection on both the facial feature enhancement video and the target person image, obtaining dynamic facial keypoints in the facial feature enhancement video and static facial keypoints of the target person in the target person image. Based on these dynamic and static facial keypoints, a second keypoint mapping relationship is determined between the facial feature enhancement video and the target person image. The action generation module transforms the static facial keypoints of the target person in the target person image into second target dynamic facial keypoints based on the second keypoint mapping relationship. The video generation module generates a target person dynamic video based on the second target dynamic facial keypoints and the target person image.

[0083] As described above, the aforementioned character dynamic video generation model can employ deep learning models or machine learning models known in the art. Of course, those skilled in the art can also independently develop the aforementioned character dynamic video generation model. Furthermore, those skilled in the art can train the aforementioned character dynamic video generation model using model training methods known in the art. The embodiments of this application do not limit the type, structure, or training method of the character dynamic video generation model.

[0084] According to the embodiments of this application, a single target person image can be driven by any audio and any video to generate a target person's lip-sync video. That is, by using a single target person image, target audio, and driving video, dynamic facial feature transfer, lip-sync, feature enhancement, and image quality enhancement are performed sequentially to generate a target person's lip-sync video with high image quality and high lip-sync, which is equivalent to generating a virtual digital human video of the target person. It can decouple the person image, audio, and video from the model, so that there is no need to collect a large number of customized sample videos of the target person to train a specific model to generate a virtual digital human video. It can generate a virtual dynamic image of the target person. The transfer and generalization ability is strong. In particular, through the process of facial dynamic feature enhancement and image quality enhancement, the virtual dynamic image in the target person's lip-sync video can be made more realistic and the image quality is higher, that is, the clarity and accuracy are higher.

[0085] Based on the above Figures 3 to 6 The processing procedure shown in this application embodiment also provides Figure 7 One video generation method is shown, such as Figure 7 As shown, this video generation method generates a lip-sync video of a target person based on the target person image, target audio, and driving video using a pre-trained model. The pre-trained model includes: a person dynamic video generation model, a lip-sync model, and a facial feature enhancement model.

[0086] The video generation method described above utilizes a person dynamic video generation model to transfer the dynamic facial features of the person in the driving video to the target person image, thereby obtaining a target person dynamic video; it uses a lip-sync model to generate a lip-sync video based on the target audio and the target person dynamic video; it uses a facial feature enhancement model to enhance the dynamic facial features in the lip-sync video based on the target person image, thereby obtaining a facial feature enhanced video; and it uses a person dynamic video generation model to transfer the dynamic facial features in the facial feature enhanced video to the target person image, thereby obtaining a target person lip-sync video with enhanced image quality.

[0087] Specifically, Figure 7 The video generation method shown includes: adaptively cropping and resizing the original target person image to obtain the target person image; inputting the driving video and the target person image into a person motion video generation model to output a target person motion video; inputting the target audio and the target person motion video into a lip-sync model to output a lip-sync video; inputting the target person image and the lip-sync video into a facial feature enhancement model to output a facial feature enhanced video; and inputting the facial feature enhanced video and the target person image into a person motion video generation model to output a target person lip-sync video. It should be understood that achieving... Figure 7 The various models mentioned above can be referenced. Figures 3 to 6 The specific implementation method shown will not be elaborated here.

[0088] According to the embodiments of this application, the image, audio and video of the person can be decoupled from the model. In this way, it is not necessary to collect a large number of customized sample videos of the target person to train a specific model to generate virtual digital human videos. The virtual dynamic image of the target person can be generated, and the transfer and generalization ability is strong. In particular, through the process of facial dynamic feature enhancement and image quality enhancement, the virtual dynamic image in the lip-sync video of the target person can be more realistic and the image quality is higher, that is, the clarity and accuracy are higher.

[0089] Figure 8 This diagram illustrates a video generation system according to an embodiment of the present application, such as... Figure 8 As shown, the video generation system includes:

[0090] Image capture module, used to capture a single high-quality image of the target person;

[0091] The video recording module is used to record a small number of high-quality videos of any person, which serve as the driving video for the target person's dynamic video generation module.

[0092] The audio recording module is used to record the target audio of any person as the driving audio, without needing to consider actions, expressions, posture, clothing, etc.

[0093] The target person dynamic video generation module I is used to generate a target person dynamic video based on the input target person image and the built-in driving video.

[0094] The lip-sync module is used to generate low-resolution lip-sync videos based on the target audio and the target person's dynamic video.

[0095] The facial feature enhancement module is used to generate a dynamic facial feature enhancement video based on low-resolution lip-sync video and the target person's image.

[0096] The Target Person Dynamic Video Generation Module II is used to generate a lip-sync video of the target person based on the input target person image and facial feature enhancement video.

[0097] The system comprises an image capture module, a video recording module, an audio recording module, a target person dynamic video generation module I, and a lip-syncing module, which together achieve lip-syncing video generation decoupled from the model. The facial feature enhancement module and the target person dynamic video generation module II together achieve high-definition lip-syncing video generation of the target person based on dynamic facial feature enhancement and image quality enhancement, i.e., generating a virtual digital human video of the target person. It should be understood that the functions of each module in the above video generation system are similar to those described above. Figures 2 to 6 The specific implementation method shown will not be elaborated here.

[0098] According to the embodiments of this application, it is possible to efficiently generate a lip-sync video of a target person, that is, to generate a virtual dynamic image of the target person. It has strong transfer and generalization capabilities. In particular, through the process of enhancing facial dynamic features and image quality, the lip-sync video of the target person can be made more realistic and have higher image quality, that is, higher clarity and accuracy.

[0099] According to the video generation method and system described in the embodiments of this application, only a single image of the target person needs to be input to generate a dynamic video of the target person. Combined with the target audio input, the dynamic video of the target person can be driven to generate a lip-sync video with synchronized facial expressions and lip movements based on the target audio. In this case, by decoupling the video from the model, the generalization ability of the model to generate lip-sync videos of different target persons can be improved. For the generated lip-sync videos, a model based on dynamic facial feature enhancement is constructed to enhance the features and image quality of the lip-sync videos. This not only improves the clarity of the lip-sync videos, but also provides additional enhancement to the most critical dynamic facial features, making the dynamic facial features of the target person in the lip-sync model higher definition, easier to recognize, and more realistic.

[0100] Figure 9 This diagram illustrates a structural diagram of a video generation apparatus according to an embodiment of the present application. This apparatus can be used in the aforementioned video generation system 100, such as… Figure 9 As shown, the device includes:

[0101] The acquisition module 901 is used to acquire the target person's image, the target audio set by the target person, and the preset driving video, wherein the person's face in the driving video is dynamic.

[0102] The migration module 902 is used to migrate the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video. The face of the target person in the target person dynamic video is dynamic, and the dynamic facial features include at least one of the following: expression, action, and lip shape.

[0103] The synchronization module 903 is used to generate a lip-sync video based on the target audio and the target person's dynamic video, wherein the dynamic facial features in the lip-sync video are synchronized with the target audio.

[0104] The enhancement module 904 is used to enhance the dynamic facial features and image quality of the lip-sync video based on the target person image, so as to obtain the lip-sync video of the target person.

[0105] According to the embodiments of this application, a single target person image can be driven by any audio and any video to generate a target person's lip-sync video. That is, by using a single target person image, target audio, and driving video, dynamic facial feature transfer, lip-sync, feature enhancement, and image quality enhancement are performed sequentially to generate a target person's lip-sync video with high image quality and high lip-sync, which is equivalent to generating a virtual digital human video of the target person. It can decouple the person image, audio, and video from the model, so that there is no need to collect a large number of customized sample videos of the target person to train a specific model to generate a virtual digital human video. It can generate a virtual dynamic image of the target person. The transfer and generalization ability is strong. In particular, through the process of facial dynamic feature enhancement and image quality enhancement, the virtual dynamic image in the target person's lip-sync video can be made more realistic and the image quality is higher, that is, the clarity and accuracy are higher.

[0106] In one possible implementation, the dynamic facial features of a person in the driving video are transferred to the target person image to obtain a dynamic video of the target person. This includes: performing keypoint detection on both the driving video and the target person image to obtain dynamic facial keypoints of the person in the driving video and static facial keypoints of the target person in the target person image; determining a first keypoint mapping relationship between the driving video and the target person image based on the dynamic facial keypoints of the person in the driving video and the static facial keypoints of the target person in the target person image; and transferring the dynamic facial features of the person in the driving video to the target person image based on the first keypoint mapping relationship to obtain a dynamic video of the target person.

[0107] According to the embodiments of this application, the dynamic facial features of a person in a driving video can be efficiently and accurately transferred to a target person image, so that the dynamic facial features of the target person in the target person dynamic video are the same as the dynamic facial features of the person in the driving video. This is equivalent to generating a dynamic target person dynamic video using a static target person image.

[0108] In one possible implementation, based on the first keypoint mapping relationship, the dynamic facial features of the person in the driving video are transferred to the target person image to obtain the target person dynamic video, including: based on the first keypoint mapping relationship, transforming the static facial keypoints of the target person in the target person image into first target dynamic facial keypoints, wherein the dynamic facial features represented by the first target dynamic facial keypoints are the same as the dynamic facial features of the person in the driving video; and generating the target person dynamic video based on the first target dynamic facial keypoints and the target person image.

[0109] According to the embodiments of this application, by transforming the static facial key points in the target person image into dynamic first target dynamic facial key points based on the first key point mapping relationship, it is equivalent to generating the dynamic facial features of the target person. Then, based on the first target dynamic facial key points and the target person image, a target person dynamic video is generated, which is equivalent to combining the dynamic facial features of the target person with the target person image to generate the target person dynamic video. This can effectively make the dynamic facial features of the target person in the generated target person dynamic video the same as the dynamic facial features of the person in the driving video.

[0110] In one possible implementation, based on the image of the target person, the dynamic facial features and image quality of the lip-sync video are enhanced to obtain the lip-sync video of the target person, including: enhancing the dynamic facial features of the lip-sync video based on the image of the target person to obtain a facial feature enhancement video with enhanced dynamic facial features; and transferring the dynamic facial features in the facial feature enhancement video to the image of the target person to obtain a lip-sync video of the target person with enhanced image quality.

[0111] According to the embodiments of this application, by first enhancing the dynamic facial features in the lip-sync video and then transferring the enhanced dynamic facial features to the target person image, a lip-sync video of the target person with higher image quality and stronger dynamic facial features can be obtained, which means that the image quality of the lip-sync video of the target person is clearer and the lip-sync effect is more realistic.

[0112] In one possible implementation, dynamic facial features from a facial feature enhancement video are transferred to a target person image to obtain a target person lip-synced video with enhanced image quality. This includes: performing keypoint detection on both the facial feature enhancement video and the target person image to obtain dynamic facial keypoints in the facial feature enhancement video and static facial keypoints of the target person in the target person image; determining a second keypoint mapping relationship between the facial feature enhancement video and the target person image based on the dynamic facial keypoints in the facial feature enhancement video and the static facial keypoints of the target person in the target person image; and transferring the dynamic facial features from the facial feature enhancement video to the target person image based on the second keypoint mapping relationship to obtain the target person lip-synced video.

[0113] According to the embodiments of this application, by transferring the enhanced dynamic facial features to the target person image, a target person lip-sync video with higher image quality and stronger dynamic facial features is obtained, which means that the image quality of the target person lip-sync video is clearer and the lip-sync effect is more realistic.

[0114] In one possible implementation, based on the second keypoint mapping relationship, dynamic facial features in the facial feature enhancement video are transferred to the target person image to obtain a lip-sync video of the target person. This includes: transforming the static facial keypoints of the target person in the target person image into second target dynamic facial keypoints based on the second keypoint mapping relationship, wherein the dynamic facial features represented by the second target dynamic facial keypoints are the same as the dynamic facial features in the facial feature enhancement video; and generating a lip-sync video of the target person based on the second target dynamic facial keypoints and the target person image.

[0115] According to the embodiments of this application, by transforming the static facial key points in the target person image into dynamic second target dynamic facial key points based on the second key point mapping relationship, it is equivalent to generating dynamic facial features of the target person. Then, based on the second target dynamic facial key points and the target person image, a lip-sync video of the target person is generated. This is equivalent to combining the dynamic facial features of the target person with the target person image to generate a lip-sync video of the target person. This can effectively make the dynamic facial features of the target person in the generated lip-sync video the same as the enhanced dynamic facial features in the facial feature enhancement video and synchronized with the target audio, thereby making the dynamic facial features in the lip-sync video of the target person more realistic.

[0116] In one possible implementation, the method generates a lip-sync video of the target person based on the target person image, the target audio, and the driving video using a pre-trained model. The pre-trained model includes a person dynamic video generation model, a lip-sync model, and a facial feature enhancement model. The step of transferring the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video includes: using the person dynamic video generation model to transfer the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video. The step of generating a lip-sync video based on the target audio and the target person dynamic video... The process includes: generating a lip-sync video using the lip-sync model based on the target audio and the target person's dynamic video; enhancing the dynamic facial features and image quality of the lip-sync video based on the target person's image to obtain a target person's lip-sync video, which includes: enhancing the dynamic facial features of the lip-sync video based on the target person's image using the facial feature enhancement model to obtain a facial feature enhancement video with enhanced dynamic facial features; and transferring the dynamic facial features from the facial feature enhancement video to the target person's image using the person's dynamic video generation model to obtain a target person's lip-sync video with enhanced image quality.

[0117] According to the embodiments of this application, the image, audio and video of the person can be decoupled from the model. In this way, it is not necessary to collect a large number of customized sample videos of the target person to train a specific model to generate virtual digital human videos. The virtual dynamic image of the target person can be generated, and the transfer and generalization ability is strong. In particular, through the process of enhancing facial dynamic features and enhancing image quality, the lip-sync video of the target person can be made more realistic and the image quality is higher, that is, the clarity and accuracy are higher.

[0118] Embodiments of this application provide a video generation apparatus, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions.

[0119] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0120] Embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0121] Figure 10 A structural diagram of an electronic device 1300 according to an embodiment of this application is shown. Figure 10 As shown, the electronic device 1300 can be a server or a terminal device, performing the above-mentioned functions. Figure 2 The electronic device 1300 includes at least one processor 1801, at least one memory 1802, and at least one communication interface 1803. Furthermore, the electronic device may also include general-purpose components such as antennas, which will not be detailed here.

[0122] The following is combined Figure 10 A detailed introduction to each component of the electronic device 1300 is provided.

[0123] Processor 1801 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of programs in the above scheme. Processor 1801 may include one or more processing units, such as: processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0124] The communication interface 1803 is used to communicate with other electronic devices or communication networks, such as Ethernet, Radio Access Network (RAN), core network, Wireless Local Area Networks (WLAN), etc.

[0125] The memory 1802 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0126] The memory 1802 stores the application code for executing the above scheme, and its execution is controlled by the processor 1801. The processor 1801 executes the application code stored in the memory 1802.

[0127] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0128] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.

[0129] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0130] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0131] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0135] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0136] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0137] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A video generation method, characterized in that, include: The system acquires an image of the target person, the target audio set by the target person, and a preset driving video, wherein the person's face in the driving video is dynamic. The dynamic facial features of the person in the driving video are transferred to the target person image to obtain the target person dynamic video. The face of the target person in the target person dynamic video is dynamic, and the dynamic facial features include at least one of the following: expression, action, and lip shape. Based on the target audio and the target person's dynamic video, a lip-sync video is generated, wherein the dynamic facial features in the lip-sync video are synchronized with the target audio. Based on the target person image, the dynamic facial features and image quality in the lip-sync video are enhanced to obtain the lip-sync video of the target person. The step of enhancing the dynamic facial features and image quality of the lip-sync video based on the target person image to obtain the lip-sync video of the target person includes: Based on the target person image, the dynamic facial features in the lip-sync video are enhanced to obtain a facial feature enhancement video with enhanced dynamic facial features. The dynamic facial features in the enhanced facial feature video are transferred to the target person image to obtain a target person lip-sync video with enhanced image quality.

2. The method according to claim 1, characterized in that, The step of transferring the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video includes: Key point detection is performed on the driving video and the target person image respectively to obtain the dynamic facial key points of the person in the driving video and the static facial key points of the target person in the target person image. Based on the dynamic facial key points of the person in the driving video and the static facial key points of the target person in the target person image, a first key point mapping relationship between the driving video and the target person image is determined. Based on the first key point mapping relationship, the dynamic facial features of the person in the driving video are transferred to the target person image to obtain the target person dynamic video.

3. The method according to claim 2, characterized in that, The step of transferring the dynamic facial features of the person in the driving video to the target person image based on the first key point mapping relationship to obtain the target person dynamic video includes: Based on the first key point mapping relationship, the static facial key points of the target person in the target person image are transformed into the first target dynamic facial key points. The dynamic facial features represented by the first target dynamic facial key points are the same as the dynamic facial features of the person in the driving video. Based on the key points of the first target's dynamic face and the image of the target person, a dynamic video of the target person is generated.

4. The method according to claim 1, characterized in that, The step of transferring dynamic facial features from the enhanced facial feature video to the target person image to obtain a target person lip-synced video with enhanced image quality includes: Key point detection is performed on the facial feature enhancement video and the target person image respectively to obtain the dynamic facial key points in the facial feature enhancement video and the static facial key points of the target person in the target person image. Based on the dynamic facial key points in the facial feature enhancement video and the static facial key points of the target person in the target person image, a second key point mapping relationship between the facial feature enhancement video and the target person image is determined. Based on the second key point mapping relationship, the dynamic facial features in the facial feature enhancement video are transferred to the target person image to obtain the target person's lip-sync video.

5. The method according to claim 4, characterized in that, The step of transferring dynamic facial features from the facial feature enhancement video to the target person image based on the second keypoint mapping relationship to obtain the lip-sync video of the target person includes: Based on the second key point mapping relationship, the static facial key points of the target person in the target person image are transformed into the second target dynamic facial key points. The dynamic facial features represented by the second target dynamic facial key points are the same as the dynamic facial features in the facial feature enhancement video. Based on the second target dynamic facial key points and the target person image, a lip-sync video of the target person is generated.

6. The method according to any one of claims 1 to 5, characterized in that, The method generates a lip-sync video of the target person based on the target person image, the target audio, and the driving video using a pre-trained model; the pre-trained model includes: a person dynamic video generation model, a lip-sync model, and a facial feature enhancement model; The step of transferring the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video includes: using the person dynamic video generation model to transfer the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video. The step of generating a lip-sync video based on the target audio and the target person's dynamic video includes: using the lip-sync model to generate a lip-sync video based on the target audio and the target person's dynamic video; The step of enhancing the dynamic facial features and image quality of the lip-sync video based on the target person image to obtain a lip-sync video of the target person includes: using the facial feature enhancement model to enhance the dynamic facial features in the lip-sync video based on the target person image to obtain a facial feature enhancement video with enhanced dynamic facial features; and using the person dynamic video generation model to transfer the dynamic facial features in the facial feature enhancement video to the target person image to obtain a lip-sync video of the target person with enhanced image quality.

7. A video generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire the target person's image, the target audio set by the target person, and the preset driving video, wherein the person's face in the driving video is dynamic. The migration module is used to migrate the dynamic facial features of the person in the driving video to the target person image to obtain the target person dynamic video. The face of the target person in the target person dynamic video is dynamic, and the dynamic facial features include at least one of the following: expression, action, and lip shape. A synchronization module is used to generate a lip-sync video based on the target audio and the target person's dynamic video, wherein the dynamic facial features in the lip-sync video are synchronized with the target audio. An enhancement module is used to enhance the dynamic facial features and image quality of the lip-sync video based on the target person image, so as to obtain the lip-sync video of the target person. The step of enhancing the dynamic facial features and image quality of the lip-sync video based on the target person image to obtain the lip-sync video of the target person includes: Based on the target person image, the dynamic facial features in the lip-sync video are enhanced to obtain a facial feature enhancement video with enhanced dynamic facial features. The dynamic facial features in the enhanced facial feature video are transferred to the target person image to obtain a target person lip-sync video with enhanced image quality.

8. A video generation apparatus, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 6 when executing the instructions.

9. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

10. A computer program product comprising computer-readable code, wherein when the computer-readable code is run in an electronic device, a processor in the electronic device performs the method of any one of claims 1 to 6.

Images