Video generation method, apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to a video generation method, device, equipment and medium, wherein the method comprises: obtaining a first video and target audio; the first video is a video containing a target face; determining target key points in a target frame image to be generated corresponding to an original frame image in the first video based on the target audio; generating a first image and a second image corresponding to the original frame image based on the first video and the target key points; the first image carries information of a complete face region of the target face, and the second image carries information of an occlusion corresponding to the target face; generating a target frame image corresponding to the original frame image based on the first image and the second image corresponding to the original frame image; and arranging the target frame image corresponding to the original frame image in sequence to obtain a target video. The embodiments of the present disclosure can effectively improve the problems such as serious occlusion deformation and degraded face generation quality caused by the influence of the occlusion in the related art.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video generation method, apparatus, device, and medium. Background Technology

[0002] In scenarios such as multimedia editing, it may be necessary to replace the original audio in a video of someone speaking. The new audio used to replace the original audio can be translated into a specific language or audio unrelated to the content of the original audio. To ensure that the lip movements in the video match the new audio, creating a realistic effect where the person speaking in the video is the content of the new audio, the face of the person in the original video is usually regenerated based on the new audio, resulting in a new video with matching lip movements. However, the inventors discovered that existing video generation methods are ineffective and prone to problems such as degraded facial generation quality, thus affecting the user experience. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a video generation method, apparatus, device, and medium.

[0004] This disclosure provides a video generation method, the method comprising: acquiring a first video and target audio; wherein the first video is a video containing a target face; determining target key points in a target frame image to be generated corresponding to an original frame image in the first video based on the target audio; wherein the target key points are key points of the target face in the target frame image to be generated; generating a first image and a second image corresponding to the original frame image based on the first video and the target key points; wherein the first image carries information about the complete facial region of the target face, and the second image carries information about occlusions corresponding to the target face; generating a target frame image corresponding to the original frame image based on the first image and the second image corresponding to the original frame image; and arranging the target frame images corresponding to the original frame image in sequence to obtain a target video.

[0005] Optionally, generating a first image and a second image corresponding to the original frame image based on the first video and the target key points includes: selecting multiple reference frame images from the first video and obtaining reference key points of the target face in the reference frame images; wherein the target face in the reference frame images is completely unobstructed; performing fusion processing on the multiple reference frame images based on the target key points and the reference key points to obtain the first image; and generating a second image corresponding to the original frame image based on the target key points and the original frame image in the first video.

[0006] Optionally, selecting multiple reference frame images from the first video includes: performing occlusion detection processing on the first video to obtain occlusion detection results corresponding to the original frame images in the first video; selecting original frame images where the target face is completely unoccluded as candidate frame images based on the occlusion detection results corresponding to the original frame images; and selecting multiple reference frame images with different lip shapes from the candidate frame images based on the lip shape information of the target face in the candidate frame images.

[0007] Optionally, the step of fusing the multiple reference frame images based on the target key points and the reference key points to obtain a first image includes: obtaining deformation information and weights corresponding to each of the multiple reference frame images based on the target key points and the reference key points; wherein, the deformation information of the reference frame images is used to indicate the mapping relationship between the target face in the reference frame image and the target face in the target frame image corresponding to the reference frame image; performing grid sampling processing on the reference frame images based on the deformation information corresponding to the reference frame images to obtain the deformation images corresponding to the reference frame images; and performing weighted fusion of the deformation images corresponding to each of the multiple reference frame images based on their respective weights to obtain the first image.

[0008] Optionally, generating a second image corresponding to the original frame image based on the target key points and the original frame image in the first video includes: obtaining deformation information corresponding to the original frame image based on the target key points corresponding to the original frame image in the first video; wherein the deformation information is used to indicate the mapping relationship between occluders in the original frame image and occluders in the target frame image corresponding to the original frame image, and the mapping relationship between the unoccluded facial region in the original frame image and the unoccluded facial region in the target frame image corresponding to the original frame image; and performing grid sampling processing on the original frame image based on the deformation information corresponding to the original frame image to obtain the second image corresponding to the original frame image.

[0009] Optionally, generating a target frame image corresponding to the original frame image based on the first image and the second image corresponding to the original frame image includes: obtaining a mask frame image corresponding to the original frame image; wherein the mask frame image is an image obtained by performing local masking processing on a target face in the original frame image; performing fusion processing on the first image, the mask frame image corresponding to the original frame image, and the second image corresponding to the original frame image to obtain a fusion processing result; and obtaining the target frame image corresponding to the original frame image based on the fusion processing result.

[0010] Optionally, the fusion processing based on the first image, the mask frame image corresponding to the original frame image, and the second image corresponding to the original frame image to obtain the fusion processing result includes: extracting target facial features based on the first image and the mask frame image corresponding to the original frame image; extracting specified features based on the second image corresponding to the original frame image; wherein the specified features include at least occlusion features; and performing multi-scale fusion processing on the target facial features and the specified features based on spatial adaptive normalization technology to obtain the fusion processing result.

[0011] Optionally, the step of performing multi-scale fusion processing on the target facial features and the specified features based on spatial adaptive normalization technology to obtain a fusion processing result includes: generating a fusion processing result based on the target facial features and the specified features using a preset set of sequentially connected first networks and a set of second networks; wherein the first network and the second network are both spatial adaptive normalization networks, the input information of the first network and the second network both include the target facial features, and the input information of the non-first first network also includes the output result of the preceding network of the non-first first network, and the input information of the second network also includes the output result of the preceding network of the second network and the specified features.

[0012] Optionally, the second image also carries information about the unobstructed facial region of the target face, and the specified feature further includes features of the unobstructed facial region of the target face.

[0013] Optionally, the fusion processing result is represented by a third image; obtaining the target frame image corresponding to the original frame image based on the fusion processing result includes: optimizing the third image to obtain the target frame image corresponding to the original frame image.

[0014] Optionally, the video generation method is executed through a preset target network model, and the target network model is trained in the following manner: acquiring sample video and sample audio, and preprocessing the sample video; wherein, the preprocessing includes: adding occlusions to a portion of the reference frame images corresponding to the sample video, and / or adding occlusions to at least a portion of the original frame images in the sample video; based on the sample audio and the preprocessed sample video, acquiring a first image and a target frame image generated by a preset initial generation model; and performing supervised training based on the first image and target frame image generated by the initial generation model to obtain a target network model based on the trained initial generation model.

[0015] This disclosure also provides a video generation apparatus, comprising: an audio / video acquisition module for acquiring a first video and target audio; wherein the first video is a video containing a target face; a key point determination module for determining target key points in a target frame image to be generated corresponding to an original frame image in the first video based on the target audio; wherein the target key points are key points of the target face in the target frame image to be generated; a first generation module for generating a first image and a second image corresponding to the original frame image based on the first video and the target key points; wherein the first image carries information about the complete facial region of the target face, and the second image carries information about an occlusion corresponding to the target face; a second generation module for generating a target frame image corresponding to the original frame image based on the first image and the second image corresponding to the original frame image; and a video acquisition module for arranging the target frame images corresponding to the original frame image in sequence to obtain a target video.

[0016] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the video generation method provided in this disclosure.

[0017] This disclosure also provides a computer-readable storage medium storing a computer program for performing the video generation method provided in this disclosure.

[0018] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the video generation method provided in this disclosure.

[0019] The technical solution provided in this disclosure fully considers that a person's face may sometimes be obscured when speaking. Therefore, when generating video, the target key points (i.e., key points of the target face in the target frame image) corresponding to the original frame image in the first video (i.e., the video containing the target face) can be determined first based on the target audio. Then, based on the first video and the target key points, a first image carrying complete facial region information of the target face and a second image carrying information of the occlusion corresponding to the target face are generated. On this basis, the corresponding target frame image is generated. This helps to ensure that the unobscured part of the face in the generated target video can be generated normally without being affected by the occlusion, and to ensure that the occlusion is generated normally. This can effectively improve the problems of severe occlusion deformation and facial generation quality degradation caused by the occlusion in related technologies, and can effectively improve the user experience.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0022] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating a video generation method provided in this embodiment of the present disclosure;

[0024] Figure 2 This is a schematic diagram of the structure of a fusion network provided in an embodiment of the present disclosure;

[0025] Figure 3 This is a schematic diagram of a video generation process provided in an embodiment of the present disclosure;

[0026] Figure 4 This is a schematic diagram of the structure of a video generation device provided in an embodiment of the present disclosure;

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0029] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0030] Figure 1 This is a flowchart illustrating a video generation method provided in an embodiment of the present disclosure. The method can be executed by a video generation device, which can be implemented using software and / or hardware, and is generally integrated into an electronic device. Figure 1As shown, the method mainly includes the following steps S102 to S110:

[0031] Step S102: Obtain the first video and the target audio. The first video is a video containing the target's face.

[0032] The target face can be any face of a target object with facial features. The target object can be a real object such as a person, or a virtual object such as a doll or a digital human. The first video can be a video of the target object speaking. When the target object speaks, the lip movements of the target face will usually change. The target audio is the new audio intended to replace the original audio in the first video (i.e., the audio of the target object speaking). This embodiment of the disclosure does not limit the content of the target audio. For example, the target audio is the audio obtained by translating the audio in the first video into a specified language. That is, the target audio is the audio related to the content of the first video. In other words, the first video is a video of user A speaking in language A, and the target audio is the audio that translates language A into language B. Or, the target audio is audio that is completely unrelated to the audio in the first video. The specific settings can be flexible. In practical applications, the duration of the target audio can be equal to the duration of the target object speaking in the first video. For example, the first video is entirely a video of the target object speaking. For example, the first video can be an interview video, a spoken video, etc. That is, if each frame of the original image in the first video contains the target face, then the duration of the target audio is equal to the duration of the first video. After obtaining the first video and the target audio in this embodiment, the first video and the target audio are processed to obtain a new video in which the lip movements of the target face in the first video match the target audio.

[0033] Step S104: Based on the target audio, determine the target key points in the target frame image to be generated corresponding to the original frame image in the first video; wherein, the target key points are the key points of the target face in the target frame image to be generated.

[0034] The original frame images in the first video specifically refer to each frame image contained in the first video. That is, for each original frame image contained in the first video, target key points in the corresponding target frame image are determined based on the target audio. Specifically, based on the time point of the original frame image in the first video, the audio content corresponding to the original frame image in the target audio can be determined, thereby determining the facial expressions such as lip movements corresponding to the audio content. This is used to determine the key points of the target face in the target frame image to be generated, thus obtaining the target key points. In practical applications, lip-syncing algorithms such as the Blendshape algorithm can be used to determine the target key points based on the target audio. These target key points are the key points of the target face in the target frame image to be generated. In other words, the target key points can be used to generate the target face in the target frame image, so that the facial lip movements in the generated image present a visual effect that matches the new audio.

[0035] Step S106: Based on the first video and target key points, generate a first image and a second image corresponding to the original frame image; wherein, the first image carries information about the complete facial region of the target face, and the second image carries information about the occlusions corresponding to the target face.

[0036] The first image can be an image obtained by fusing a portion of the original frame images in the first video. For example, original frame images where the face is completely unobstructed can be selected from the first video, and then fused based on the selected original frame images to obtain a first image carrying information about the complete facial region of the target face, for reference during subsequent face generation. Additionally, for each original frame image in the first video, this embodiment can also generate a corresponding second image. The second image carries information about any occlusion of the target face in the original frame image. Any object obstructing the target face can be used as an occluder, such as a hand, a water cup, a book, a doll, etc., without limitation. This embodiment fully considers the impact of occlusions on face generation, therefore, an additional second image is generated for each original frame image. It should also be noted that if the target face in a certain original frame image does not have an occluder, the occlusion information corresponding to the target face carried in the corresponding second image is empty or in a default state.

[0037] Step S108: Based on the first image and the second image corresponding to the original frame image, generate the target frame image corresponding to the original frame image.

[0038] For each original frame image in the first video, a corresponding target frame image can be generated based on the unified first image and the corresponding second image. The first image can be used to provide complete facial information and avoid the influence of occlusions on facial features, while the second image can provide the corresponding occlusion information. This comprehensive approach ensures that the final generated target frame image can accurately present the information of the occlusions and also present the information of the unoccluded facial areas.

[0039] Step S110: Arrange the target frame images corresponding to the original frame images in order to obtain the target video.

[0040] Each original frame image corresponds to a corresponding target frame image. Based on the arrangement order of the original frame images in the first video, the target frame images corresponding to each original frame image can be arranged to obtain the target video.

[0041] The embodiments disclosed herein fully consider that a person's face may sometimes be obscured when speaking. The above-mentioned technical solutions help ensure that the unobscured part of the face in the generated target video can be generated normally without being affected by the obstruction, and ensure that the obstruction is generated normally. This can effectively improve the problems of severe occlusion deformation and degraded facial generation quality caused by the obstruction in related technologies.

[0042] In some implementations, step S106 above, that is, the step of generating a first image and a second image corresponding to the original frame image based on the first video and the target key points, can be performed with reference to steps A to C below:

[0043] Step A involves selecting multiple reference frame images from the first video and obtaining reference key points for the target face in these images; the target face in the reference frame images must be completely unobstructed. The number of reference frame images can be flexibly set, such as 20 to 30, and is not limited here. In practical applications, a 3D face model can be constructed using methods such as face reconstruction algorithms, and then reference key points for the target face can be generated based on the 3D face model. The reference key points are the key points of the target face in the reference frame images, and the number of reference key points is the same as the number of target key points, exhibiting a one-to-one correspondence.

[0044] For example, the steps of selecting multiple reference frame images from the first video described above can be performed as follows (1) to (3):

[0045] (1) Perform occlusion detection processing on the first video to obtain the occlusion detection result corresponding to the original frame image in the first video. Specifically, an appropriate detection algorithm can be used to perform occlusion detection processing on the original frame image of the first video to determine whether there is an occluded area on the target face in the original frame image, thereby obtaining the occlusion detection result corresponding to the original frame image. The occlusion detection result can at least indicate whether there is an occlusion on the target face, and can further indicate the position information of the occlusion on the target face.

[0046] (2) Based on the occlusion detection results corresponding to the original frame images, the original frame images in which the target face is completely unoccluded are selected as candidate frame images. It is understandable that there may be a lot of original frame images in the first video in which the target face is completely unoccluded, so they can be initially screened as candidate frame images for further selection in the future.

[0047] (3) Based on the lip shape information of the target face in the candidate frame images, multiple reference frame images with different lip shapes are selected from the candidate frame images. In practical applications, multiple reference frame images with different lip shapes can be selected based on the lip shape information of the target face in each candidate frame image using methods such as uniform sampling. Alternatively, based on the lip shape information of the target face in each candidate frame image, the multiple lip shape categories covered by the candidate frame images can be determined, and then at least one candidate frame image is selected as a reference frame image from the candidate frame images corresponding to each lip shape category. The number of reference frame images selected for different lip shape categories can be the same or different, and can be flexibly set. Through the above method, multiple reference frame images with different lip shapes can be obtained, which helps to fuse multiple reference frame images more comprehensively and reasonably in the future.

[0048] Step B involves fusing multiple reference frame images based on the target key points and reference key points to obtain the first image. As mentioned earlier, the target key points and reference key points have a corresponding relationship, and multiple reference frame images can cover different lip shapes. Different lip shapes result in different facial features. Fusion helps to make the obtained first image more richly and comprehensively present the features of the target face. In some specific implementation examples, step B can be performed as follows: Steps B1 to B3

[0049] Step B1 involves acquiring deformation information and weights corresponding to multiple reference frame images based on target keypoints and reference keypoints. The deformation information of the reference frame images indicates the mapping relationship between the target face in the reference frame image and the target face in the corresponding target frame image. Specifically, deformation information corresponding to the target face in multiple reference frame images can be generated based on the target keypoints of the target frame images and the reference keypoints of the multiple reference frame images. This deformation information can be used to deform the reference frame images, thereby aligning the reference keypoints in the reference frame images with the target keypoints in the target frame images, enabling the conversion of the reference frame images into target frame images. The deformation information can also be referred to as deformation flow, which can be characterized through vector field information. The weights mentioned above can also be related to the deformation information, such as depending on the deviation between the target keypoints and the corresponding reference keypoints. The specific calculation methods for the deformation information and weights can be found in relevant technologies and will not be elaborated here.

[0050] Step B2 involves performing grid sampling on the reference frame image based on the deformation information corresponding to the reference frame image to obtain the deformation image corresponding to the reference frame image. For example, based on the deformation information represented by the vector field, the reference frame image can be converted into a deformation image, specifically an RGB image, using the above method.

[0051] Step B3 involves weighted fusion of the deformation images corresponding to each of the multiple reference frame images based on their respective weights, to obtain the first image. For example, the pixel value of each pixel in the first image is equal to the weighted sum of the pixel values ​​corresponding to that pixel in each of the reference frame images.

[0052] The above methods help to enable the first image to present the complete information of the target face in a more objective and comprehensive way.

[0053] Step C: Based on the target key points and the original frame images in the first video, generate a second image corresponding to the original frame images. In some specific implementation examples, step C can be performed as follows: Steps C1 to C2:

[0054] Step C1: Based on the target key points corresponding to the original frame image in the first video, obtain the deformation information corresponding to the original frame image; wherein, the deformation information is used to indicate the mapping relationship between occluders in the original frame image and occluders in the corresponding target frame image, as well as the mapping relationship between the unoccluded facial regions in the original frame image and the unoccluded facial regions in the corresponding target frame image. The above deformation information can also be represented by vector field information.

[0055] Step C2 involves performing grid sampling on the original frame image based on the deformation information corresponding to the original frame image to obtain a second image corresponding to the original frame image. That is, based on the deformation information corresponding to the original frame image, the original frame image can be converted into a second image carrying information about occlusions corresponding to the target face and information about the areas of the target face not occluded by occlusions. It should be noted that if the target face in the original frame image is not occluded, the target face in the second image will also not be occluded, and the occlusion information corresponding to the target face carried in the second image will be empty or in a default state.

[0056] In some implementations, step S108, which is the step of generating the target frame image corresponding to the original frame image based on the first image and the second image corresponding to the original frame image, can be performed with reference to steps a to c below:

[0057] Step a: Obtain the mask frame image corresponding to the original frame image; wherein, the mask frame image is the image obtained by locally masking the target face in the original frame image. The area of ​​the local mask can be flexibly set according to requirements, for example, the lower half of the target face in the original frame image can be masked.

[0058] Step b involves fusing the first image, the mask frame image corresponding to the original frame image, and the second image corresponding to the original frame image to obtain the fusion result. The first image primarily provides complete facial feature information, the second image primarily provides information on occlusions and unoccluded facial information, and the mask frame image primarily removes interference information. By fusing these three types of images, the resulting fusion result can be represented by images. The effect achieved is that even with occlusions, the unoccluded parts of the target face and the corresponding occlusions can be generated normally. This effectively improves the problems of severe occlusion deformation and degraded facial generation quality caused by directly generating faces without considering the influence of occlusions in related technologies.

[0059] In some implementation examples, step b can be performed with reference to steps b1 to b3 as follows:

[0060] Step b1: Extract target facial features based on the masked frame images corresponding to the first image and the original frame image. In practical applications, neural networks can be used to extract target facial features. The masked frame images help eliminate interference and facilitate network learning, thereby further improving the reliability of the target facial features extracted by the network.

[0061] Step b2: Extract specified features based on the second image corresponding to the original frame image; wherein, the specified features include at least occlusion features, and in addition, the second image also carries information about the unoccluded facial region of the target face, and the specified features also include features of the unoccluded facial region of the target face.

[0062] Step b3 involves performing multi-scale fusion processing on the target facial features and specified features based on spatial adaptive normalization technology to obtain the fusion result. Spatial adaptive normalization technology can be implemented through a spatial adaptive normalization network, which can adaptively adjust the normalization parameters and generate spatially adaptive scaling and offset parameters. These parameters can be applied to different spatial locations of the feature map during the normalization process. Specifically, adaptive parameters of spatial dimensions can also be introduced, thereby better preserving information such as spatial structure and comprehensively ensuring the reliability of the fusion processing result.

[0063] In some specific implementation examples, step b3, when executed, can generate a fusion processing result based on the target facial features and specified features, using multiple pre-defined, sequentially connected first networks and multiple second networks. Here, both the first and second networks are spatially adaptive normalization networks. The input information of both the first and second networks includes the target facial features, and the input information of any network other than the first first network also includes the output result of the preceding network. Similarly, the input information of the second network includes the output result of the preceding network and the specified features. For ease of understanding, please refer to... Figure 2 The diagram illustrates a fusion network structure, using two first networks and two second networks as an example. The input information to each network other than the first network includes the output of the preceding network. Complete target facial features are input to each network, while specific features (including occlusion features and features of the unoccluded facial regions) can be input only to the second network to better process occlusion features later. It should be noted that... Figure 2 This is merely an example; in practical applications, the number of the first and second networks can be flexibly set. Through this method, multi-scale feature fusion can be achieved, effectively ensuring the richness and comprehensiveness of the final fusion result.

[0064] Step c: Obtain the target frame image corresponding to the original frame image based on the fusion processing result. For example, the fusion processing result is represented by a third image. The third image can present unobstructed target facial features matching the content of the target audio, as well as occlusion features corresponding to the target face, and can also present features such as the original background. In some implementation examples, the third image can be directly used as the target frame image. In other implementation examples, the third image can be optimized to obtain the target frame image corresponding to the original frame image. That is, the optimized third image is used as the target frame image. This optimization processing may include facial blemish removal processing, including but not limited to discontinuities between occlusions and the face. In practical applications, the third image can be optimized using methods such as an encoder. This approach can further improve the generation quality of the target frame image, ensuring that the target frame image has better visual effects.

[0065] For easier understanding, you can also refer to Figure 3 The diagram illustrates a video generation process. It shows that a reference frame image, reference key points, target key points, original frame image, and mask frame image are first obtained based on the input first video and target audio. The specific determination method can be found in the aforementioned related content. Then, facial motion prediction processing is performed based on the reference frame image, reference key points, and target key points to generate a first image. The execution flow of facial motion prediction processing can be found in steps B1 to B3. Furthermore, occlusion motion prediction processing is performed based on the target key points and the original frame image to generate a second image. The execution flow of occlusion motion prediction processing can be found in steps C1 to C2. Then, the first image, the second image, and the mask frame image can be fused to obtain the third image. The fusion process can be referred to steps b1 to b3 above. Subsequently, the third image can be optimized by encoding methods such as encoders to obtain the target frame image. Based on the arrangement order of the original frame images, the target frame images corresponding to each original frame image are arranged in order to obtain the target video. The target video is mainly based on the target audio, which changes the facial features such as the mouth shape of the target face in the first video to match the facial features such as the mouth shape of the target face with the content of the target video. Moreover, even when the target face is occluded, it can accurately present the occluded object and the unoccluded area of ​​the target face.

[0066] In practical applications, the above video generation method is executed through a preset target network model, and networks such as the aforementioned fusion network are all included in the target network model. This embodiment of the disclosure uses the target network model as an example to illustrate the training method. For example, the target network model is trained through the following steps 1 to 3:

[0067] Step 1: Acquire sample video and sample audio, and preprocess the sample video; wherein, the preprocessing includes: adding occlusions to a portion of the reference frame images corresponding to the sample video, and / or adding occlusions to at least a portion of the original frame images in the sample video. Specifically, occlusions can be added to the reference frame images with a certain probability, and / or, occlusions can be added to the original frame images.

[0068] Step 2: Based on the sample audio and the preprocessed sample video, obtain the first image and the target frame image generated by the preset initial generation model. The specific method for obtaining the first image and the target frame image can be found in the aforementioned related content, and will not be repeated here.

[0069] Step 3 involves supervised training based on the first image generated by the initial generation model and the target frame image, to obtain the target network model based on the trained initial generation model. For example, when occlusion is added to a portion of the reference frame image corresponding to the sample video, the first image generated by the supervised model can remain unchanged, that is, to ensure that the first image with complete facial information generated by the model is unaffected by occlusion as much as possible. Similarly, when occlusion is added to at least a portion of the original frame images in the sample video, the occlusion region of the target frame image generated by the supervised model can remain unchanged. In practical applications, a first loss corresponding to the first image and a second loss corresponding to the target frame image can be obtained. The total loss is determined based on the first and second losses, and training ends when the total loss converges to a preset threshold. Through this method, the trained initial generation model can accurately perceive facial occlusions, ensuring that the unoccluded parts of the face in the generated target frame image can be generated normally without being affected by occlusions, and ensuring that occlusions are generated normally, exhibiting strong occlusion robustness.

[0070] In practical applications, the initial generated model after training can be used directly as the target network model, or other functional modules can be added to the initial generated model after training to obtain the target network model. The specific settings can be flexibly configured and are not restricted here.

[0071] In summary, the method provided in this disclosure fully considers that a person's face may sometimes be obscured when speaking. In scenarios where the target face of the original first video is regenerated based on the target audio to obtain a target video whose lip movements match the target audio, additional modeling of occlusions can be performed. Even when the face is obscured, it can effectively ensure that the unobscured parts of the face in the generated target video can be generated normally without being affected by occlusions, and also ensure the normal generation of occlusions. This effectively improves the problems of severe occlusion deformation and degraded face generation quality caused by occlusions in related technologies, comprehensively ensuring the generation quality and visual experience of the final target video, and effectively enhancing the user experience.

[0072] Corresponding to the aforementioned video generation method, this disclosure further provides a video generation apparatus. Figure 4 This is a schematic diagram of a video generation device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 4 As shown, the video generation device includes:

[0073] The audio and video acquisition module 402 is used to acquire a first video and a target audio; wherein, the first video is a video containing the target's face;

[0074] The key point determination module 404 is used to determine the target key points in the target frame image to be generated corresponding to the original frame image in the first video based on the target audio; wherein, the target key points are the key points of the target face in the target frame image to be generated.

[0075] The first generation module 406 is used to generate a first image and a second image corresponding to the original frame image based on the first video and target key points; wherein, the first image carries information about the complete facial region of the target face, and the second image carries information about the occlusions corresponding to the target face;

[0076] The second generation module 408 is used to generate a target frame image corresponding to the original frame image based on the first image and the second image corresponding to the original frame image.

[0077] The video acquisition module 410 is used to arrange the target frame images corresponding to the original frame images in order to obtain the target video.

[0078] The aforementioned device helps ensure that the unobstructed facial portion of the generated target video can be generated normally without being affected by obstructions, and also ensures that obstructions are generated normally. It can effectively improve problems such as severe occlusion deformation and degraded facial generation quality caused by obstructions in related technologies.

[0079] In some implementations, the first generation module 406 is specifically used to: select multiple reference frame images from the first video and obtain reference key points of the target face in the reference frame images; wherein the target face in the reference frame images is completely unobstructed; perform fusion processing on the multiple reference frame images based on the target key points and the reference key points to obtain a first image; and generate a second image corresponding to the original frame image based on the target key points and the original frame image in the first video.

[0080] In some implementations, the first generation module 406 is specifically used to: perform occlusion detection processing on the first video to obtain occlusion detection results corresponding to the original frame images in the first video; select original frame images where the target face is completely unoccluded as candidate frame images based on the occlusion detection results corresponding to the original frame images; and select multiple reference frame images with different lip shapes from the candidate frame images based on the lip shape information of the target face in the candidate frame images.

[0081] In some implementations, the first generation module 406 is specifically used to: obtain deformation information and weights corresponding to each of the plurality of reference frame images based on the target key points and the reference key points; wherein, the deformation information of the reference frame images is used to indicate the mapping relationship between the target face in the reference frame image and the target face in the target frame image corresponding to the reference frame image; perform grid sampling processing on the reference frame images based on the deformation information corresponding to the reference frame images to obtain the deformation images corresponding to the reference frame images; and perform weighted fusion on the deformation images corresponding to each of the plurality of reference frame images based on the weights corresponding to each of the plurality of reference frame images to obtain a first image.

[0082] In some embodiments, the first generation module 406 is specifically used to: obtain deformation information corresponding to the original frame image based on the target key points corresponding to the original frame image in the first video; wherein, the deformation information is used to indicate the mapping relationship between occluders in the original frame image and occluders in the target frame image corresponding to the original frame image, and the mapping relationship between the unoccluded facial region in the original frame image and the unoccluded facial region in the target frame image corresponding to the original frame image; and perform grid sampling processing on the original frame image based on the deformation information corresponding to the original frame image to obtain a second image corresponding to the original frame image.

[0083] In some embodiments, the second generation module 408 is specifically used to: obtain a mask frame image corresponding to the original frame image; wherein the mask frame image is an image obtained by performing local masking processing on the target face in the original frame image; perform fusion processing based on the first image, the mask frame image corresponding to the original frame image, and the second image corresponding to the original frame image to obtain a fusion processing result; and obtain a target frame image corresponding to the original frame image based on the fusion processing result.

[0084] In some embodiments, the second generation module 408 is specifically used to: extract target facial features based on the mask frame image corresponding to the first image and the original frame image; extract specified features based on the second image corresponding to the original frame image; wherein the specified features include at least occlusion features; and perform multi-scale fusion processing on the target facial features and the specified features based on spatial adaptive normalization technology to obtain a fusion processing result.

[0085] In some embodiments, the second generation module 408 is specifically used to: generate a fusion processing result based on the target facial features and the specified features, using a preset plurality of sequentially connected first networks and a plurality of second networks; wherein the first network and the second network are both spatial adaptive normalization networks, the input information of the first network and the second network both include the target facial features, and the input information of the non-first first network also includes the output result of the previous network of the non-first first network, and the input information of the second network also includes the output result of the previous network of the second network and the specified features.

[0086] In some embodiments, the second image also carries information about the unobstructed facial region of the target face, and the designated feature further includes features of the unobstructed facial region of the target face.

[0087] In some implementations, the fusion processing result is represented by a third image; the second generation module 408 is specifically used to: optimize the third image to obtain a target frame image corresponding to the original frame image.

[0088] In some embodiments, the video generation method is executed through a preset target network model. The device further includes a model training module for training the target network model in the following manner: acquiring sample video and sample audio, and preprocessing the sample video; wherein the preprocessing includes: adding occlusions to a portion of the reference frame images corresponding to the sample video, and / or adding occlusions to at least a portion of the original frame images in the sample video; acquiring a first image and a target frame image generated by a preset initial generation model based on the sample audio and the preprocessed sample video; and performing supervised training based on the first image and target frame image generated by the initial generation model to obtain the target network model based on the trained initial generation model.

[0089] The video generation apparatus provided in this disclosure can execute the video generation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0091] This disclosure provides an electronic device, which includes: a storage device storing a computer program thereon; and a processing device for executing the computer program in the storage device to implement the steps of any method of this disclosure.

[0092] The following is for reference. Figure 5 This document illustrates a structural schematic diagram of an electronic device 500 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Devices), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital televisions and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0093] like Figure 5As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0094] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0095] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0096] In addition to the methods and devices described above, embodiments of this disclosure can also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the methods provided in the embodiments of this disclosure. The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0097] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the video generation method provided in embodiments of this disclosure.

[0098] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0099] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the video generation method of this disclosure.

[0100] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0101] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0102] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0103] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0104] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0105] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video generation method, characterized in that, include: Acquire a first video and a target audio; wherein the first video is a video containing the target's face; Based on the target audio, target key points in the target frame image to be generated corresponding to the original frame image in the first video are determined; wherein, the target key points are key points of the target face in the target frame image to be generated. Based on the first video and the target key points, a first image and a second image corresponding to the original frame image are generated; wherein, the first image carries information about the complete facial region of the target face, and the second image carries information about the occlusions corresponding to the target face; Based on the first image and the second image corresponding to the original frame image, a target frame image corresponding to the original frame image is generated; The target frame images corresponding to the original frame images are arranged in order to obtain the target video.

2. The method according to claim 1, characterized in that, The step of generating a first image and a second image corresponding to the original frame image based on the first video and the target key points includes: Multiple reference frame images are selected from the first video, and reference key points of the target face in the reference frame images are obtained; wherein the target face in the reference frame images is completely unobstructed; Based on the target key points and the reference key points, the multiple reference frame images are fused to obtain the first image; Based on the target key points and the original frame images in the first video, a second image corresponding to the original frame images is generated.

3. The method according to claim 2, characterized in that, The step of selecting multiple reference frame images from the first video includes: Perform occlusion detection processing on the first video to obtain the occlusion detection results corresponding to the original frame images in the first video. Based on the occlusion detection results corresponding to the original frame image, the original frame image in which the target face is completely unoccluded is selected as the candidate frame image. Based on the lip shape information of the target face in the candidate frame image, multiple reference frame images with different lip shapes are selected from the candidate frame image.

4. The method according to claim 2, characterized in that, The step of fusing the multiple reference frame images based on the target key points and the reference key points to obtain a first image includes: Based on the target key points and the reference key points, the deformation information and weights corresponding to each of the multiple reference frame images are obtained; wherein, the deformation information of the reference frame images is used to indicate the mapping relationship between the target face in the reference frame image and the target face in the target frame image corresponding to the reference frame image; Based on the deformation information corresponding to the reference frame image, the reference frame image is subjected to grid sampling processing to obtain the deformation image corresponding to the reference frame image; Based on the weights corresponding to each of the multiple reference frame images, the deformation images corresponding to each of the multiple reference frame images are weighted and fused to obtain the first image.

5. The method according to claim 2, characterized in that, The step of generating a second image corresponding to the original frame image based on the target key points and the original frame image in the first video includes: Based on the target key points corresponding to the original frame image in the first video, the deformation information corresponding to the original frame image is obtained; wherein, the deformation information is used to indicate the mapping relationship between the occluder in the original frame image and the occluder in the target frame image corresponding to the original frame image, and the mapping relationship between the unoccluded facial region in the original frame image and the unoccluded facial region in the target frame image corresponding to the original frame image. Based on the deformation information corresponding to the original frame image, the original frame image is subjected to grid sampling processing to obtain the second image corresponding to the original frame image.

6. The method according to any one of claims 1 to 5, characterized in that, The step of generating a target frame image corresponding to the original frame image based on the first image and the second image corresponding to the original frame image includes: Obtain the masked frame image corresponding to the original frame image; wherein, the masked frame image is an image obtained by performing local masking processing on the target face in the original frame image; A fusion process is performed based on the first image, the masked frame image corresponding to the original frame image, and the second image corresponding to the original frame image to obtain the fusion process result. The target frame image corresponding to the original frame image is obtained based on the fusion processing result.

7. The method according to claim 6, characterized in that, The process of fusing the first image, the masked frame image corresponding to the original frame image, and the second image corresponding to the original frame image to obtain the fusion result includes: Based on the masked frame image corresponding to the first image and the original frame image, extract the target facial features; Based on the second image corresponding to the original frame image, specified features are extracted; wherein, the specified features include at least occlusion features; Based on spatial adaptive normalization technology, the target facial features and the specified features are fused at multiple scales to obtain the fusion result.

8. The method according to claim 7, characterized in that, The spatial adaptive normalization technique is used to perform multi-scale fusion processing on the target facial features and the specified features to obtain the fusion result, including: Based on the target facial features and the specified features, a fusion processing result is generated using a pre-defined, sequentially connected set of first networks and a set of second networks. Wherein, both the first network and the second network are spatial adaptive normalization networks. The input information of both the first network and the second network includes the target facial features. The input information of the non-first first network also includes the output result of the previous network. The input information of the second network also includes the output result of the previous network and the specified features.

9. The method according to claim 7, characterized in that, The second image also carries information about the unobstructed facial region of the target face, and the specified feature further includes features of the unobstructed facial region of the target face.

10. The method according to claim 6, characterized in that, The fusion processing result is represented by a third image; obtaining the target frame image corresponding to the original frame image based on the fusion processing result includes: The third image is optimized to obtain the target frame image corresponding to the original frame image.

11. The method according to claim 2, characterized in that, The video generation method is executed through a preset target network model, and the target network model is trained in the following manner: Acquire sample video and sample audio, and preprocess the sample video; wherein, the preprocessing includes: adding occlusions to a portion of the reference frame images corresponding to the sample video, and / or adding occlusions to at least a portion of the original frame images in the sample video; Based on the sample audio and the preprocessed sample video, obtain the first image and the target frame image generated by the preset initial generation model; Supervised training is performed on the first image and the target frame image generated by the initial generation model to obtain the target network model based on the trained initial generation model.

12. A video generation apparatus, characterized in that, include: The audio and video acquisition module is used to acquire a first video and target audio; wherein, the first video is a video containing the target face; The key point determination module is used to determine, based on the target audio, target key points in the target frame image to be generated corresponding to the original frame image in the first video; wherein, the target key points are key points of the target face in the target frame image to be generated. A first generation module is configured to generate a first image and a second image corresponding to the original frame image based on the first video and the target key points; wherein the first image carries information about the complete facial region of the target face, and the second image carries information about the occlusions corresponding to the target face; The second generation module is used to generate a target frame image corresponding to the original frame image based on the first image and the second image corresponding to the original frame image; The video acquisition module is used to arrange the target frame images corresponding to the original frame images in sequence to obtain the target video.

13. An electronic device, characterized in that, The electronic device includes: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the video generation method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the video generation method according to any one of claims 1-11.

15. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the video generation method according to any one of claims 1-11.

Images