Digital human video generation method and system based on monocular video replay

By adopting optical flow enhancement method of generative adversarial network and autoregressive architecture in monocular video reenactment technology or introducing texture adhesion improvement method of StyleGAN3 module, the problem of insufficient timing continuity and authenticity of digital human video generation is solved, and the problem of High-fidelity and controllable digital video generation reduces development difficulty and manual modeling workload.

CN118972676AActive Publication Date: 2024-11-15ZHEJIANG UNIV +1

Patent Information

Application Number
CN202410856057.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-28
Publication Date
2024-11-15
Estimated Expiration
2044-06-28

AI Technical Summary

Technical Problem

When generating digital human videos, the existing monocular video reenactment technology lacks timing continuity, controllability and authenticity, making it difficult to meet the needs of large-scale digital human applications.

Method used

The optical flow enhancement method based on the generative adversarial network and autoregressive architecture, or the texture adhesion improvement method of StyleGAN3's aliasing-free convolution module and global translation rotation transformation module are introduced into the generative adversarial network architecture to generate digital human videos.

Benefits of technology

It improves the timing continuity and authenticity of generated videos, realizes the high fidelity and controllability of digital human videos, reduces the difficulty of developing digital human-related applications, and reduces the workload of manual modeling.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972676A_ABST
    Figure CN118972676A_ABST
Patent Text Reader

Abstract

The invention discloses a digital human video generation method and device based on monocular video replay. The method comprises the following steps: generating a key point sequence based on a monocular video; generating an image sequence representing neural rendering based on the key point sequence; generating a digital human video based on the neural rendering image sequence, namely generating the digital human video by adopting an optical flow enhancement mode based on a generative adversarial network and an autoregression architecture, or generating a digital human video by adopting a texture adhesion improvement mode of introducing a non-aliasing convolution module and a global translation and rotation transformation module of StyleGAN3 in a generative adversarial network architecture. Therefore, the time sequence continuity, controllability and authenticity of the generated video can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of monocular video replay, and in particular relates to a method and system for generating digital human video based on monocular video replay. Background Art

[0002] With the development of artificial intelligence and computer graphics, virtual digital humans have gradually entered the public eye and are widely used in various fields, such as intelligent customer service, virtual anchors, online education, game entertainment, etc. In human-computer interaction, virtual digital humans can establish connections with users faster and improve user experience.

[0003] Virtual digital human technology consists of four parts: modeling, driving, presentation and interaction. Modeling is to create the specific form of digital human, driving is to use captured or generated data to change the representation of the target digital human, presentation is to convert various forms of digital human into visual images, and interaction is the interaction between digital human and environment or objects, involving physics, emotion, speech, action, etc.

[0004] The traditional method of making digital humans requires modeling before driving, which is a complex and time-consuming process. It is not suitable for large-scale digital human production needs, which hinders the popularization and application of digital humans. Therefore, using generative artificial intelligence technology to generate highly realistic, large-scale virtual digital humans has become a current research hotspot. For example, the patent application with publication number CN 118196243 A discloses a digital human video generation method and system, which generates multiple digital human images based on a face rendering model after obtaining expression data and head posture data, and merges the digital human images into a video. For example, the patent application with publication number CN117593683A discloses a video generation algorithm and system for 2D digital humans, which trains the neural radiation field model based on the video generation task after image segmentation and key point extraction, and inputs the real-time collected model body image into the trained neural radiation field model to generate a video when applied.

[0005] The generative digital human technology based on monocular video replay aims to use monocular video as input to generate digital human videos that are consistent with the driving video’s expressions, head movements, and body postures, and maintain the identity of the target person. This can reduce the difficulty of developing digital human-related applications and reduce the workload of manual modeling. However, generative digital human technology is still in its infancy, and there are still many deficiencies in the authenticity, controllability, and temporal continuity of the generated digital human videos. Summary of the invention

[0006] In view of the above, an object of the present invention is to provide a method and system for generating digital human videos based on monocular video playback to improve the temporal continuity, controllability and authenticity of the generated videos.

[0007] To achieve the above-mentioned purpose of the invention, an embodiment provides a method for generating a digital human video based on monocular video playback, comprising the following steps:

[0008] Generate key point sequence based on monocular video;

[0009] Generate a representation of a neural rendered image sequence based on a sequence of key points;

[0010] Digital human videos are generated based on neural rendering image sequences, including generating digital human videos by using an optical flow enhancement method based on a generative adversarial network and an autoregressive architecture, or generating digital human videos by using a texture adhesion improvement method that introduces an aliasing-free convolution module and a global translation and rotation transformation module of StyleGAN3 into a generative adversarial network architecture.

[0011] Preferably, generating a key point sequence based on a monocular video includes:

[0012] Key points are extracted from monocular video, and time-series smoothing and redirection processing is performed on the key points to obtain a key point sequence.

[0013] Preferably, the method of generating a digital human video by using an optical flow enhancement method based on a generative adversarial network and an autoregressive architecture includes:

[0014] The optical flow enhancement method framework is constructed based on the generative adversarial network, which includes a first generator part and a first discriminator part. The first generator part includes a first generator G1 and an optical flow model G o The first discriminator part includes an image quality discriminator D c , face discriminator D F , and the multi-scale dynamic discriminator D D , and establish autoregression between the output and input of the first generator, taking the generated video frames at the output as the historical generated video frames at the input, and the input neural rendering image sequence X t-K:t and history to generate video frames The current generated video frame is obtained through the first generator G1 The image X will be rendered by the neural t and the real video frame Y t The real image pair (X t ,Y t ) and generate image pairs Input to the image quality discriminator D c Determine the authenticity of the video frame and compare the real image corresponding to the facial area and generate image pairs Input to the face discriminator D F Determine whether the facial region is real or fake, input the real frame sequence Y t-K:t After the optical flow model Go Get the optical flow change W between video frames t-K:t-1 , the real image pair (W t-K:t-1 ,Y t-K:t ) and generate image pairs Input to the multi-scale dynamic discriminator D D Determine the authenticity of the video frame based on the change of optical flow, so that the dynamic changes of the generated video frame based on the change of optical flow are consistent with the dynamic changes of the real video frame;

[0015] After training the optical flow enhancement framework, the trained first generator is extracted as a video generation model for generating digital human videos.

[0016] Preferably, the loss function used for training the optical flow enhancement framework includes the first generator loss L G1 and the second discriminator loss L D1 ;

[0017] First generator loss in Represents the VGG loss constructed based on different layers of visual features extracted by the VGG network for the generated video frames and the real video frames. express The weight hyperparameters are represents the feature matching loss of the discriminator features, which contains the real video frame Y t and generate video frames The difference between the features extracted by the image quality discriminator and the real video frame sequence Y t-K:t and generate a sequence of video frames The difference between the features extracted by the multi-scale dynamic discriminator, express The weight hyperparameters are Denotes the adversarial loss, expressed as:

[0018]

[0019] Among them, E t represents the expected value, Denotes the multi-scale dynamic discriminator D D Based on input The judgment results obtained are Denotes the image quality discriminator D c Based on input The judgment results obtained are Represents the face discriminator based on the input The judgment result of

[0020] First discriminator loss in, Denotes the multi-scale dynamic discriminator D D The discriminant loss, Denotes the image quality discriminator D c The discriminant loss is expressed as:

[0021]

[0022] Among them, D D (W t-K+1:t ,Y t-K:t ) represents the multi-scale dynamic discriminator D D Based on the input (W t-K+1:t ,Y t-K:t ) The discrimination result obtained, D c (X t ,Y t ) represents the image quality discriminator D c Based on the input (X t ,Y t ) to obtain the discrimination result.

[0023] Preferably, the optical flow model includes a VideoFlow model, a GMFlow+ model, or a FlowFormer++ model.

[0024] Preferably, the method of generating a digital human video by introducing an aliasing-free convolution module and a texture adhesion improvement method of a global translation and rotation transformation module of StyleGAN3 into a generative adversarial network architecture comprises:

[0025] Based on the generative adversarial network architecture StyleUNet, a texture adhesion improvement framework is constructed, which includes a second generator part and a second discriminator part. The second generator part includes a mapping network G f , image encoder G e , global translation and rotation transformation module G r , and generate network G2, where the convolution module in the generated network G2 is replaced by the non-aliasing convolution module of StyleGAN3, and the noise input in the generated network G2 is cancelled. e The output of each layer is connected to the non-aliasing convolution module of each layer in the generating network G2 to form a residual structure. The second discriminator part includes the discriminator D2, the random noise Z t After mapping network G f Mapped to latent code W t , the input neural rendering image X tThe image encoder obtains multi-level image features. The image features of the middle level are connected to the generating network G2 through the residual structure. The image features of the last level are converted into Fourier features through Fourier transform and then input into the global translation and rotation transformation module. The global translation and rotation parameters are calculated and input into the generating network G f , in the generated network G f The video frame is generated by combining the global translation and rotation parameters and the intermediate level image features. The image X will be rendered by the neural t and the real video frame Y t The real image pair (X t ,Y t ) and generate image pairs Input to discriminator D2 to determine the authenticity of the video frame;

[0026] After training the texture adhesion improvement framework, the mapping network G contained in the trained second generator is extracted. f , image encoder G e , global translation and rotation transformation module G r , and the generation network G2 constitute a video generation model for generating digital human videos.

[0027] Preferably, in the global translation and rotation transformation module, the global translation and rotation parameters are calculated based on the Fourier feature, including:

[0028] The Fourier features are converted into a four-dimensional vector t = (r c ,r s ,t x ,t y ), and normalize it to get the normalized result t′=(r c ′,r s ′,t x ′,t′ y ):

[0029]

[0030] Among them, r c ′,r s ′ represents the center of the canvas [0,1] 2 The rotation parameter, the rotation angle α is defined as r c ′ = cosα and r s ′=sinα,t x ′,t′ y Indicated by (t x ′,t′ y ) as the unit, the rotation parameter and the translation parameter together constitute the global translation and rotation parameter.

[0031] Preferably, the loss function used for training the texture adhesion improvement framework includes the second generator loss L G2 and the second discriminator loss L D2 ;

[0032] Second generator loss in, Represents the VGG loss constructed based on different layers of visual features extracted by the VGG network for the generated video frames and the real video frames. express The weight hyperparameter, L rec represents the reconstruction loss between the generated video frame and the real video frame, λ rec Indicates L rec The weight hyperparameters are and λ adv Denote adversarial loss and adversarial loss respectively. The weight hyperparameters are It is expressed as:

[0033]

[0034] Among them, fake i represents the discrimination result given by the discriminator D2 for the input generated image pair i, and N represents the total number of image pairs;

[0035] The second discriminator loss L D2 It is expressed as:

[0036]

[0037] Among them, real i Represents the discrimination result given by the discriminator D2 for the input real image pair i.

[0038] To achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a digital human video generation device based on monocular video playback, comprising:

[0039] A key point generation module, which is used to generate a key point sequence based on a monocular video;

[0040] A neural rendering module, which is used to generate a neural rendering image sequence based on the key point sequence;

[0041] A video generation module is used to generate digital human videos based on neural rendering image sequences, including generating digital human videos by using an optical flow enhancement method based on a generative adversarial network and an autoregressive architecture, or generating digital human videos by using a texture adhesion improvement method that introduces the aliasing-free convolution module and the global translation and rotation transformation module of StyleGAN3 into the generative adversarial network architecture.

[0042] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned digital human video generation method based on monocular video replay.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] After extracting key point sequences and generating neural rendering image sequences based on key point sequences, digital human videos are generated based on neural rendering image sequences. When generating videos, the optical flow enhancement method based on generative adversarial networks and autoregressive architecture is used to ensure the temporal continuity and authenticity of the generated videos. The texture adhesion improvement method of introducing the non-aliasing convolution module and global translation and rotation transformation module of StyleGAN3 in the generative adversarial network architecture can achieve the controllability and authenticity of the generated videos. Both methods can generate high-fidelity digital human videos that are consistent with the expressions and movements of the input videos. This technology reduces the difficulty of developing digital human-related applications and reduces the workload of manual modeling. It has broad application prospects in virtual reality, digital entertainment, audio and video games, education and finance, and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 is a flow chart of a method for generating a digital human video based on monocular video playback provided by an embodiment;

[0047] Figure 2 is a flow chart of generating a key point sequence based on a monocular video provided by an embodiment;

[0048] Figure 3 is a structural schematic diagram of the optical flow enhancement method framework provided in the embodiment;

[0049] Figure 4 is a structural schematic diagram of a framework of a texture adhesion improvement method provided in an embodiment;

[0050] Figure 5 It is a structural schematic diagram of a digital human video generation device based on monocular video playback provided in an embodiment. DETAILED DESCRIPTION

[0051] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0052] The technical concept of the present invention is as follows: the technical solution of digital human video based on monocular video replay provided by the embodiment of the present invention aims to realize the generation of target digital human half-body video based on monocular video, which mainly includes three stages: the first stage, generating a key point sequence including face and half-body posture according to the monocular video including the upper body of the person; the second stage, generating a neural rendering image sequence representing the semantics of face and upper body posture according to the key point sequence obtained in the first stage; the third stage, generating a video sequence that conforms to the identity of the target person and is consistent with the expression and action of the driving video according to the neural rendering image sequence.

[0053] like Figure 1 As shown, the method for generating a digital human video based on monocular video playback provided in the embodiment includes the following steps:

[0054] S1, generates key point sequence based on monocular video.

[0055] In the embodiment, according to the input N-frame monocular video sequence V containing the upper body of the person 1:N Generate a sequence of N frames of key points of the target person including the face and upper body posture L 1:N , where L i ∈R 528×3 represents the 3D coordinates of the key points of the i-th frame, including 478 facial key points, 8 torso key points and 21 key points for each hand. Figure 2 As shown, key point sequence generation includes three processes: key point extraction, timing smoothing and redirection.

[0056] For key point extraction, MediaPipe and DWPose are used to extract the key points of the face, body and hands of the characters in the monocular video. MediaPipe is used to extract 478 facial key points, and DWPose is used to extract 33 body key points and 21 hand key points for each hand.

[0057] Regarding temporal smoothing, there will be problems of missing key points and jitter after key point extraction. The video frame context is used to complete the missing key points using linear interpolation, and the exponentially weighted average algorithm is used to perform temporal smoothing on the face, body and hand key points, thereby smoothing small amplitude jitters while retaining the original data samples.

[0058] For redirection, the key point sequence after time series smoothing is redirected using Protoss analysis to redirect the expression and posture of the source character to the target character. Specifically, the average key points of the face, body and hands of the source character and the target character are calculated as the reference key point matrix, and then the transformation matrix from the source character reference key point matrix to the target character reference key point matrix is ​​solved, which includes translation, rotation and scaling matrices. Finally, the transformation matrix is ​​applied to the key point sequence of the face, hand and body of the source character respectively to obtain the redirected key point sequence.

[0059] S2, generates a neural rendered image sequence based on the key point sequence.

[0060] In the embodiment, using the N-frame key sequence L 1:N As input, a neural rendered image sequence X of N frames representing semantic information of facial expressions, head movements, and body postures is generated. 1:N Specifically, according to the dense neural rendering representation method, CCBR images representing the semantic information of facial and upper body postures and eye gaze images representing eye features are generated according to the key point sequence, and these CCBR images and eye gaze images are spliced ​​in the channel dimension as the neural rendering image generated by the video.

[0061] Among them, the CCBR image is represented as an 8-bit RGB image, which is used to represent the semantic information of the face and body posture. For the body posture key points, each key point is drawn as a circle with a fixed radius, and each key point is given a predefined color, where the values ​​of the red and green channels are obtained by the normalized UV coordinates, and the blue channel adopts a predefined fixed value. For the facial key points, rendering is performed by nvdiffrast according to the normalized coordinates of the key points. In addition, since increasing the number of bone points can improve the replay performance, the embodiment interpolates 8 body bone points, where the number and color of the interpolation between the joint points are fixed. After interpolation, a total of 79 body posture key points are obtained.

[0062] The eye gaze image is represented as an 8-bit RGB image, which is used to control finer eye features. Based on the five eye key points of the upper, lower, left, right and center of the eyeball, the pupil center position and pupil radius are obtained, and a red circle with the pupil center as the origin and the pupil radius as the radius is drawn, and the eye edge points are connected and filled with white to generate the eye gaze image.

[0063] S3, Digital Human Video Generation from Neurally Rendered Image Sequences.

[0064] In the embodiment, according to the N-frame neural rendering image sequence X 1:N , generate N frames of video sequence Y of the target person's half-body region that is consistent with the facial expression, head movement, and body posture represented by the neural rendering image1:N .

[0065] In order to improve the temporal continuity and quality of the generated video, the present invention explores two generation paradigms, namely "Generative Adversarial Network + Autoregression" and "Generative Adversarial Network Only", and proposes corresponding improvement schemes. On the one hand, the present invention designs and implements an optical flow enhancement scheme based on the generative adversarial network and autoregressive architecture, introduces and transforms the optical flow model in the multi-scale dynamic discriminator, and improves the temporal continuity of the generated video; on the other hand, the present invention designs and implements a texture adhesion improvement scheme based on StyleUNet, which only adopts the generative adversarial network architecture, and improves the texture adhesion problem by introducing the non-aliasing convolution module and global translation and rotation transformation module of StyleGAN3, thereby improving the quality of the generated video.

[0066] In the embodiment, the optical flow enhancement scheme based on the generative adversarial network and the autoregressive architecture is designed based on the generative adversarial network (GAN) and the autoregressive architecture, and the enhancement of the optical flow information is achieved through the collaborative training of the generator and the discriminator. The generator is responsible for generating enhanced optical flow information to improve the quality and continuity of the video frames, while the discriminator is used to evaluate the authenticity of the generated optical flow information and guide the generator to learn more accurate optical flow information. In terms of experimental design and evaluation, the scheme selects the optimal optical flow model for optical flow enhancement by comparing the performance of different optical flow models.

[0067] Specifically, Figure 3 As shown, the optical flow enhancement method framework is constructed based on the generative adversarial network, which includes a first generator part and a first discriminator part. The first generator part includes a first generator G1 and an optical flow model G o The first discriminator part includes an image quality discriminator D c , face discriminator D F , and the multi-scale dynamic discriminator D D , and establish autoregression between the output and input of the first generator, taking the generated video frames at the output as the historical generated video frames at the input, and the input neural rendering image sequence X t-K:t and history to generate video frames The current generated video frame is obtained through the first generator G1 The image X will be rendered by the neural t and the real video frame Y t The real image pair (X t ,Y t ) and generate image pairs Input to the image quality discriminator D c Determine the authenticity of the video frame and compare the real image corresponding to the facial area and generate image pairs Input to the face discriminator DF Determine whether the facial region is real or fake, input the real frame sequence Y t-K:t After the optical flow model G o Get the optical flow change W between video frames t-K:t-1 , the real image pair (W t-K:t-1 ,Y t-K:t ) and generate image pairs Input to the multi-scale dynamic discriminator D D Determine the authenticity of the video frame based on the change of optical flow, so that the dynamic changes of the generated video frame based on the change of optical flow are consistent with the dynamic changes of the real video frame;

[0068] After training the optical flow enhancement framework, the trained first generator is extracted as the video generation model for the generation of digital human videos. The loss function used in training includes the first generator loss L G1 and the second discriminator loss L D1 ;

[0069] First generator loss in Represents the VGG loss constructed based on different layers of visual features extracted by the VGG network for the generated video frames and the real video frames, where the VGG loss can be constructed using the LI distance. express The weight hyperparameters are represents the feature matching loss of the discriminator features, which contains the real video frame Y t and generate video frames The difference between the features extracted by the image quality discriminator and the real video frame sequence Y t-K:t and generate a sequence of video frames The difference between the features extracted by the multi-scale dynamic discriminator, express The weight hyperparameters are represents the adversarial loss, marking the generated fake samples as 1 and the real samples as 0. It is expressed as:

[0070]

[0071] Among them, E t represents the expected value, Denotes the multi-scale dynamic discriminator D D Based on input The judgment results obtained are Denotes the image quality discriminator D c Based on input The judgment results obtained are Represents the face discriminator based on the input The judgment result of

[0072] In order to learn complex facial dynamics, a multi-scale dynamic discriminator D is added to the GAN framework. D During the training process, D D Accept a set of three consecutive real video frames Y t-K:t Or generate video frames These frames are compared with the optical flow W computed from the real video frames. t-K+1:t After being correlated, they are passed to D D Subsequently, the first generator G1 is encouraged to create forged frames that show the same flow (dynamics) as the corresponding real frames. Multi-scale dynamics discriminator D D Learn to distinguish the real t-K+1:t ,Y t-K:t ) and forged right,, Denotes the multi-scale dynamic discriminator D D The discriminant loss is expressed as:

[0073]

[0074] In addition, the image quality discriminator D c Aims to distinguish between real and synthetic image pairs. The goal is to maximize the probability of identifying real image pairs and minimize the probability of identifying synthetic image pairs. Therefore, the image quality discriminator D is defined c The discriminant loss is expressed as:

[0075]

[0076] Then the first discriminator loss L D1 =L DD +L Dc , where D D (W t-K+1:t ,Y t-K:t ) represents the multi-scale dynamic discriminator D D Based on the input (W t-K+1:t ,Y t-K:t ) The discrimination result obtained, D c (X t ,Y t ) represents the image quality discriminator D c Based on the input (X t ,Y t ) to obtain the discrimination result.

[0077] In order to improve the multi-scale dynamic discriminator D DThe optical flow model uses three optical flow models: VideoFlow, GMFlow+, or FlowFormer++ to improve the accuracy and efficiency of optical flow estimation. The specific implementation scheme is as follows:

[0078] VideoFlow is a three-frame optical flow estimation model that estimates the bidirectional optical flow of the central frame in a three-frame manner and introduces a motion transfer module to transfer motion information in adjacent three-frame modules to enhance the accuracy of optical flow estimation. The first frame is copied and stacked into the complete video frame, normalized and then input into the optical flow model to obtain an optical flow image sequence, which is then stacked with real frames or generated frames of different scales and input into the multi-scale dynamic discriminator D. D middle.

[0079] GMFlow+ is a two-frame optical flow estimation model that represents the optical flow estimation task as a global matching problem. It obtains the matching relationship by comparing feature similarities to output the optical flow. The normalized video frames are directly input into the optical flow model to obtain an optical flow image sequence, which is then stacked with real frames or generated frames of different scales and input into the multi-scale dynamic discriminator D. D middle.

[0080] FlowFormer++ is a two-frame optical flow estimation model. The cost-volume encoder is pre-trained through a new MAE scheme to enhance FlowFormer. The normalized video frames are directly input into the optical flow model to obtain the optical flow image sequence, which is stacked with real frames or generated frames of different scales and input into the multi-scale dynamic discriminator D. D middle.

[0081] In the embodiment, the texture adhesion improvement solution based on StyleUNet refers to constructing a texture adhesion improvement method framework based on the generative adversarial network architecture StyleUNet, such as Figure 4 As shown, it includes a second generator part and a second discriminator part, and the second generator part includes a mapping network G f , image encoder G e , global translation and rotation transformation module G r , and the generative network G2, where the convolution module in the generative network G2 is replaced with the non-aliasing convolution module of StyleGAN3, and the noise input in the generative network G2 is cancelled, so that the feature information comes entirely from the rough features of the previous layer, and can prevent the feature distortion caused by the noise input of each layer. At the same time, in the image encoder G e The output of each layer is connected to the aliasing-free convolution module of each layer in the generating network G2 by skip connection to form a residual structure, and the second discriminator part includes the discriminator D2.

[0082] Random Noise Z t After mapping network G fMapped to latent code W t , used to control the style and details of the generated video frames, the input neural rendering image X t The image encoder obtains multi-level image features. The image features of the middle level are connected to the generating network G2 through the residual structure. The image features of the last level are converted into Fourier features through Fourier transform and then input into the global translation and rotation transformation module. The global translation and rotation parameters are calculated and input into the generating network G f , in the generated network G f The video frame is generated by combining the global translation and rotation parameters and the intermediate level image features. The image X will be rendered by the neural t and the real video frame Y t The real image pair (X t ,Y t ) and generate image pairs Input to discriminator D2 to determine the authenticity of the video frame;

[0083] Specifically, the mapping network can adopt 2 fully connected layers, the generating network can include 7 layers, the sampling rate of each layer is determined by the minimum acceptable stop band frequency, the output resolution is the same as the sampling rate, and the minimum acceptable stop band frequency obeys a geometric series.

[0084] In order to achieve precise control of global continuous translation and rotation, the image features of the last level are converted into Fourier features through Fourier transform. The Fourier features are 64-dimensional feature vectors. Fourier features can map image features to an infinite-dimensional natural space, improving the representation ability of features. At the same time, the features of the lowest level can be passed all the way to the top level, which can better utilize image features to guide the rotation and translation of high-level features, thereby enhancing the consistency of the generated image.

[0085] In the global translation and rotation transformation module, the global translation and rotation parameters are obtained through the affine layer. Since the bottom layer features will be passed to the top layer, the overall rotation and translation performance can be continuously controlled. Specifically, the Fourier features are converted into a four-dimensional vector t = (r c ,r s ,t x ,t y ), the four-dimensional vector can be initially set to t = (1, 0, 0, 0), and it is allowed to change freely during the training process. The generated four-dimensional vector t is normalized to obtain the normalized result t′ = (r c ′,r s ′,t x ′,t′ y ):

[0086]

[0087] Among them, r c ′,r s ′ represents the center of the canvas [0,1] 2 The rotation parameter, the rotation angle α is defined as r c ′ = cosα and r s ′=sinα,t x ′,t′ y Indicated by (t x ′,t′ y ) is the unit of translation parameter, rotation parameter and translation parameter together constitute the global translation rotation parameter, and the geometric transformation is realized by modifying the phase and two-dimensional frequency of Fourier characteristics.

[0088] Discriminator D2 is used to distinguish real image pairs (X t ,Y t ) and generate image pairs Improve the quality of generated video frames. The discriminator D2 adopts a residual network architecture, which includes 7 downsampling layers. Each layer is connected to the upper layer features after downsampling, and finally outputs the score through the convolution layer and the fully connected layer to improve the quality of the generated video frames.

[0089] After training the texture adhesion improvement framework, the mapping network G contained in the trained second generator is extracted. f , image encoder G e , global translation and rotation transformation module G r , and the generative network G2 form a video generation model for the generation of digital human videos. The loss Ha number used in training includes the second generator loss L G2 and the second discriminator loss L D2 ;

[0090] Second generator loss in, Represents the VGG loss constructed based on different layers of visual features extracted by the VGG network for the generated video frame and the real video frame. The VGG loss can adopt LI distance or L2 distance. express The weight hyperparameter, L rec Represents the reconstruction loss between the generated video frame and the real video frame. The reconstruction loss can also use LI distance or L2 distance. rec Indicates L rec The weight hyperparameters are and λ adv Denote adversarial loss and adversarial loss respectively. The weight hyperparameters are It is expressed as:

[0091]

[0092] Among them, fake i represents the discrimination result given by the discriminator D2 for the input generated image pair i, and N represents the total number of image pairs;

[0093] The second discriminator loss L D2 It is expressed as:

[0094]

[0095] Among them, real i Represents the discrimination result given by the discriminator D2 for the input real image pair i.

[0096] In an embodiment, a video generation model is generated for each character, and the video generation model can be used to generate a self-playing digital human video. Specifically, a monocular video uploaded by a user is obtained, and the monocular video is subjected to key point extraction and neural rendering to obtain a neural rendering image sequence, and the neural rendering image sequence is input into the video generation model. After reasoning, a digital human self-playing video similar to the user is generated, retaining the user's facial expressions and body movements.

[0097] Of course, cross-identity re-enactment digital human video generation can also be performed. Identity conversion is performed between the monocular video uploaded by user A and the digital human model corresponding to user B, and a cross-identity digital human re-enactment video is inferred and generated, migrating the user's expressions and movements to digital human models of different identities.

[0098] Based on the same inventive concept, Figure 5 As shown, the embodiment also provides a digital human video generation device 50 based on monocular video replay, including a key point generation module 51, a neural rendering module 52, and a video generation module 53, wherein the key point generation module 51 is used to generate a key point sequence based on the monocular video; the neural rendering module 52 is used to generate a neural rendering image sequence based on the key point sequence; the video generation module 53 is used to generate a digital human video based on the neural rendering image sequence, which includes using an optical flow enhancement method based on a generative adversarial network and an autoregressive architecture to generate a digital human video, or using a texture adhesion improvement method that introduces an aliasing-free convolution module and a global translation and rotation transformation module of StyleGAN3 in a generative adversarial network architecture to generate a digital human video.

[0099] It should be noted that the digital human video generation device based on monocular video replay provided in the above embodiment should be illustrated by the division of the above functional modules when performing digital human video generation based on monocular video replay, and the above functional allocation can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the digital human video generation device based on monocular video replay provided in the above embodiment and the digital human video generation construction method embodiment based on monocular video replay belong to the same concept, and the specific implementation process is detailed in the digital human video generation method embodiment based on monocular video replay, which will not be repeated here.

[0100] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the method for generating a digital human video based on monocular video replay is implemented, specifically including the following steps:

[0101] S1, generates key point sequence based on monocular video;

[0102] S2, generates a neural rendering image sequence based on the key point sequence;

[0103] S3, Digital Human Video Generation from Neurally Rendered Image Sequences.

[0104] The computing device provided in the embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the digital human video generation method based on monocular video replay described in S1-S3 above. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0105] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating digital human video based on monocular video replay, characterized in that: The following steps are involved: Generate key point sequence based on monocular video; Generate a representation of a neural rendered image sequence based on a sequence of key points; Digital human videos are generated based on neural rendering image sequences, including generating digital human videos by using an optical flow enhancement method based on a generative adversarial network and an autoregressive architecture, or generating digital human videos by using a texture adhesion improvement method that introduces an aliasing-free convolution module and a global translation and rotation transformation module of StyleGAN3 into a generative adversarial network architecture.

2. The method for generating digital human video based on monocular video playback according to claim 1, characterized in that: The generating of a key point sequence based on a monocular video comprises: Key points are extracted from monocular video, and time-series smoothing and redirection processing is performed on the key points to obtain a key point sequence.

3. The method for generating digital human video based on monocular video playback according to claim 1, characterized in that: The method of generating a digital human video by using an optical flow enhancement method based on a generative adversarial network and an autoregressive architecture includes: The optical flow enhancement method framework is constructed based on the generative adversarial network, which includes a first generator part and a first discriminator part. The first generator part includes a first generator G1 and an optical flow model G o The first discriminator part includes an image quality discriminator D c , face discriminator D F , and the multi-scale dynamic discriminator D D , and establish autoregression between the output and input of the first generator, taking the generated video frames at the output as the historical generated video frames at the input, and the input neural rendering image sequence X t-K:t and history to generate video frames The current generated video frame is obtained through the first generator G1 The image X will be rendered by the neural t and the real video frame Y t The real image pair (X t ,Y t ) and generate image pairs Input to the image quality discriminator D c Determine the authenticity of the video frame and compare the real image corresponding to the facial area and generate image pairs Input to the face discriminator D F Determine whether the facial region is real or fake, input the real frame sequence Y t-K:t After the optical flow model G o Get the optical flow change W between video frames t-K:t-1 , the real image pair (W t-K:t-1 ,Y t-K:t ) and generate image pairs Input to the multi-scale dynamic discriminator D D Determine the authenticity of the video frame based on the change of optical flow, so that the dynamic changes of the generated video frame based on the change of optical flow are consistent with the dynamic changes of the real video frame; After training the optical flow enhancement framework, the trained first generator is extracted as a video generation model for generating digital human videos.

4. The method for generating digital human video based on monocular video playback according to claim 3, characterized in that: The loss function used to train the optical flow enhancement framework includes the first generator loss L G1 and the second discriminator loss L D1 ; First generator loss in Represents the VGG loss constructed based on different layers of visual features extracted by the VGG network for the generated video frames and the real video frames. express The weight hyperparameters are represents the feature matching loss of the discriminator features, which contains the real video frame Y t and generate video frames The difference between the features extracted by the image quality discriminator and the real video frame sequence Y t-K:t and generate a sequence of video frames The difference between the features extracted by the multi-scale dynamic discriminator, express The weight hyperparameters are Denotes the adversarial loss, expressed as: Among them, E t represents the expected value, Denotes the multi-scale dynamic discriminator D D Based on input The judgment results obtained are Denotes the image quality discriminator D c Based on input The judgment results obtained are Represents the face discriminator based on the input The judgment result of First discriminator loss in, Denotes the multi-scale dynamic discriminator D D The discriminant loss, Denotes the image quality discriminator D c The discriminant loss is expressed as: Among them, D D (W t-K+1:t ,Y t-K:t ) represents the multi-scale dynamic discriminator D D Based on the input (W t-K+1:t ,Y t-K:t ) The discrimination result obtained, D c (X t ,Y t ) represents the image quality discriminator D c Based on the input (X t ,Y t ) to obtain the discrimination result.

5. The method for generating digital human video based on monocular video playback according to claim 3, characterized in that: The optical flow model includes a VideoFlow model, a GMFlow+ model, or a FlowFormer++ model.

6. The method for generating digital human video based on monocular video playback according to claim 1, characterized in that: The method of generating a digital human video by introducing an aliasing-free convolution module and a texture adhesion improvement method of a global translation and rotation transformation module of StyleGAN3 into a generative adversarial network architecture includes: Based on the generative adversarial network architecture StyleUNet, a texture adhesion improvement framework is constructed, which includes a second generator part and a second discriminator part. The second generator part includes a mapping network G f , image encoder G e , global translation and rotation transformation module G r , and generate network G2, where the convolution module in the generated network G2 is replaced by the non-aliasing convolution module of StyleGAN3, and the noise input in the generated network G2 is cancelled. e The output of each layer is connected to the aliasing-free convolution module of each layer in the generating network G2 to form a residual structure. The second discriminator part includes the discriminator D2, the random noise Z t After mapping network G f Mapped to latent code W t , the input neural rendering image X t The image encoder obtains multi-level image features. The image features of the middle level are connected to the generating network G2 through the residual structure. The image features of the last level are converted into Fourier features through Fourier transform and then input into the global translation and rotation transformation module. The global translation and rotation parameters are calculated and input into the generating network G f , in the generated network G f The video frame is generated by combining the global translation and rotation parameters and the intermediate level image features. The image X will be rendered by the neural t and the real video frame Y t The real image pair (X t ,Y t ) and generate image pairs Input to discriminator D2 to determine the authenticity of the video frame; After training the texture adhesion improvement framework, the mapping network G contained in the trained second generator is extracted. f , image encoder G e , global translation and rotation transformation module G r , and the generation network G2 constitute a video generation model for generating digital human videos.

7. The method for generating digital human video based on monocular video playback according to claim 6, characterized in that: In the global translation and rotation transformation module, the global translation and rotation parameters are calculated based on the Fourier features, including: The Fourier features are converted into a four-dimensional vector t = (r c ,r s ,t x ,t y ), and normalize it to get the normalized result t ′ =(r c ′ ,r s ′ ,t x ′ ,t ′ y ): Among them, r c ′ ,r s ′ Represents the canvas center [0,1] 2 The rotation parameter, the rotation angle α is defined as r c ′ = cosα and r s ′ = sinα,t x ′ ,t ′ y Indicated by (t x ′ ,t ′ y ) as the unit, the rotation parameter and the translation parameter together constitute the global translation and rotation parameter.

8. The method for generating digital human video based on monocular video playback according to claim 6, characterized in that: The loss function used to train the texture adhesion improvement framework includes the second generator loss L G2 and the second discriminator loss L D2 ; Second generator loss in, Represents the VGG loss constructed based on different layers of visual features extracted by the VGG network for the generated video frames and the real video frames. express The weight hyperparameter, L rec represents the reconstruction loss between the generated video frame and the real video frame, λ rec Indicates L rec The weight hyperparameters are and λ adv Denote adversarial loss and adversarial loss respectively. The weight hyperparameters are It is expressed as: Among them, fake i represents the discrimination result given by the discriminator D2 for the input generated image pair i, and N represents the total number of image pairs; The second discriminator loss L D2 It is expressed as: Among them, real i Represents the discrimination result given by the discriminator D2 for the input real image pair i.

9. A digital human video generation device based on monocular video replay, characterized in that: include: A key point generation module, which is used to generate a key point sequence based on a monocular video; A neural rendering module, which is used to generate a neural rendering image sequence based on the key point sequence; A video generation module is used to generate digital human videos based on neural rendering image sequences, including generating digital human videos by using an optical flow enhancement method based on a generative adversarial network and an autoregressive architecture, or generating digital human videos by using a texture adhesion improvement method that introduces the aliasing-free convolution module and the global translation and rotation transformation module of StyleGAN3 into the generative adversarial network architecture.

10. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the digital human video generation method based on monocular video replay according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Unsupervised moving target detection method based on information reduction rate

    CN114494934A

  • Focus image generation method and device based on style generative adversarial network model

    CN116030158A

  • Face image generation method and apparatus, device, and storage medium

    US20210241521A1

Cited By

  • Single-phase flow field real-time synchronous measurement method and system

    CN120598909A