Digital human video generation method and system based on monocular video replay

Abstract

The application discloses a kind of digital person video generation method and device based on monocular video replay, comprising: generating key point sequence based on monocular video;Generate a sequence of neural rendering images based on the key point sequence;Digital person video is generated based on the sequence of neural rendering images, which includes using a light flow enhancement method based on a generative adversarial network and an autoregressive architecture to generate digital person video, or using a texture adhesion improvement method that introduces a non-aliasing convolution module and a global translation rotation transformation module of StyleGAN3 in the generative adversarial network architecture to generate digital person video.This can improve the temporal continuity, controllability and authenticity of the generated video.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of monocular video replay, and particularly relates to a digital human video generation method and system based on monocular video replay. BACKGROUND

[0002] With the development of artificial intelligence and computer graphics, virtual digital humans gradually enter the public view and are widely used in various fields, such as intelligent customer service, virtual anchors, online education, game entertainment, etc. In human-computer interaction, virtual digital humans can quickly establish contact with users and improve user experience.

[0003] Virtual digital human technology consists of four parts: modeling, driving, rendering, and interaction. Modeling means creating the specific form of a digital human, driving means using captured or generated data to change the representation of the target digital human, rendering means converting various forms of digital humans into visible images, and interaction means the interaction between digital humans and the environment or objects, involving physics, emotions, speech, and actions.

[0004] Traditional digital human production methods require modeling and driving, which is a complex and time-consuming process, and is not suitable for large-scale digital human production needs, hindering the popularization and application of digital humans. Therefore, using generative artificial intelligence technology to generate high-fidelity and large-scale virtual digital humans has become a current research hotspot. For example, patent application CN 118196243 A discloses a digital human video generation method and system, which generates multiple digital human images based on a face rendering model after obtaining expression data and head pose data, and merges the digital human images into a video. For another example, patent application CN 117593683 A discloses a video generation algorithm and system for 2D digital humans, which trains a neural radiance field model based on a video generation task after image segmentation and key point extraction, and inputs real-time collected model body images into the trained neural radiance field model to generate a video.

[0005] The generative digital human technology based on monocular video replay aims to use monocular video as input to generate a digital human video consistent with the expression, head movement, and body posture of the driving video, while maintaining the identity of the target person, which can reduce the development difficulty of digital human-related applications and reduce the workload of artificial modeling. However, the current generative digital human technology is still in its infancy, and there are many deficiencies in the authenticity, controllability, and temporal continuity of the generated digital human video. SUMMARY

[0006] In view of the above, the present application aims to provide a digital human video generation method and system based on monocular video replay to improve the temporal continuity, controllability, and authenticity of the generated video.

[0007] To achieve the above-mentioned object of the invention, an embodiment provides a method for generating a digital human video based on monocular video playback, comprising the following steps:

[0008] Generate key point sequence based on monocular video;

[0009] Generate a representation of a neural rendered image sequence based on a sequence of keypoints;

[0010] Digital human videos are generated based on neural rendered image sequences, including generating digital human videos using an optical flow enhancement method based on a generative adversarial network and an autoregressive architecture, or generating digital human videos using a texture adhesion improvement method that introduces the aliasing-free convolution module and the global translation and rotation transformation module of StyleGAN3 into the generative adversarial network architecture.

[0011] Preferably, generating a key point sequence based on a monocular video includes:

[0012] Key points are extracted from monocular video and temporally smoothed and redirected to obtain a key point sequence.

[0013] Preferably, the method of generating a digital human video by using an optical flow enhancement method based on a generative adversarial network and an autoregressive architecture includes:

[0014] The optical flow enhancement method framework is constructed based on the generative adversarial network, which includes the first generator part and the first discriminator part. The first generator part includes the first generator G1 and the optical flow model G o , the first discriminator part includes an image quality discriminator D c , face discriminator D F , and the multi-scale dynamic discriminator D D , and establish autoregression between the output and input of the first generator, taking the generated video frame at the output as the historical generated video frame at the input, and the input neural rendering image sequence X t-K:t and history to generate video frames The current generated video frame is obtained through the first generator G1 The image X will be rendered by the neural t and the real video frame Y t The real image pair (X t ,Y t ) and generate image pairs Input to the image quality discriminator D c Determine the authenticity of the video frame and compare the real image corresponding to the facial area and generate image pairs Input to face discriminator D F Determine whether the facial region is real or fake, input real frame sequence Y t-K:t After the optical flow model Go Get the optical flow change W between video frames t-K:t-1 , the real image pair (W t-K:t-1 ,Y t-K:t ) and generate image pairs Input to the multi-scale dynamic discriminator D D Determine the authenticity of video frames based on optical flow changes, so that the dynamic changes of the generated video frames based on optical flow changes are consistent with the dynamic changes of the real video frames;

[0015] After training the optical flow enhancement framework, the trained first generator is extracted as a video generation model for generating digital human videos.

[0016] Preferably, the loss function used to train the optical flow enhancement framework includes the first generator loss L G1 and the second discriminator loss L D1 ;

[0017] First generator loss in Represents the VGG loss constructed based on different layers of visual features extracted by the VGG network for the generated video frames and the real video frames. express The weight hyperparameters, represents the feature matching loss of the discriminator features, which contains the real video frame Y t and generate video frames The difference between the features extracted by the image quality discriminator and the real video frame sequence Y t-K:t and generate a sequence of video frames The difference between the features extracted by the multi-scale dynamic discriminator, express The weight hyperparameters, Denotes the adversarial loss, expressed as:

[0018]

[0019] Among them, E t Indicates the expected value, Denotes the multi-scale dynamic discriminator D D Based on input The obtained discrimination results, Denotes the image quality discriminator D c Based on input The obtained discrimination results, Represents the face discriminator based on the input The judgment result of

[0020] First discriminator loss in, denotes a discriminative loss of the multi-scale dynamic discriminator D D , denotes a discriminative loss of the image quality discriminator D c , denoted as:

[0021]

[0022] wherein D D (W t-K+1:t ,Y t-K:t ) denotes a discriminative result of the multi-scale dynamic discriminator D D based on the input (W t-K+1:t ,Y t-K:t ), D c (X t ,Y t ) denotes a discriminative result of the image quality discriminator D c based on the input (X t ,Y t ).

[0023] Preferably, the optical flow model comprises a VideoFlow model, a GMFlow+ model, or a FlowFormer++ model.

[0024] Preferably, the texture adhesion improvement method employing the non-aliasing convolution module and the global translation-rotation transformation module of StyleGAN3 introduced in the generative adversarial network architecture is used to generate the digital human video, comprising:

[0025] The texture adhesion improvement method framework is constructed based on the generative adversarial network architecture StyleUNet, which comprises a second generator part and a second discriminator part, the second generator part comprises a mapping network G f , an image encoder G e , a global translation-rotation transformation module G r , and a generation network G2, wherein the convolution module in the generation network G2 is replaced by the non-aliasing convolution module of StyleGAN3, and the noise input in the generation network G2 is cancelled, while a residual structure is formed by constructing a skip connection between the output of each layer of the image encoder G e and each non-aliasing convolution module in the generation network G2, and the second discriminator part comprises a discriminator D2, and the random noise Z t is mapped into a latent code W f by the mapping network G t , and the input neural rendering image X tThe multi-level image features are obtained through the image encoder, the image features of the intermediate levels are connected to the generation network G2 through the residual structure, and the image features of the last level are converted into Fourier features through Fourier transform and then input into the global translation and rotation transformation module, the global translation and rotation parameters are calculated and input into the generation network G f The generation network G f combines the global translation and rotation parameters and the image features of the intermediate levels to calculate the generated video frame The real image pair (X t , Y t ) composed of the neural rendering image X t and the real video frame Y t and the generated image pair are input into the discriminator D2 to determine the authenticity of the video frame.

[0026] After training the texture adhesion improvement method framework, the mapping network G f , the image encoder G e , the global translation and rotation transformation module G r , and the generation network G2 contained in the second generator part are extracted to form a video generation model for the generation of digital human videos.

[0027] Preferably, in the global translation and rotation transformation module, the global translation and rotation parameters are calculated based on the Fourier features, including:

[0028] The Fourier features are converted into a four-dimensional vector t=(r c ,r s ,t x ,t y ) through an affine layer, and normalized to obtain a normalized result t′=(r c ′,r s ′,t x ′,t′ y ):

[0029]

[0030] wherein r c ′,r s ′ represent the rotation parameters around the center [0,1] 2 of the canvas, the rotation angle α is defined as r c ′=cosα and r s ′=sinα, t x ′,t′ y represent the translation parameters in units of (t x ′,t′ y ), and the rotation parameters and the translation parameters jointly constitute the global translation and rotation parameters.

[0031] Preferably, the loss function used for training the texture adhesion improvement framework comprises a second generator loss L G2 and a second discriminator loss L D2 ;

[0032] The second generator loss L wherein, represents a VGG loss constructed based on visual features extracted from different layers of a VGG network for generated video frames and real video frames, represents a weight hyperparameter of L rec represents a reconstruction loss between generated video frames and real video frames, rec represents a weight hyperparameter of L rec ; and λ adv respectively represent an adversarial loss and a weight hyperparameter of the adversarial loss ; is represented as:

[0033]

[0034] wherein, fake i represents a discrimination result given by the discriminator D2 for an input generated image pair i, and N represents a total amount of image pairs;

[0035] The second discriminator loss L D2 is represented as:

[0036]

[0037] wherein, real i represents a discrimination result given by the discriminator D2 for an input real image pair i.

[0038] To achieve the above-mentioned purposes, the embodiments of the present application further provide a digital human video generation device based on monocular video replay, comprising:

[0039] A key point generation module for generating a key point sequence based on a monocular video;

[0040] A neural rendering module for generating a neural rendering image sequence based on the key point sequence;

[0041] A video generation module for generating a digital human video based on the neural rendering image sequence, which comprises using a flow enhancement method based on a generative adversarial network and an autoregressive architecture to generate a digital human video, or using a texture adhesion improvement method in which a non-aliasing convolution module and a global translation rotation transformation module of StyleGAN3 are introduced into a generative adversarial network architecture to generate a digital human video.

[0042] To achieve the above-mentioned object of the application, the embodiment further provides a computing device comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the above-mentioned digital human video generation method based on monocular video replay.

[0043] Compared with the prior art, the application has at least the following beneficial effects:

[0044] After extracting the key point sequence and generating the neural rendering image sequence based on the key point sequence, the digital human video is generated based on the neural rendering image sequence. Specifically, in the video generation, the optical flow enhancement method based on the generative adversarial network and the autoregressive architecture is adopted to ensure the time sequence continuity and authenticity of the generated video. The texture adhesion improvement method of introducing the non-aliasing convolution module and the global translation rotation transformation module of StyleGAN3 in the generative adversarial network architecture can realize the controllability and authenticity of the generated video. Both methods can generate high-fidelity digital human videos consistent with the input video expressions and actions. This technology reduces the development difficulty of digital human related applications and reduces the workload of artificial modeling, and has a wide application prospect in virtual reality, digital entertainment, video games, education and finance. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief descriptions will be given below for the drawings needed to be used in the embodiments or prior art descriptions. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0046] Figure 1 is a flowchart of the digital human video generation method based on monocular video replay provided by the embodiment;

[0047] Figure 2 is a flowchart of the key point sequence generation based on monocular video provided by the embodiment;

[0048] Figure 3 is a structural schematic diagram of the optical flow enhancement method framework provided by the embodiment;

[0049] Figure 4 is a structural schematic diagram of the texture adhesion improvement method framework provided by the embodiment;

[0050] Figure 5 is a structural schematic diagram of the digital human video generation device based on monocular video replay provided by the embodiment. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the protection scope of the present application.

[0052] The technical concept of the present application is that the technical solution of digital person video based on monocular video replay provided by the embodiments of the present application aims to realize target digital person half-body video generation based on monocular video, mainly including three stages: the first stage is to generate a key point sequence containing a face and a half-body pose according to a monocular video containing an upper body of a person; the second stage is to generate a neural rendering image sequence representing the semantics of the face and the upper body pose according to the key point sequence obtained in the first stage; and the third stage is to generate a video sequence consistent with the target person's identity and consistent with the driving video expression action according to the neural rendering image sequence.

[0053] As shown in Figure 1 , the method for generating digital person video based on monocular video replay provided by the embodiments includes the following steps:

[0054] S1, generating a key point sequence based on a monocular video.

[0055] In the embodiments, according to an input N-frame monocular video sequence V 1:N containing an upper body of a person, a target person's N-frame key point sequence L 1:N containing a face and an upper body pose is generated. i Wherein L 528×3 represents the three-dimensional coordinates of the i-th frame key point, containing 478 face key points, 8 torso key points and 21 key points per hand. As shown in Figure 2 , the key point sequence generation includes three processes of key point extraction, time sequence smoothing and reorientation.

[0056] For key point extraction, MediaPipe and DWPose are used to extract the key points of the face, body and hands of the person in the monocular video. Among them, MediaPipe is used to extract 478 face key points, and DWPose is used to extract 33 body key points and 21 hand key points per hand.

[0057] For time sequence smoothing, after key point extraction, there will be key point missing and jitter problems. By using the context of the video frame, the missing key points are completed by using linear interpolation, and the face, body and hand key points are respectively processed by using exponential weighted average algorithm for time sequence smoothing, so as to smooth the small amplitude jitter while retaining the original data samples.

[0058] For the re-targeting, Procrustes analysis is adopted to re-target the keypoint sequence after time smoothing, to re-target the expression and pose of the source character into the target character. Specifically, the average keypoint of the face, body and hand of the source character and the target character are calculated respectively as the reference keypoint matrix, and then the transformation matrix of the reference keypoint matrix of the source character to the reference keypoint matrix of the target character is solved, which includes translation, rotation and scaling matrix. Finally, the transformation matrix is applied to the keypoint sequence of the face, hand and body of the source character respectively to obtain the re-targeted keypoint sequence.

[0059] S2, generating a neural rendering image sequence representing the keypoint sequence.

[0060] In the embodiment, N frames of keypoint sequence L 1:N As input, N frames of neural rendering image sequence X 1:N representing the semantic information of facial expression, head movement and body pose are generated.

[0061] Wherein, the CCBR image is represented as an 8-bit RGB image, which is used to represent the semantic information of facial and body pose. For the body pose keypoint, each keypoint is drawn as a circle with a fixed radius, and each keypoint is given a predefined color, wherein the values of red and green channels are obtained by normalized UV coordinates, and the blue channel adopts a predefined fixed value. For facial keypoint, according to the normalized coordinates of the keypoint, rendering is performed by nvdiffrast. In addition, since increasing the number of skeleton points can improve the performance of reenactment, the embodiment interpolates 8 body skeleton points, wherein the number and color of interpolation between the joints are fixed. After interpolation, a total of 79 body pose key points are obtained.

[0062] The eye gaze image is represented as an 8-bit RGB image, which is used to control the finer eye features. According to the 5 eye key points of the eyeball up, down, left, right and center, the pupil center position and pupil radius are obtained, a red circle with the pupil center as the origin and the pupil radius as the radius is drawn, and the eye edge points are connected and filled with white, thereby generating the eye gaze image.

[0063] S3, generating a digital human video based on the neural rendering image sequence.

[0064] In the embodiment, according to N frames of neural rendering image sequence X 1:N , N frames of video sequence Y of the target character half body region consistent with the facial expression, head movement and body pose represented by the neural rendering image are generated1:N .

[0065] To improve the temporal continuity and quality of the generated video, the present application explores two generation paradigms of "generative adversarial network + autoregressive" and "only generative adversarial network", and respectively puts forward the corresponding improvement scheme. On the one hand, the present application designs and implements an optical flow enhancement scheme based on generative adversarial network and autoregressive architecture, introduces and transforms the optical flow model in the multi-scale dynamic discriminator, and improves the temporal continuity of the generated video; on the other hand, the present application designs and implements a texture adhesion improvement scheme based on StyleUNet, only uses the generative adversarial network architecture, and through the introduction of the aliasing-free convolution module and global translation rotation transformation module of StyleGAN3, improves the texture adhesion problem and improves the quality of the generated video.

[0066] In the embodiment, the optical flow enhancement scheme based on the generative adversarial network and the autoregressive architecture is designed based on the generative adversarial network (GAN) and the autoregressive architecture, and the optical flow information is enhanced through the collaborative training of the generator and the discriminator. The generator is responsible for generating enhanced optical flow information to improve the quality and continuity of the video frames, while the discriminator is used to evaluate the authenticity of the generated optical flow information, guiding the generator to learn more accurate optical flow information. In the experimental design and evaluation, the performance of different optical flow models is compared, and the optimal optical flow model is selected for optical flow enhancement.

[0067] Specifically, as shown in Figure 3 , the optical flow enhancement method framework is constructed based on the generative adversarial network, which includes a first generator part and a first discriminator part, the first generator part includes a first generator G1 and an optical flow model G o , the first discriminator part includes an image quality discriminator D c , a face discriminator D F , and a multi-scale dynamic discriminator D D , and an autoregressive is established between the output and the input of the first generator, the generated video frame at the output end is taken as the historical generated video frame at the input end, the input neural rendering image sequence X t-K:t and the historical generated video frame The current generated video frame is obtained by the first generator G1. t The real image pair (X t , Y t ) composed of the neural rendering image X t and the real video frame Y c and the generated image pair are input to the image quality discriminator D c to discriminate the authenticity of the video frame, and the real image pair and the generated image pair corresponding to the face area are input to the face discriminator DF discriminate the authenticity of the face region, the input real frame sequence Y t-K:t passing through the optical flow model G o obtain the optical flow change W between video frames t-K:t-1 , input the real image pair (W t-K:t-1 , Y t-K:t ) and the generated image pair to the multi-scale dynamic discriminator D D discriminate the authenticity of the video frame based on the optical flow change, so that the dynamic change based on the optical flow change of the generated video frame is consistent with the dynamic change of the real video frame;

[0068] After training the optical flow enhancement mode framework, the first generator after training is extracted as a video generation model for digital human video generation. The loss function used during training includes the first generator loss L G1 and the second discriminator loss L D1 ;

[0069] The first generator loss , wherein represents the VGG loss constructed based on the visual features of different layers of the generated video frame and the real video frame extracted based on the VGG network, wherein the VGG loss can adopt the LI distance, represents a weight hyperparameter of represents the feature matching loss of the discriminator features, which includes the difference between the image quality discriminator extracted from the real video frame Y t and the generated video frame , and the difference between the real video frame sequence Y t-K:t and the generated video frame sequence extracted from the multi-scale dynamic discriminator, represents a weight hyperparameter of represents the adversarial loss, which marks the generated false samples as 1 and marks the real samples as 0, represents

[0070]

[0071] wherein, E t represents the expected value, represents the discrimination result obtained by the multi-scale dynamic discriminator D D based on the input , represents the discrimination result obtained by the image quality discriminator D c based on the input , represents the discrimination result of the face discriminator based on the input ;

[0072] In order to learn complex facial dynamics, a multi-scale dynamic discriminator D is added to the GAN framework. D During the training process, D D Accepts a set of three consecutive real video frames Y t-K:t Or generate video frames These frames are compared with the optical flow W calculated from the real video frames. t-K+1:t After being correlated, they are passed to D D Subsequently, the first generator G1 is encouraged to create forged frames that show the same flow (dynamics) as the corresponding real frames. Multi-scale dynamics discriminator D D Learn to distinguish the real t-K+1:t ,Y t-K:t ) and forged right,, Denotes the multi-scale dynamic discriminator D D The discriminant loss is expressed as:

[0073]

[0074] In addition, the image quality discriminator D c Aims to distinguish between real and synthetic image pairs. Its goal is to maximize the probability of identifying real image pairs and minimize the probability of identifying synthetic image pairs. Therefore, the image quality discriminator D is defined c The discriminant loss is expressed as:

[0075]

[0076] Then the first discriminator loss L D1 =L DD +L Dc , where D D (W t-K+1:t ,Y t-K:t ) represents the multi-scale dynamic discriminator D D Based on the input (W t-K+1:t ,Y t-K:t ) The discrimination result obtained, D c (X t ,Y t ) represents the image quality discriminator D c Based on the input (X t ,Y t ) to obtain the discrimination result.

[0077] In order to improve the multi-scale dynamic discriminator D DThe discriminative ability of the model is improved, and the quality of the replay is improved. The optical flow model adopts VideoFlow, GMFlow+, or FlowFormer++, three optical flow models, to improve the accuracy and efficiency of optical flow estimation. The specific implementation scheme is as follows:

[0078] VideoFlow is a three-frame optical flow estimation model that estimates the bidirectional optical flow of the center frame in a three-frame manner and introduces a motion transfer module to transfer motion information between adjacent three-frame modules to enhance the accuracy of optical flow estimation. The first frame is copied and stacked into a complete video frame, and after normalization, the optical flow model is input to obtain an optical flow image sequence, which is stacked with real frames or generated frames of different scales to input a multi-scale dynamic discriminator D D .

[0079] GMFlow+ is a two-frame optical flow estimation model that represents the optical flow estimation task as a global matching problem and outputs the matching relationship by comparing feature similarity. The normalized video frame is directly input into the optical flow model to obtain an optical flow image sequence, which is stacked with real frames or generated frames of different scales to input a multi-scale dynamic discriminator D D .

[0080] FlowFormer++ is a two-frame optical flow estimation model that pre-trains the cost-volume encoder using a new MAE scheme to enhance FlowFormer. The normalized video frame is directly input into the optical flow model to obtain an optical flow image sequence, which is stacked with real frames or generated frames of different scales to input a multi-scale dynamic discriminator D D .

[0081] In the embodiment, the texture adhesion improvement scheme based on StyleUNet refers to a framework for improving texture adhesion based on the StyleUNet architecture of the generative adversarial network, as shown in Figure 4 , which includes a second generator part and a second discriminator part. The second generator part includes a mapping network G f , an image encoder G e , a global translation and rotation transformation module G r , and a generation network G2. In the generation network G2, the convolution module is replaced with the aliasing-free convolution module of StyleGAN3, and the noise input in the generation network G2 is removed. In this way, the feature information comes entirely from the coarse features of the previous layer, and the noise input in each layer can prevent the features from being distorted. At the same time, a residual structure is formed by constructing a skip connection between the output of each layer of the image encoder G e and the aliasing-free convolution module in each layer of the generation network G2, and the second discriminator part includes a discriminator D2.

[0082] Random noise Z t is input into the mapping network G fMapping to latent code W t , for controlling the style and details of the generated video frame, the input neural rendered image X t is fed into an image encoder to obtain multi-level image features, the image features of the intermediate levels are connected to the generation network G2 through a residual structure, and the image features of the last level are converted into Fourier features through Fourier transform and then input into a global translation and rotation transformation module to calculate the global translation and rotation parameters and input them into the generation network G f , in which the global translation and rotation parameters and the image features of the intermediate levels are combined to calculate the generated video frame f The real image pair (X t , Y t ) composed of the neural rendered image X t and the real video frame Y t and the generated image pair are input into the discriminator D2 to determine the authenticity of the video frame.

[0083] Specifically, the mapping network can adopt two fully connected layers, and the generation network can include 7 layers, the sampling rate of each layer is determined by the minimum acceptable stop band frequency, the output resolution is the same as the sampling rate, and the minimum acceptable stop band frequency obeys a geometric progression.

[0084] To achieve accurate control of global continuous translation and rotation, the image features of the last level are converted into Fourier features through Fourier transform, the Fourier features are 64-dimensional feature vectors, the Fourier features can map the image features to an infinite-dimensional natural space, improving the representation ability of the features; at the same time, the features of the lowest level can be transmitted to the highest level, which can better utilize the image features to guide the rotation and translation of the high-level features, and enhance the consistency of the generated image.

[0085] In the global translation and rotation transformation module, the global translation and rotation parameters are obtained through an affine layer, since the bottom-level features are transmitted to the top layer, the overall rotation and translation performance can be continuously controlled. Specifically, the Fourier features are converted into a four-dimensional vector t = (r c , r s , t x , t y ), which can be initially set to t = (1, 0, 0, 0) and allowed to change freely during the training process, and the generated four-dimensional vector t is normalized to obtain the normalized result t' = (r c ', r s ', t x ', t' y ):

[0086] ​

[0087] where r c ′,r s ′ represents the rotation parameter around the center [0, 1] of the canvas, and the rotation angle a is defined as r 2 ′ = cos a and r c ′ = sin a, t s ′,t′ x ′,t′ y represent the translation parameter in the unit of (t x ′,t′ y ), the rotation parameter and the translation parameter jointly constitute the global translation-rotation parameter, and the geometric transformation is realized by modifying the phase and two-dimensional frequency of the Fourier feature.

[0088] The discriminator D2 is used to distinguish the real image pair (X t ,Y t ) and the generated image pair to improve the quality of the generated video frame. The discriminator D2 adopts a residual network architecture as a whole, and contains 7 layers of down-sampling layers. Each layer is connected to the feature of the upper layer after down-sampling, and finally outputs the score through the convolution layer and the fully connected layer, which is used to improve the quality of the generated video frame.

[0089] After training the texture adhesion improvement mode framework, the mapping network G f , the image encoder G e , the global translation-rotation transformation module G r , and the generation network G2 contained in the trained second generator are extracted to constitute a video generation model, which is used for digital human video generation. The loss function used in training includes the second generator loss L G2 and the second discriminator loss L D2 ;

[0090] The second generator loss L wherein, represents the VGG loss constructed based on the visual features of different layers of the VGG network extracted from the generated video frame and the real video frame. The VGG loss can adopt LI distance or L2 distance, represents the weight hyperparameter of , L rec represents the reconstruction loss between the generated video frame and the real video frame. The reconstruction loss can also adopt LI distance or L2 distance, and λ rec represents the weight hyperparameter of L rec , λ adv and λ respectively represent the weight hyperparameters of the adversarial loss and the adversarial loss , represents

[0091]

[0092] wherein fake i represents the discrimination result given by the discriminator D2 to the input real image pair i.

[0093] The second discriminator loss L D2 is represented as:

[0094]

[0095] wherein real i represents the discrimination result given by the discriminator D2 to the input real image pair i.

[0096] In the embodiment, a video generation model is generated for each digital person, which can be used for the generation of a digital person video of self-reenactment. Specifically, a monocular video uploaded by a user is obtained, and after key point extraction and neural rendering, a sequence of neural rendered images is obtained. The sequence of neural rendered images is input into the video generation model to generate a digital person video of self-reenactment similar to the user, which retains the facial expression and body movement of the user.

[0097] Of course, cross-identity digital person video generation can also be performed. Identity conversion is performed between the monocular video uploaded by the user A and the digital person model corresponding to the user B, and a cross-identity digital person video of self-reenactment is generated, which migrates the expression and movement of the user to the digital person model of different identity.

[0098] Based on the same inventive concept, as Figure 5 shown, the embodiment also provides a digital person video generation apparatus 50 based on monocular video reenactment, which comprises a key point generation module 51, a neural rendering module 52, and a video generation module 53. The key point generation module 51 is configured to generate a sequence of key points based on a monocular video. The neural rendering module 52 is configured to generate a sequence of neural rendered images based on the sequence of key points. The video generation module 53 is configured to generate a digital person video based on the sequence of neural rendered images, which comprises a light flow enhancement method based on a generative adversarial network and a self-recurrent architecture to generate a digital person video, or a texture adhesion improvement method of introducing a non-aliasing convolution module and a global translation rotation transformation module of StyleGAN3 in a generative adversarial network architecture to generate a digital person video.

[0099] It should be noted that the digital human video generation device based on monocular video replay provided in the above embodiment should be divided into the above functional modules when performing digital human video generation based on monocular video replay, and the above functions can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the digital human video generation device based on monocular video replay and the digital human video generation construction method embodiment provided in the above embodiment belong to the same concept, and the specific implementation process is described in detail in the digital human video generation method embodiment, which will not be repeated here.

[0100] Based on the same inventive concept, the embodiment also provides a computing device comprising a memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the above-mentioned digital human video generation method based on monocular video replay, specifically comprising the following steps:

[0101] S1, generating a key point sequence based on monocular video;

[0102] S2, generating a neural rendering image sequence based on the key point sequence;

[0103] S3, generating a digital human video based on the neural rendering image sequence.

[0104] The computing device provided by the embodiment, in addition to including a processor and a memory in the hardware layer, also includes an internal bus, a network interface, a memory and other hardware required by the business. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the digital human video generation method based on monocular video replay described in S1-S3. Of course, in addition to the software implementation, the present application does not exclude other implementation manners, such as logic devices or software and hardware combined manner, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but also can be hardware or logic device.

[0105] The specific embodiments described above explain the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modification, supplement and equivalent replacement within the principle range of the present application should be included in the protection scope of the present application.

Claims

1. A digital human video generation method based on monocular video replay, characterized in that, The method comprises the following steps: generating a key point sequence based on a monocular video; generating a neural rendering image sequence based on the key point sequence; The invention generates digital human videos based on neural rendering image sequences, which includes adopting an optical flow enhancement method based on a generative adversarial network and an autoregressive architecture to generate digital human videos, including: constructing an optical flow enhancement method framework based on a generative adversarial network, which includes a first generator part and a first discriminator part, wherein the first generator part includes a first generator G1 and an optical flow model G o , the first discriminator part includes an image quality discriminator D c , face discriminator D F , and the multi-scale dynamic discriminator D D , and establish autoregression between the output and input of the first generator, taking the generated video frame at the output as the historical generated video frame at the input, and the input neural rendering image sequence X t-K:t and history to generate video frames The current generated video frame is obtained through the first generator G1 The image X will be rendered by the neural t and the real video frame Y t The real image pair (X t ,Y t ) and generate image pairs Input to the image quality discriminator D c Determine the authenticity of the video frame and compare the real image corresponding to the facial area and generate image pairs Input to face discriminator D F Determine whether the facial region is real or fake, input real frame sequence Y t-K:t After the optical flow model G o Get the optical flow change W between video frames t-K:t-1 , the real image pair (W t-K:t-1 ,Y t-K:t ) and generate image pairs Input to the multi-scale dynamic discriminator D D The authenticity of video frames is determined based on optical flow changes, so that the dynamic changes of the generated video frames based on optical flow changes are consistent with the dynamic changes of the real video frames. After training the optical flow enhancement method framework, the trained first generator is extracted as the video generation model for the generation of digital human videos. Or a texture adhesion improvement method that introduces the non-aliasing convolution module and global translation and rotation transformation module of StyleGAN3 into the generative adversarial network architecture is used to generate digital human videos, including: constructing a texture adhesion improvement method framework based on the generative adversarial network architecture StyleUNet, which includes a second generator part and a second discriminator part, and the second generator part includes a mapping network G f , image encoder G e , global translation and rotation transformation module G r , and generate network G2, where the convolution module in the generation network G2 is replaced by the non-aliasing convolution module of StyleGAN3, and the noise input in the generation network G2 is cancelled. e The output of each layer is connected to the aliasing-free convolution module of each layer in the generative network G2 to form a skip connection to form a residual structure. The second discriminator part includes the discriminator D2, the random noise Z t After mapping network G f Mapping to latent code W t , the input neural rendered image X t The multi-level image features are obtained through the image encoder. The image features of the middle level are connected to the generation network G2 through the residual structure. The image features of the last level are converted into Fourier features through Fourier transform and then input into the global translation and rotation transformation module. The global translation and rotation parameters are calculated and input into the generation network G f , in the generated network G f The video frame is generated by combining the global translation and rotation parameters and the intermediate level image features. The image X will be rendered by the neural t and the real video frame Y t The real image pair (X t ,Y t ) and generate image pairs Input to the discriminator D2 to determine the authenticity of the video frame; after training the texture adhesion improvement framework, extract the mapping network G contained in the trained second generator part f , image encoder G e , global translation and rotation transformation module G r , and the generative network G2 constitute a video generation model for generating digital human videos.

2. The digital human video generation method based on monocular video replay according to claim 1, wherein, The generating of the key point sequence based on the monocular video comprises: extracting key points from the monocular video, and performing time sequence smoothing processing and redirection processing on the key points to obtain the key point sequence.

3. The digital human video generation method based on monocular video replay according to claim 1, wherein, The loss function used for training the optical flow enhancement mode framework includes a first generator loss L G1 and a second discriminator loss L D1 ; The first generator loss wherein represents a VGG loss constructed based on visual features extracted from different layers of a VGG network for the generated video frame and the real video frame, represents a weight hyperparameter of represents a feature matching loss of the discriminator features, which includes the real video frame Y t and the generated video frame the difference between the features extracted by the image quality discriminator, and the real video frame sequence Y t-K:t and the generated video frame sequence the difference between the features extracted by the multi-scale dynamic discriminator, represents a weight hyperparameter of represents an adversarial loss, which is represented as: wherein E t denotes an expected value, denotes a multi-scale dynamic discriminator D D based on the input the obtained discrimination result, denotes an image quality discriminator D c based on the input the obtained discrimination result, denotes a discrimination result based on the input by the face discriminator; First discriminator loss wherein, denotes the discriminative loss of the multi-scale dynamic discriminator D D denotes the discriminative loss of the image quality discriminator D c is given by:​ where D D (W t-K+1:t ,Y t-K:t ) represents a multi-scale dynamic discriminator D D based on the input (W t-K+1:t ,Y t-K:t ), D c (X t ,Y t ) represents an image quality discriminator D c based on the input (X t ,Y t ).

4. The digital human video generation method based on monocular video replay according to claim 1, wherein, The optical flow model comprises a VideoFlow model, a GMFlow+ model, or a FlowFormer++ model.

5. The monocular video replay based digital human video generation method of claim 1, wherein, In the global translation rotation transformation module, global translation rotation parameters are calculated based on Fourier features, comprising: The Fourier features are converted into a four-dimensional vector t = (r c ,r s ,t x ,t y ), and perform normalization to obtain the normalized result t ′ =(r c ′ ,r s ′ ,t x ′ ,t ′ y ): where r c ′ , s ′ denotes the rotation parameter around the canvas center [0, 1] 2 , the rotation angle a is defined as r c ′ = cos a and r s ′ = sin a, t x ′ , t ′ y denotes the translation parameter in (t x ′ , t ′ y ) units, the rotation parameter and the translation parameter together form the global translation-rotation parameter.

6. The monocular video replay based digital human video generation method of claim 1, wherein, The loss function used to train the texture adhesion improvement manner framework includes a second generator loss L G2 and a second discriminator loss L D2 ; Second generator loss wherein, represents a VGG loss constructed based on visual features extracted from different layers of a VGG network for the generated video frame and the real video frame, represents a weight hyperparameter of L rec represents a reconstruction loss between the generated video frame and the real video frame, λ rec represents L rec a weight hyperparameter of and λ adv respectively represent an adversarial loss and a weight hyperparameter of the adversarial loss represents:​ wherein fake i denotes the discrimination result given by the discriminator D2 for the input generated image pair i, and N denotes the total number of image pairs. Second discriminator loss L D2 is represented as: where real i denotes the discriminative result given by the discriminator D2 for the input real image pair i.

7. A digital human video generation apparatus based on monocular video replay, characterized by, comprising: a key point generation module configured to generate a key point sequence based on a monocular video; a neural rendering module configured to generate a neural rendering image sequence based on the key point sequence; The video generation module is used for generating a digital human video based on a neural rendering image sequence, and includes using an optical flow enhancement method based on a generative adversarial network and a self-recurrent architecture to generate the digital human video, comprising: constructing an optical flow enhancement method framework based on a generative adversarial network, which includes a first generator part and a first discriminator part, the first generator part includes a first generator G1 and an optical flow model G o , the first discriminator part includes an image quality discriminator D c , a face discriminator D F , and a multi-scale dynamic discriminator D D , and a self-recurrence is established between the output and the input of the first generator, the generated video frame at the output end is taken as the historical generated video frame at the input end, the input neural rendering image sequence X t-K:t and the historical generated video frame are obtained through the first generator G1 to obtain the current generated video frame Real image pairs (X t , Y t ) composed of the neural rendering image X t and the real video frame Y t and generated image pairs are input to the image quality discriminator D c to discriminate the authenticity of the video frame, at the same time, real image pairs and generated image pairs corresponding to the face region are input to the face discriminator D F to discriminate the authenticity of the face region, the input real frame sequence Y t-K:t passes through the optical flow model G o to obtain the optical flow change W t-K:t-1 between the video frames, real image pairs (W t-K:t-1 , Y t-K:t ) and generated image pairs are input to the multi-scale dynamic discriminator D D to discriminate the authenticity of the video frame based on the optical flow change, so that the dynamic change embodied by the generated video frame based on the optical flow change is consistent with the dynamic change of the real video frame; after training the optical flow enhancement method framework, the trained first generator is extracted as a video generation model for generating a digital human video; Or also used for generating digital human video by using the texture adhesion improvement method of introducing the non-aliasing convolution module and the global translation rotation transformation module of StyleGAN3 in the generative adversarial network architecture, comprising: constructing a texture adhesion improvement method framework based on the generative adversarial network architecture StyleUNet, which includes a second generator part and a second discriminator part, the second generator part includes a mapping network G f , an image encoder G e , a global translation rotation transformation module G r , and a generation network G2, wherein the convolution module in the generation network G2 is replaced by the non-aliasing convolution module of StyleGAN3, and the noise input in the generation network G2 is cancelled, while a jump connection is constructed between the output of each layer of the image encoder G e and each layer of the non-aliasing convolution module in the generation network G2 to form a residual structure, and the second discriminator part includes a discriminator D2, and the random noise Z t is mapped into latent code W f by the mapping network G t , the input neural rendering image X t goes through the image encoder to obtain multi-level image features, the image features of the intermediate level are connected to the generation network G2 through the residual structure, and the image features of the last level are converted into Fourier features after Fourier transform, and then input to the global translation rotation transformation module, calculate the global translation rotation parameters and input them to the generation network G f , in which the global translation rotation parameters and the image features of the intermediate level are combined to calculate the generated video frame G f The real image pair (X t , Y t ) composed of the neural rendering image X t and the real video frame Y t , and the generated image pair are input to the discriminator D2 to determine the authenticity of the video frame; after training the texture adhesion improvement method framework, the mapping network G f , the image encoder G e , the global translation rotation transformation module G r , and the generation network G2 contained in the second generator part are extracted to form a video generation model, which is used for generating digital human video.​ 8. A computing device comprising a memory and one or more processors, the memory having stored therein executable code, wherein, The one or more processors, when executing the executable code, are configured to implement the digital human video generation method based on monocular video replay according to any one of claims 1-6.

Images