Digital human video generation method and system based on local key position enhancement

Abstract

The present invention discloses a method and system for generating digital human videos based on local key position enhancement, comprising the following steps: acquiring a monocular video and extracting a key point sequence; rendering the key point sequence in real time using a real-time neural semantic image rendering method based on OPENGL to obtain a neural semantic image sequence and an eye gaze image sequence; introducing a local discriminator for distinguishing the authenticity of local features into a StyleUNet network comprising a generator and an image quality discriminator to construct an adversarial learning framework for joint optimization of multiple discriminators, constructing a two-stage adversarial training of the adversarial learning framework using normal samples and difficult samples, and using the generator as a video generation model after the training; and continuously generating digital human images using the video generation model based on the neural semantic image sequence, the eye gaze image sequence, and introduced random noise, and adding audio to the continuous digital human images to obtain high-fidelity, time-series continuous digital human videos.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital human generation, and in particular relates to a method and system for generating digital human videos based on local key position enhancement. Background Art

[0002] In recent years, with the growth of the internet, virtual human technology has been widely applied in various fields, such as intelligent customer service and virtual livestreaming. Traditional virtual human technology involves 3D modeling, texture mapping, motion capture, speech synthesis, graphics rendering, and audio and video synthesis, ultimately simulating the appearance, voice, and behavior of real humans. In human-computer interaction applications, the use of digital humans can more quickly establish connections with users and improve the user experience. Furthermore, the dual-modality of visual and auditory interaction helps enhance human language perception and provide a more accurate interactive experience.

[0003] However, the modeling process of traditional three-dimensional digital humans based on traditional graphics is cumbersome and lengthy, which hinders the popularization and application of virtual digital humans. Therefore, with the rapid development of artificial intelligence technology, the use of generative artificial intelligence technology to produce highly realistic and large-scale virtual digital humans is gradually becoming a research hotspot.

[0004] With the rise of generative adversarial networks (GANs), attempts have been made to use them to generate digital humans. For example, patent application publication number CN116912421A discloses a 2D digital human modeling optimization method based on an improved GAN. The method includes facial image data preprocessing, facial image cartoonization using U-Net, generator and discriminator adversarial training, and model optimization. This method enhances existing 2D digital human modeling methods by utilizing an improved U-Net structure generator training to convert input cartoon facial images into realistic facial images. The generated model is also optimized using the EMA algorithm to smooth out differences in pixel points within the image, ensuring both image stability and efficiency and practicality for practical applications.

[0005] For example, the patent application with publication number CN115471886A discloses a method and system for generating digital humans. Based on the deep autoencoding theory, the system calls the encoder to extract facial images and voice features respectively, and performs feature fusion in the latent space to establish a correlation between voice and facial features. The fused features are then input into a pre-trained GAN network to generate a sequence of digital human images with dynamic changes, and the correlation between voice and face is used to drive the digital human to perform dynamic activities.

[0006] In the above two technical solutions, when using GAN to generate digital humans, no attention is paid to local features, so the fidelity of the generated digital humans in local features needs to be improved. Summary of the Invention

[0007] In view of the above, the purpose of the present invention is to provide a method and system for generating digital human videos based on local key position enhancement, which uses key point features extracted from human videos for modeling, relies on multimodal driving information and focuses on local key positions to generate identity-specific, high-fidelity, and temporally continuous digital human videos that are consistent with input facial expressions and movements. This method and system can reduce the difficulty of developing digital human-related applications and alleviate the workload of manual modeling, and has broad application prospects in multiple fields.

[0008] To achieve the above-mentioned object of the invention, an embodiment provides a method for generating a digital human video based on local key position enhancement, comprising the following steps:

[0009] Obtain a monocular video and generate a key point sequence based on the monocular video. After preprocessing the key point sequence, divide it into different action segments according to the action. Align the action segments based on the redirection method of Proctor analysis to obtain the redirected key point sequence.

[0010] The redirected key point sequence is rendered in real time using a real-time neural semantic image rendering method based on OPENGL to obtain a neural semantic image sequence and an eye gaze image sequence.

[0011] A local discriminator for distinguishing authenticity from local features is introduced into the StyleUNet network, which includes a generator and an image quality discriminator. This constructs an adversarial learning framework for joint optimization of multiple discriminators. The adversarial learning framework is trained in two stages using normal and difficult samples. After training, the generator is used as a video generation model.

[0012] A video generation model is used to continuously generate digital human images based on neural semantic image sequences, eye gaze image sequences, and introduced random noise, and audio is added to the continuous digital human images to obtain digital human videos.

[0013] Preferably, preprocessing the key point sequence includes performing linear interpolation processing on the key points and performing time series smoothing processing on the key points based on a Euro filter algorithm.

[0014] Preferably, the redirection method based on Proctor analysis is used to align the action segments, including:

[0015] Calculate the average key points of the key points in the source action segment and the target action segment respectively as the reference key point matrix, and then solve the transformation matrix from the source action reference key point matrix to the target action reference key point matrix according to the least squares method in Prof. Pfizer analysis. The transformation matrix includes translation matrix, rotation matrix and scaling matrix.

[0016] Apply the transformation matrix to the key point sequence in each action clip separately to obtain the redirected key point sequence.

[0017] Preferably, the redirected key point sequence is rendered in real time using a real-time neural semantic image rendering method based on OPENGL to obtain a neural semantic image sequence and an eye gaze image sequence, including:

[0018] Key point color definition, predefine a color for each key point;

[0019] Key point interpolation: interpolate the torso key points contained in the key point sequence to increase the number of torso key points;

[0020] Key point neural semantic graph definition, using different graphs to mark key points in different parts;

[0021] Real-time rendering of different parts is performed, and head image rendering, body image rendering, and eye image rendering are performed respectively according to predefined colors, primitives, and key points, to obtain a neurosemantic image sequence obtained by head image rendering and body image rendering, and an eye gaze image sequence obtained by eye image rendering.

[0022] Preferably, in the adversarial learning framework, the generator is used to generate a video frame sequence according to an input image sequence and a random noise sequence, wherein the input image sequence is composed of a concatenation of a neural semantic image sequence and an eye gaze image sequence;

[0023] The image quality discriminator is used to discriminate the authenticity of an input video frame, wherein the input video frame includes a combination of a real video frame and a neural semantic image, and a combination of a generated video frame and a neural semantic image;

[0024] The local feature discriminator is used to discriminate the authenticity of a region of interest that is considered to be potentially blurred, where the region of interest includes a combination of a real region of interest extracted from a real video frame and a region of interest extracted from a neural semantic image, and a combination of a region of interest extracted from a generated video frame and a region of interest extracted from a neural semantic image.

[0025] Preferably, all regions of interest are extracted in the same way, which is:

[0026] Based on the redirected key point sequence, the coordinate mean point of the key points of the part of interest is calculated as the center point, and the minimum cropping length that includes the part of interest is determined based on the center point. The region of interest is cropped from the image with a size greater than the minimum cropping length and centered on the center point to achieve the extraction of the region of interest.

[0027] Preferably, the loss function used in adversarial training of the adversarial learning framework includes generation loss and discrimination loss;

[0028] The generation loss includes the reconstruction loss L based on the difference between the generated video frame and the real video frame rec , image perception loss L based on the difference between the image features of the generated video frame and the image features of the real video frame sen , adversarial loss L based on image quality discriminator adv1 and the adversarial loss L based on the local discriminator adv2 ;

[0029] The discriminant loss includes the discriminant loss L based on the image quality discriminator D1 and the discriminative loss L based on the local discriminator D2 .

[0030] Preferably, constructing normal samples and difficult samples to conduct two-stage adversarial training on the adversarial learning framework includes:

[0031] In the pre-training stage, normal samples are used to conduct adversarial training on the adversarial learning framework. In this stage, the adversarial loss L corresponding to the local discriminator is adv2 Set to low weight; low weight settings range from 0.1-0.5.

[0032] In the fine-tuning stage, normal samples and several times difficult samples are used to conduct adversarial training on the pre-trained adversarial learning framework. In this stage, the adversarial loss L corresponding to the local discriminator is adv2 Set to high weight; high weight settings range from 1-5.

[0033] Normal samples refer to video frames in digital human materials that tend to be static and have no obvious hand and body movements. Difficult samples refer to video frames in video materials with hand and body movements, especially the intermediate states of character motion clips. The number of samples is smaller than that of normal samples, and motion blur is more likely to occur, making it more difficult for the model to learn correctly, so they are called difficult samples.

[0034] To achieve the above-mentioned object of the invention, an embodiment of the present invention further provides a digital human video generation system based on local key position enhancement, comprising:

[0035] The key point extraction module is used to obtain a monocular video and generate a key point sequence based on the monocular video. The key point sequence is divided into different action segments according to the action. After preprocessing the action segments, the action segments are aligned based on the redirection method of Proctor analysis to obtain the redirected key point sequence;

[0036] A neural rendering module is used to render the redirected key point sequence in real time using a real-time neural semantic image rendering method based on OPENGL to obtain a neural semantic image sequence and an eye gaze image sequence;

[0037] The video generation model construction module is used to introduce a local discriminator for distinguishing authenticity from local features into the StyleUNet network that includes the generator and image quality discriminator to build an adversarial learning framework for joint optimization of multiple discriminators. The adversarial learning framework is trained in two stages using normal and difficult samples. After training, the generator is used as the video generation model.

[0038] The digital human video generation module is used to continuously generate digital human images based on a neural semantic image sequence, an eye gaze image sequence, and introduced random noise using a video generation model, and to add audio to the continuous digital human images to obtain digital human videos.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] By redirecting the extracted key points, the accuracy of the key points is improved, stable and accurate data is provided for subsequent adversarial learning and digital human video generation, and the ontology modeling of digital humans is realized. On this basis, image sequences are generated through neural semantic image rendering, and based on the image sequences, local discriminators for local feature authenticity judgment are introduced into the StyleUNet network to construct an adversarial learning framework for joint optimization of multiple discriminators. Two-stage training of the adversarial learning framework can improve the generator's focus on local features when generating video frames, improve the fidelity of local features, greatly improve the fidelity of the entire video frame, and enable continuous generation in time sequence, greatly improving the generation efficiency of digital human videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 is a flow chart of a method for generating a digital human video based on local key position enhancement provided by an embodiment;

[0043] Figure 2 It is a neural rendering semantic graph drawing hierarchical structure diagram based on OPENGL provided by the embodiment;

[0044] Figure 3 is a schematic diagram of a neural semantic image and an eye gaze image provided by an embodiment;

[0045] Figure 4 Schematic diagram of the structure of the adversarial learning framework provided by the embodiment;

[0046] Figure 5 is a sample generation flow chart provided by the embodiment;

[0047] Figure 6 3 is a structural diagram of a digital human video generation system based on local key position enhancement provided by an embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0049] The technical concept of this invention is that existing digital human technology still faces technical challenges, such as the authenticity of generated videos, the clarity of parts like teeth and hands, and the speed of generation, which still require improvement. To address these technical issues, this invention provides a digital human video generation solution based on local key position enhancement. This solution incorporates a graphics rendering pipeline into monocular video replay technology, performs local key position enhancement, and incorporates a difficult sample learning approach to further improve the fidelity of digital human videos, thereby better meeting user needs and application requirements.

[0050] like Figure 1 As shown, the embodiment provides a method for generating a digital human video based on local key position enhancement, comprising the following steps:

[0051] S1: Obtain a monocular video and generate a key point sequence based on the monocular video. After preprocessing the key point sequence, divide it into different action segments according to the action. Align the action segments based on the redirection method of Prof. Pfizer analysis to obtain the redirected key point sequence.

[0052] In the embodiment, N frames of monocular video sequence V are input 1：N ={V 1 ,…,V N}, where V i ∈R H×W×3 , V i Represents the i-th video frame, H×W is the video frame size, 3 represents the number of channels, and generates N frames containing the key point sequence L of facial key points, torso key points, hand key points, and eye key points. 1 : N ={L 1 ,…,L N}, where L i ∈R J×3 , J is the number of key points, 3 is the three-dimensional coordinates of the key points, L i Represents the i-th key point sequence. In actual extraction, L i ∈R560×3 Specifically, it contains 468 facial key points, 8 torso key points, 21 key points per hand, and 21 eye key points per eye. Specifically, the Faceverse 3DMM model, DWpose model, and Mediapipe tool can be used to extract facial key points, torso and hand key points, and eye key points, respectively.

[0053] In an embodiment, in order to address the problems of missing and jittering key points that may exist after key point extraction, the key points are preprocessed, including: utilizing the video frame context to complete the possible missing key points using linear interpolation, and performing temporal smoothing processing on the facial, body and hand key points using a Euro filter algorithm, thereby smoothing small amplitude jitters while retaining the original data samples.

[0054] In the embodiment, in order to meet the needs of subsequent real-time driving applications, the digital human is required to be able to perform limb movements coherently in different action segments. Therefore, the pre-processed key point sequence is divided into different action segments according to the action, and the action segments are aligned based on the redirection method of Proctor analysis. The posture of the source action is redirected to the target action to obtain the redirected key point sequence.

[0055] The redirection process includes: calculating the average key points of the key points in the source action clip and the target action clip respectively as the reference key point matrix, and then solving the transformation matrix from the source action reference key point matrix to the target action reference key point matrix according to the least squares method in the Proctor analysis. The transformation matrix includes the translation matrix, rotation matrix and scaling matrix; applying the transformation matrix to the key point sequence in each action clip respectively to obtain the redirected key point sequence.

[0056] S2, uses the real-time neural semantic image rendering method based on OPENGL to render the redirected key point sequence in real time to obtain the neural semantic image sequence and the eye gaze image sequence.

[0057] In the embodiment, a real-time neural semantic image rendering method based on OPENGL is used to use the redirected N-frame key point sequence L 1：N As input, generate N frames of neural semantic image sequence CCBR 1:N ={CCBR 1 ,...,CCBR N}, where CCBR i ∈R H×W×3 , eye gaze image sequence E 1:N ={E N ,...,E N}, where E i ∈R H×W×3Specifically, for OPENGL's real-time neural image rendering pipeline, such as Figure 2 As shown, it includes the following key sub-steps:

[0058] 1. Color definition, predefine a color for each key point;

[0059] Specifically, for the body, hand, and facial key points, each key point is given a predefined color, where the values ​​of the red and green channels are obtained by the normalized UV coordinates, and the blue channel adopts a predefined fixed value.

[0060] 2. Key point interpolation: interpolate the torso key points contained in the key point sequence to increase the number of torso key points;

[0061] Since increasing the number of skeleton points can improve the replay performance, interpolation is performed between 8 body skeleton points. The number and color of the interpolation between joint points are fixed. After interpolation, a total of 79 body posture key points are obtained.

[0062] 3. NDC (normalized device coordinates) space coordinate transformation;

[0063] Specifically, the key points to be drawn in all video frames are converted into the coordinate space of (x, y)∈[-1, 1].

[0064] 4. Key point neural semantic graph definition, using different graphs to mark key points in different parts;

[0065] Specifically, when defining neural semantic graphs for key points, dots are used to represent the body and pupil key points, triangle patches are used to represent the palm and face key points, polygonal patches are used to represent the mouth and eye socket key points, and line segments are used to represent the finger key points.

[0066] 5. Draw cache definition;

[0067] OPENGL interacts with the GPU through various buffer object management. In the embodiment, off-screen rendering is mainly used to control image drawing. EGL is used as the underlying API for docking with the platform window system to process the graphics context. Frame buffer objects and texture buffer objects are defined to buffer word rendering results. When docking with key point data, VAO (vertex array object) is used to define the vertex array of the specified body part, and specific VBO (vertex buffer object) and IBO (index buffer object) are bound to it. Two VBO objects are used respectively, corresponding to the vertex color and the body key point position of the specified frame. IBO specifies the index order of the vertex part of the body to parse the VBO object.

[0068] 6. Drawing primitives and shader definitions

[0069] This step controls the specific rendering behavior of OpenGL, specifically using shader program fragments to control the graphics rendering pipeline. In this embodiment, vertex shaders and fragment shaders are primarily used. For different body part definitions, the shader accepts corresponding color and position buffers and selects different primitive rendering methods based on the graphics defined in the key point neural semantic graph definition step. The result is a separate rendered image for each body part.

[0070] 7. Graphics drawing overlay and control

[0071] In this step, the actual drawing process of the OPENGL renderer and the external interface of the graphics drawing process are mainly defined. For example, the specific primitive drawing order of eye drawing and hand drawing is implemented. The head image drawing, body image drawing, and eye image drawing are completed in sequence according to the similar process as above, and finally the N frames of neural semantic image sequence CCBR obtained by head image drawing and body image drawing are obtained. 1:N ={CCBR 1 ...,CCBR N}, where CCBR i ∈R H×W×3 , eye gaze image sequence E obtained by eye image rendering 1:N ={E N ,...,E N}, where E i ∈R H×W×3 ,like Figure 3 shown.

[0072] S3, introduces a local discriminator for distinguishing authenticity from local features into the StyleUNet network that includes a generator and an image quality discriminator to construct an adversarial learning framework for joint optimization of multiple discriminators, and constructs normal samples and difficult samples to conduct two-stage adversarial training on the adversarial learning framework. After the training, the generator is used as a video generation model.

[0073] In the embodiment, Figure 4 As shown, the generator is used to generate a video frame sequence based on an input image sequence and a random noise sequence, where the input image sequence is spliced ​​by a neural semantic image sequence and an eye gaze image sequence.

[0074] Specifically, the neural semantic image sequence CCBR of N frames 1:N and N frames of eye gaze image sequence E 1:N Splicing is performed on the channel dimension and uniformly represented as the input image sequence X 1:N ={X 1 ,...,X N}, where X i ∈R H×W×6, randomly sample the random noise variable Z from Gaussian noise t ∈N(0,1), the input image sequence X 1:N and random noise variable Z t Input to the generator to get a generated video sequence of N frames in

[0075] The generator G consists of three parts: mapping network, image encoder and generation network. The mapping network is used to convert the random noise Z t Converted to a latent code W∈R for controlling style features 64 The image feature encoder encodes multi-level image features, with each layer of features connected to the features of the previous layer. The generator network generates video frames based on the image features and the latent codes. The generator adopts a UNet architecture. The image encoder consists of three layers and is fed into the generator network via skip connections. The network performs a wavelet transform on the input image and then feeds it into the backbone network. Finally, an inverse wavelet transform is performed to restore the image to its original size.

[0076] In the embodiment, the image quality discriminator D1 is used to determine the authenticity of the input video frame, wherein the input video frame includes a combination of a real video frame and a neural semantic image (V t ,CCBR t ), and generate a combination of video frames and neural semantic images

[0077] In the embodiment, the local feature discriminator D2 is used to determine the authenticity of the region of interest that may be blurred in order to improve the clarity of the local position, wherein the region of interest includes a real region of interest CROP extracted from a real video frame. t CCBRC with Regions of Interest Extracted from Neural Semantic Images t Combination (CROP t ,CCBRC t ), and the regions of interest extracted from the generated video frames Combination with regions of interest extracted from neural semantic images

[0078] Specifically, the extraction method of the region of interest of each type of image is the same: the coordinate mean point of the key points of the part of interest is calculated based on the redirected key point sequence as the center point, and the minimum cropping length min(x) that includes the part of interest is determined based on the center point, and the size greater than the minimum cropping length min(x) is selected. Right now The region of interest is cropped from the image with the center point as the center to achieve the extraction of the region of interest.

[0079] The loss functions used in adversarial learning frameworks for adversarial training include generation loss and discrimination loss. G It includes the reconstruction loss L based on the difference between the generated video frame and the real video frame rec , image perception loss L based on the difference between the image features of the generated video frame and the image features of the real video frame sen , adversarial loss L based on image quality discriminator adv1 and the adversarial loss L based on the local discriminator adv2 ;

[0080] Specifically, the reconstruction loss L rec Using LI loss, image perception loss L sen The image features used can be the image features extracted by inputting the video frame into the VGG network. adv1 and the adversarial loss L based on the local discriminator adv2 Respectively expressed as:

[0081]

[0082]

[0083] Among them, fake 1,i Representation combination The result of the true and false discrimination input to the image quality discriminator, that is, the discrimination score, fake 2,i Representation combination The true or false discrimination result input to the local discriminator is the discrimination score.

[0084] Generate loss L G Expressed as:

[0085]

[0086] Among them, λ adv1 ,λ sen , and λ rec Represent the weight of loss, L adv2,j represents the adversarial loss of the j-th local discriminator, λ adv2,j Indicates L adv2,j There can be several local discriminators, such as the left hand, right hand, eyes, mouth, head, etc., and one local discriminator can be used for each position. Therefore, there may be n local discriminators in this case.

[0087] Discriminative loss L D Including the discriminative loss L based on the image quality discriminator D1 and the discriminative loss L based on the local discriminator D2 , respectively expressed as:

[0088]

[0089]

[0090] Among them, real 1,i Indicates combination (V t ,CCBR t ) is input to the true and false discrimination result of the image quality discriminator, that is, the discrimination score, real 2,i Indicates combination (CROP t ,CCBRC t ) is input into the true or false discrimination result of the local discriminator, that is, the discrimination score.

[0091] In the embodiment, the two-stage adversarial training of the adversarial learning framework using the constructed normal samples and difficult samples includes:

[0092] Construct normal samples and difficult samples, clean and segment the original collected digital human training video data, extract key points, remove background, render neural semantic images, and build a sample annotation system. Specifically, use PY-QT to build a data annotation system that can load image frames, such as Figure 5 As shown, the system reads monocular video frames and action clip sequences that have been processed as described above, and can perform frame review in frame order. For difficult samples, the reviewer uses the system to calibrate the start and end frames of the character's motion clip. For erroneous samples, the system can calibrate the start and end frames of the erroneous sample clip. For difficult samples, their indexes serve as input for subsequent sample supersampling. For erroneous samples, these erroneous samples are skipped during the training process. Specifically, normal samples refer to video frames in digital human materials that tend to be static and have no obvious hand and body movements. Difficult samples refer to video frames in video materials that have hand and body movements, especially the intermediate states of character motion clips. The number of samples is less than that of normal samples, and motion blur is prone to occur. It is difficult for the model to learn correctly, so they are called difficult samples. Error samples are frames where the video hand image results and key points are significantly different after the character key points are visualized.

[0093] In the pre-training stage, normal samples are used to perform multiple iterative adversarial training on the adversarial learning framework until the adversarial framework as a whole fits to a better overall effect. The adversarial loss L corresponding to the local discriminator in this stage is adv2 The low weight is set to 0.1-0.5, which can be adjusted according to the actual model convergence.

[0094] In the fine-tuning stage, normal samples are combined with difficult samples for training. Using oversampling, the amount of difficult sample data is increased several times in the normal training data, and the pre-trained adversarial learning framework is trained adversarially. At this stage, the local discriminator D2 and the generator G have been well pre-trained in the fine-tuning stage, so the adversarial loss L corresponding to the local discriminator is adv2 The high weight is set to 1-5 and can be adjusted according to the actual model convergence. This can ensure high-quality replay of all actions and expressions of the digital human video.

[0095] After the above two-stage training, the trained generator is used as a video generation model for subsequent digital human videos.

[0096] S4, using a video generation model to continuously generate digital human images based on a neural semantic image sequence, an eye gaze image sequence, and the introduced random noise, and adding audio to the continuous digital human images to obtain a digital human video.

[0097] In the embodiment, corresponding facial key points and audio input interfaces are provided externally. When using, the user can select the corresponding action segment processed in step S1 in the system according to the action sequence number. First, the generated facial key point representation is aligned with the template action facial representation. Then, all key point sequences are rendered through the OPENGL neural semantic image to obtain the neural semantic image required for the video generation model input. The video generation model is then used to generate continuous video frames of the digital human. The generated video frames are synthesized with the audio input to finally generate a high-fidelity digital human video with synchronized audio and video. Considering single-frame performance, in the case of a single A100 high-performance computing card, the OPENGL neural semantic image rendering module can achieve a maximum of 4ms per frame, and the neural network rendering module can achieve a maximum of 10ms per frame. The overall latency is close to 30ms, and real-time video stream generation of more than 25fps can be achieved.

[0098] like Figure 6As shown, the embodiment also provides a digital human video generation system 60 based on local key position enhancement, including a key point extraction module 61, a neural rendering module 62, a video generation model construction module 63, and a digital human video generation module 64, wherein the key point extraction module 61 is used to obtain a monocular video and generate a key point sequence based on the monocular video; the neural rendering module 62 is used to use a real-time neural semantic image rendering method based on OPENGL to render the redirected key point sequence in real time to obtain a neural semantic image sequence and an eye gaze image sequence; the video generation model construction module 63 is used to introduce a local discriminator for local feature authenticity discrimination into a StyleUNet network including a generator and an image quality discriminator to construct a multi-discriminator joint optimization adversarial learning framework, and construct normal samples and difficult samples to perform two-stage adversarial training on the adversarial learning framework. After the training, the generator is used as a video generation model; the digital human video generation module 64 is used to use the video generation model to continuously generate digital human images based on the neural semantic image sequence, the eye gaze image sequence, and the introduced random noise, and add audio to the continuous digital human images to obtain digital human videos.

[0099] It should be noted that the digital human video generation device based on local key position enhancement provided in the above embodiment, when performing digital human video generation, should be illustrated by the division of the above-mentioned functional modules. The above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the digital human video generation device based on local key position enhancement provided in the above embodiment and the digital human video generation method based on local key position enhancement provided in the above embodiment are based on the same concept. The specific implementation process is detailed in the embodiment of the digital human video generation method based on local key position enhancement, and will not be repeated here.

[0100] The above-described method and device use monocular video as input and model a digital human through a neural network. This allows the generation of a digital human video that matches the original video's facial expressions, head movements, and body posture while preserving the target person's identity. This technology can reduce the development difficulty of digital human-related applications and alleviate the burden of manual modeling. It has broad application prospects in fields such as virtual reality, digital entertainment, video games, education, and finance.

[0101] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating digital human videos based on local key position enhancement, characterized in that: The following steps are involved: Obtain a monocular video, and generate a key point sequence based on the monocular video. After preprocessing the key point sequence, divide it into different action segments according to the action, and align the action segments based on the redirection method of Proctor analysis to obtain a redirected key point sequence. The redirection method based on Proctor analysis to align the action segments includes: calculating the average key points of the key points in the source action segment and the target action segment as a reference key point matrix, and then solving the transformation matrix from the source action reference key point matrix to the target action reference key point matrix according to the least squares method in Proctor analysis, the transformation matrix includes a translation matrix, a rotation matrix and a scaling matrix; applying the transformation matrix to the key point sequence in each action segment to obtain a redirected key point sequence; The redirected key point sequence is rendered in real time using a real-time neural semantic image rendering method based on OPENGL to obtain a neural semantic image sequence and an eye gaze image sequence, including: key point color definition, pre-defining a color for each key point; key point interpolation, interpolating the torso key points contained in the key point sequence to increase the number of torso key points; key point neural semantic graphic definition, using different graphics to mark key points in different parts; real-time rendering of each part, performing head image rendering, body image rendering, and eye image rendering respectively according to predefined colors, primitives, and key points, to obtain a neural semantic image sequence obtained by head image rendering and body image rendering, and an eye gaze image sequence obtained by eye image rendering; A local discriminator for distinguishing authenticity from local features is introduced into the StyleUNet network, which includes a generator and an image quality discriminator. This constructs an adversarial learning framework for joint optimization of multiple discriminators. The adversarial learning framework is trained in two stages using normal and difficult samples. After training, the generator is used as a video generation model. In the adversarial learning framework, the generator is used to generate a video frame sequence based on an input image sequence and a random noise sequence, wherein the input image sequence is spliced ​​by a neural semantic image sequence and an eye gaze image sequence; the image quality discriminator is used to discriminate the authenticity of the input video frame, wherein the input video frame includes a combination of a real video frame and a neural semantic image, and a combination of a generated video frame and a neural semantic image; the local feature discriminator is used to discriminate the authenticity of a region of interest that is believed to be potentially blurred, wherein the region of interest includes a combination of a real region of interest extracted from the real video frame and a region of interest extracted from the neural semantic image, and a combination of a region of interest extracted from the generated video frame and a region of interest extracted from the neural semantic image; A video generation model is used to continuously generate digital human images based on neural semantic image sequences, eye gaze image sequences, and introduced random noise, and audio is added to the continuous digital human images to obtain digital human videos.

2. The method for generating digital human video based on local key position enhancement according to claim 1, characterized in that: The preprocessing of key point sequence includes linear interpolation processing of key points and time series smoothing processing of key points based on the Euro filter algorithm.

3. The method for generating digital human video based on local key position enhancement according to claim 1, characterized in that: All regions of interest are extracted in the same way: Based on the redirected key point sequence, the coordinate mean point of the key points of the part of interest is calculated as the center point, and the minimum cropping length that includes the part of interest is determined based on the center point. The region of interest is cropped from the image with a size greater than the minimum cropping length and centered on the center point to achieve the extraction of the region of interest.

4. The method for generating digital human video based on local key position enhancement according to claim 1, characterized in that: The loss functions used in adversarial training of the adversarial learning framework include generation loss and discrimination loss; The generation loss includes the reconstruction loss based on the difference between the generated video frame and the real video frame L rec , image perception loss based on the difference between the image features of the generated video frames and the image features of the real video frames L sen , adversarial loss based on image quality discriminator L adv1 and adversarial loss based on local discriminator L adv2 ; The discriminant loss includes the discriminant loss based on the image quality discriminator L D1 and the discriminative loss based on the local discriminator L D2 .

5. The method for generating digital human video based on local key position enhancement according to claim 1, characterized in that: Constructing normal samples and difficult samples to conduct two-stage adversarial training on the adversarial learning framework, including: In the pre-training stage, normal samples are used to conduct adversarial training on the adversarial learning framework. In this stage, the adversarial loss corresponding to the local discriminator is L adv2 The low weight is set to 0.1-0.5; In the fine-tuning stage, normal samples and several times more difficult samples are used to conduct adversarial training on the pre-trained adversarial learning framework. In this stage, the adversarial loss corresponding to the local discriminator is L adv2 Set to high weight; the high weight setting range is 1-5; Normal samples refer to video frames in digital human materials that are relatively static and have no obvious hand and body movements. Difficult samples refer to video frames in video materials that have hand and body movements. The number of samples is smaller than that of normal samples, and motion blur is more likely to occur.

6. A digital human video generation system based on local key position enhancement, characterized in that: include: A key point extraction module is used to obtain a monocular video and generate a key point sequence based on the monocular video, divide the key point sequence into different action segments according to the action, and after pre-processing the action segments, align the action segments based on the redirection method of Proctor analysis to obtain a redirected key point sequence, wherein the redirection method based on Proctor analysis to align the action segments includes: calculating the average key points of the key points in the source action segment and the target action segment as a reference key point matrix, and then solving the transformation matrix from the source action reference key point matrix to the target action reference key point matrix according to the least squares method in Proctor analysis, the transformation matrix includes a translation matrix, a rotation matrix and a scaling matrix; applying the transformation matrix to the key point sequence in each action segment to obtain a redirected key point sequence; A neural rendering module is used to render the redirected key point sequence in real time using an OPENGL-based real-time neural semantic image rendering method to obtain a neural semantic image sequence and an eye gaze image sequence, including: key point color definition, predefining a color for each key point; key point interpolation, interpolating the torso key points contained in the key point sequence to increase the number of torso key points; key point neural semantic graphic definition, using different graphics to mark key points in different parts; real-time rendering by part, rendering the head image, body image, and eye image respectively according to predefined colors, primitives, and key points, to obtain a neural semantic image sequence obtained by rendering the head image and body image, and an eye gaze image sequence obtained by rendering the eye image; The video generation model construction module is used to introduce a local discriminator for distinguishing authenticity from local features into the StyleUNet network that includes the generator and image quality discriminator to build an adversarial learning framework for joint optimization of multiple discriminators. The adversarial learning framework is trained in two stages using normal and difficult samples. After training, the generator is used as the video generation model. In the adversarial learning framework, the generator is used to generate a video frame sequence based on an input image sequence and a random noise sequence, wherein the input image sequence is spliced ​​by a neural semantic image sequence and an eye gaze image sequence; the image quality discriminator is used to discriminate the authenticity of the input video frame, wherein the input video frame includes a combination of a real video frame and a neural semantic image, and a combination of a generated video frame and a neural semantic image; the local feature discriminator is used to discriminate the authenticity of a region of interest that is believed to be potentially blurred, wherein the region of interest includes a combination of a real region of interest extracted from the real video frame and a region of interest extracted from the neural semantic image, and a combination of a region of interest extracted from the generated video frame and a region of interest extracted from the neural semantic image; The digital human video generation module is used to continuously generate digital human images based on a neural semantic image sequence, an eye gaze image sequence, and introduced random noise using a video generation model, and to add audio to the continuous digital human images to obtain digital human videos.

Images