Multi-modal feature fusion digital human video generation method and device based on time sequence position coding

Abstract

The invention discloses a multi-modal feature fusion digital human video generation method and device based on time sequence position coding. The method comprises the following steps: constructing key points based on a monocular video; fitting and replacing a face Flame coefficient of the SMPLX model by using a face Faceverse coefficient detected by a Faceverse model, fitting and replacing a hand representation of the SMPLX model by using a Mano hand shape detected by a Hamer model, and obtaining an optimized SMPLX model; drawing based on the key points to obtain a color coding representation image and an eye fixation image, and drawing based on the optimized SMPLX model to obtain a depth image, a semantic image and a normal image; time sequence position codes used for enhancing time sequence consistency are introduced into an image generation model, multiple frames of digital human images are continuously generated based on multi-modal features formed by all the images, and audios are added to obtain a digital human video, so that the method has wide application prospects in multiple fields.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital human generation, and in particular relates to a method and device for generating digital human videos by multimodal feature fusion based on temporal position coding. Background Art

[0002] With the rapid development of AI, AI digital human generation tasks have been applied to various fields, including entertainment, education, medical care, and virtual reality. The generation of traditional graphics digital humans usually requires a long model building and rendering process, involving complex 3D modeling, texture mapping, and lighting calculations. These steps are not only time-consuming but also extremely demanding on computing resources. In contrast, the time cycle required for AI digital human modeling is significantly shortened and more efficient, which makes it increasingly popular in practical applications.

[0003] At present, the generation technology of AI digital human has made significant progress. Diffusion-based methods mainly use some pre-trained diffusion models, such as the Stable Diffusion model. These models can generate high-quality, detailed digital human images by converting random noise into clear images through a gradual denoising process. Diffusion-based methods usually generate higher-quality images because their generation process allows for more detailed feature learning and more complex image structure presentation. However, the inference speed of this process is slow, which may become a bottleneck, especially in application scenarios that require real-time feedback. GAN (Generative Adversarial Network)-based methods mainly use some pre-trained GAN models for digital human generation. The core concept of GAN is to use adversarial training of two networks - image generator and discriminator. The image generator is responsible for generating images, while the discriminator is used to determine whether the generated images are real.

[0004] Compared with diffusion models, GAN models usually have an advantage in inference speed, which makes GAN-based models more effective in real-time generation tasks of AI digital humans. For example, in scenarios such as virtual anchors, online education assistants, and intelligent customer service, it is crucial to quickly generate and render high-quality digital human images. The GAN model excels in this regard, and can achieve fast response while maintaining high image quality. The StyleGAN (Style Generative Adversarial Network) model maps the input random noise to the latent style space, decouples the features through the latent style space, and makes the generated images higher quality and more editable.

[0005] However, the above network is mainly used for single-frame image generation, without considering the temporal consistency between video frames. Generating digital human videos through a single-frame image generation model is prone to video frame jitter problems. In addition, digital human image generation based on a single modality will result in low image quality. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a method and device for generating digital human videos based on multimodal feature fusion of temporal position coding, which uses key points and SMPLX coefficient features extracted from character videos for modeling, and relies on multimodal feature fusion information to generate identity-specific, high-fidelity, and temporally continuous digital human videos that are consistent with input facial expressions and movements. This method can reduce the difficulty of developing digital human-related applications and reduce the workload of manual modeling, and has broad application prospects in multiple fields.

[0007] To achieve the above-mentioned purpose of the invention, the embodiment provides a method for generating a digital human video by multimodal feature fusion based on temporal position coding, comprising the following steps:

[0008] Generate key point sequences based on monocular video and divide them into different action segments after preprocessing. Align the action segments based on SMPLX coefficient migration to obtain the redirected key point sequence.

[0009] The new coefficients are fitted using the facial Faceverse coefficients detected by the Faceverse model, and the facial Flame coefficients of the SMPLX model are replaced by the new coefficients. The new hand representation is fitted using the Mano hand shape detected by the Hamer model, and the hand representation of the SMPLX model is replaced by the new hand representation to obtain the optimized SMPLX model.

[0010] Based on the redirected key point sequence, a color coding representation image sequence and an eye gaze image sequence are obtained. Based on the optimized SMPLX model, a depth image sequence, a semantic image sequence, and a normal image sequence are obtained.

[0011] A temporal position coding is introduced into the image generation model to enhance temporal consistency. The temporal position coding includes the index information of the current video and the index information of the video frame in the current video. At the same time, multiple frames of digital human images are continuously generated based on the multimodal features formed by all image sequences, and audio is added to the continuous digital human images to obtain digital human videos.

[0012] Preferably, preprocessing the key point sequence includes: performing linear interpolation processing and time series smoothing processing on the key point sequence.

[0013] Preferably, the action clips are aligned based on the coefficient migration of SMPLX to obtain a redirected key point sequence, including:

[0014] Calculate the SMPLX coefficient sequences in the source action segment and the target action segment respectively, and then replace the posture coefficients of the target action segment except the root node with the posture coefficients of the source action segment except the root node to obtain a new SMPLX coefficient sequence of the target action segment;

[0015] The SmoothNet model is used to smooth the posture coefficient sequence in the new SMPLX coefficient sequence of the target action clip, and the key point sequence is calculated according to the smoothed new SMPLX coefficient sequence to obtain the redirected key point sequence.

[0016] Preferably, the facial Faceverse coefficients detected by the Faceverse model are used to fit new coefficients, and the facial Flame coefficients of the SMPLX model are replaced with the new coefficients, including: using the facial Faceverse coefficients detected by the Faceverse model to calculate facial key points, fitting new coefficients to the facial key points, and replacing the facial Flame coefficients of the SMPLX model with the new coefficients;

[0017] The Mano hand shape detected by the Hamer model is fitted with a new hand representation, and the hand representation of the SMPLX model is replaced by the new hand representation to obtain an optimized SMPLX model, including: using the Mano hand shape detected by the Hamer model, calculating the wrist joint axis angle of the Mano hand shape according to forward kinematics and inverse kinematics to form a new hand representation, and replacing the hand representation of the SMPLX model with the new hand representation.

[0018] Preferably, a color-coded representation image sequence and an eye gaze image sequence are obtained based on the redirected key point sequence, including: inputting the redirected key point sequence into a neural network, drawing the redirected key point sequence based on a neural semantic image drawing method, and generating a color-coded representation image sequence and an eye gaze image sequence.

[0019] Preferably, the temporal position encoding is obtained by:

[0020] NOISE(VI,FI)=PE(VI / VN)+PE(FI / FN)

[0021] PE(p)=(sin(p*2 0 ),cos(p*2 0 ),sin(p*2 1 ),cos(p*2 1 ),...,sin(p*2 k ),cos(p*2 k ))

[0022] Among them, VI represents the index information of the current video, FI represents the index information of the video frame in the current video, VN is the total number of videos, FN is the total number of frames in the current video where the video frame is located, NOISE(VI,FI) represents temporal position coding, PE(p) represents the coding of variable p, in PE(VI / VN), variable p=VI / VN, and in PE(FI / FN), variable p=FI / FN.

[0023] Preferably, the image generation model is constructed based on the StyleUNet network, and the specific construction process is as follows;

[0024] The StyleUNet network includes an image generator and a discriminator. The input of the image generator is the concatenation of a color coding representation image sequence, an eye gaze image sequence, a depth image sequence, a semantic image sequence, and a normal image sequence. It is used to generate a digital human image according to the input and temporal position coding. The discriminator is used to determine whether the input image is the digital human image output by the image generator and the authenticity of the real image.

[0025] Construct the adversarial learning loss function of the StyleUNet network, including generation loss and discrimination loss, where the generation loss includes the reconstruction loss L based on the difference between the generated digital human image and the real video frame rec , based on the image perceptual loss L between the image features of the generated digital human image and the image features of the real video frame sen , and the discriminator-based adversarial loss L adv ; The discriminant loss includes the discriminant loss L based on the discriminator D ;

[0026] The loss function is used to perform adversarial learning on the StyleUNet network to optimize the network parameters, and the image generator with optimized parameters is used as the image generation model.

[0027] To achieve the above-mentioned purpose of the invention, the embodiment of the present invention further provides a multi-modal feature fusion digital human video generation device based on temporal position coding, comprising:

[0028] The key point extraction module is used to generate a key point sequence based on the monocular video and divide it into different action segments after preprocessing. The action segments are aligned based on the coefficient migration of SMPLX to obtain the redirected key point sequence;

[0029] A model optimization module, which is used to fit new coefficients using the facial Faceverse coefficients detected by the Faceverse model, and replace the facial Flame coefficients of the SMPLX model with the new coefficients, fit new hand representations using the Mano hand shape detected by the Hamer model, and replace the hand representation of the SMPLX model with the new hand representation to obtain an optimized SMPLX model;

[0030] A drawing module, which is used to draw a color coding representation image sequence and an eye gaze image sequence based on the redirected key point sequence, and to draw a depth image sequence, a semantic image sequence, and a normal image sequence based on the optimized SMPLX model;

[0031] The video generation module is used to introduce temporal position coding for enhancing temporal consistency into the image generation model. The temporal position coding includes the index information of the current video and the index information of the video frame in the current video. At the same time, multiple frames of digital human images are continuously generated based on the multimodal features formed by all image sequences, and audio is added to the continuous digital human images to obtain digital human videos.

[0032] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned multimodal feature fusion digital human video generation method based on temporal position coding.

[0033] To achieve the above-mentioned purpose of the invention, the embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned multimodal feature fusion digital human video generation method based on temporal position coding is implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] By redirecting the extracted key point sequence based on SMPLX coefficient migration, the accuracy of the key points is improved, providing stable and accurate data for subsequent adversarial learning and digital human video generation, and realizing digital human ontology modeling. At the same time, the facial coefficients and hand representations of the SMPLX model are optimized through the Faceverse model and the Hamer model to improve the accuracy of the three-dimensional human model. On this basis, by fusing the features of color coding representation images, eye gaze images, depth images, semantic images, and normal images, and introducing temporal position encoding for enhancing temporal consistency based on the image sequence image generation model, the adversarial learning framework can be trained to improve the image generation model to pay more attention to temporal features when generating video frames, improve the fidelity of temporal features, greatly improve the fidelity of the entire video frame, and can be generated in a continuous temporal sequence, greatly improving the efficiency of digital human video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 is a flow chart of a method for generating digital human video by multimodal feature fusion based on temporal position coding provided by an embodiment;

[0038] Figure 2 It is a neural rendering semantic graph drawing hierarchical structure diagram provided by an embodiment;

[0039] Figure 3 is a schematic diagram of a color-coded representation image, an eye gaze image, a depth image, a semantic image, and a normal image provided by an embodiment;

[0040] Figure 4 It is a comparison diagram of the effects before and after optimization of the SMPLX model provided in the embodiment;

[0041] Figure 5 is a sample generation flow chart provided by an embodiment;

[0042] Figure 6 is a schematic diagram of the structure of an adversarial learning framework provided in an embodiment;

[0043] Figure 7 It is a structural schematic diagram of a multimodal feature fusion digital human video generation device based on temporal position coding provided in an embodiment. DETAILED DESCRIPTION

[0044] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0045] The inventive concept of the present invention is that some technical problems existing in the existing digital human technology, such as the temporal consistency of the generated video, the generation speed and the generation quality, etc., need to be improved. To solve these technical problems, the present invention provides a digital human video generation scheme based on temporal position coding and multimodal feature fusion, and performs temporal enhancement, while introducing a difficult sample learning method to further improve the high fidelity of the digital human video, so as to better meet the needs of users and application scenarios.

[0046] S1, generates a key point sequence based on the monocular video and divides it into different action segments after preprocessing. The action segments are aligned based on the coefficient migration of SMPLX to obtain the redirected key point sequence.

[0047] In the embodiment, N frames of monocular video sequence V are input 1 : N = {V 1 ,V i ,…,V N}, where V i ∈R H×W×3 , V i represents the i-th video frame, H×W is the video frame size, 3 represents the number of channels, and based on the monocular video sequence, a key point sequence L containing N frames of facial key points, torso key points, hand key points, and eye key points is generated. 1 : N ={L 1 ,L i ,…,L N}, where L i ∈R J ×3 , J is the number of key points, 3 is the three-dimensional coordinates of the key points, L i represents the key point sequence corresponding to the i-th frame. In actual extraction, L i ∈R 560×3 , specifically including 478 facial key points, 8 torso key points, 21 key points per hand, and 21 eye key points per eye. Specifically, when extracting key points, the Faceverse 3DMM model, the SMPLX model, and the Mediapipe tool can be used to extract facial key points, torso and hand key points, and eye key points, respectively.

[0048] In an embodiment, in order to address the key point missing and jitter problems that may exist after the key point extraction, the key point sequence needs to be preprocessed, including: using a key point correction tool to manually correct the key points or correct the key points using linear interpolation, and then using a Euro filter algorithm to perform time series smoothing on the face, body and hand key points respectively, so as to smooth out small amplitude jitters while retaining the original data samples.

[0049] In the embodiment, in view of the subsequent real-time driving application requirements, the digital human is required to be able to perform body movements coherently in different action segments, so the pre-processed key point sequence is divided into different action segments according to the action, and the action segments are aligned based on the redirection method of coefficient migration of SMPLX, and the posture of the source action is redirected to the target action to obtain the redirected key point sequence. The specific process is:

[0050] Calculate the SMPLX coefficient sequences in the source action segment and the target action segment respectively, and then replace the posture coefficients of the target action segment except the root node with the posture coefficients of the source action segment except the root node to obtain a new SMPLX coefficient sequence of the target action segment;

[0051] The SmoothNet model is used to smooth the posture coefficient sequence in the new SMPLX coefficient sequence of the target action clip, and the key point sequence is calculated according to the smoothed new SMPLX coefficient sequence to obtain the redirected key point sequence.

[0052] S2, use the facial Faceverse coefficients detected by the Faceverse model to fit the new coefficients, and use the new coefficients to replace the facial Flame coefficients of the SMPLX model, use the Mano hand shape detected by the Hamer model to fit the new hand representation, and use the new hand representation to replace the hand representation of the SMPLX model to obtain the optimized SMPLX model.

[0053] In the embodiment, the SMPLX coefficients are optimized to address the problem of low accuracy of facial and hand representation detected by the SMPLX model, including:

[0054] (1) The facial Faceverse coefficients detected by the Faceverse model are used to fit new coefficients, and the facial Flame coefficients of the SMPLX model are replaced by the new coefficients. Specifically, the facial Faceverse coefficients detected by the Faceverse model are used to calculate facial key points, new coefficients are fitted to the facial key points, and the facial Flame coefficients of the SMPLX model are replaced by the new coefficients; wherein the Faceverse model is a 3D deformable parameterized model (3DMM).

[0055] More specifically, the Faceverse model is used to detect the facial Faceverse coefficients, and the Faceverse coefficients are split into identity coefficient Id, expression coefficient Exp, texture coefficient Tex, rotation coefficient Angles, illumination coefficient Light, translation coefficient Trans, and eye rotation coefficient Eye, where Id∈R 150 , Exp∈R 52 ,Tex∈R 251 ,Angles∈R 3 , Light∈R 3 ,Trans∈R 3 , Eye∈R 4. The global rotation matrix Rot, the left eye rotation matrix Rot_LeftEye and the right eye rotation matrix Rot_RightEye are calculated based on the rotation coefficient Angles and the eye rotation coefficient Eye. The facial key points are obtained by linearly combining the identity basis vector and the expression basis vector:

[0056] V shape =IdBase*Id+ExpBase*Exp+MeanShape

[0057] Among them, V shape ∈R 478 , represents the facial key points in the standard coordinate system (canonical space), IdBase is the identity basis vector of Faceverse, ExpBase is the expression basis vector of Faceverse, and MeanShape is the average key point of Faceverse; use the rigid transformation V t =Rot*V shape +Trans, calculate the facial key points V after projection mapping t , where V t ∈R 478 According to the expression dictionary matrix D of the Flame model, the formula V t =D*C and SparseCoder (sparse coding, a class in the pythonsklearn library), reversely calculate the expression coefficient C of Flame, where C∈R 100 . Replace the facial Flame coefficient of SMPLX with the expression coefficient C to obtain the optimized Flame coefficient;

[0058] (2) The Mano hand shape detected by the Hamer model is used to fit a new hand representation, and the new hand representation is used to replace the hand representation of the SMPLX model to obtain an optimized SMPLX model. Specifically, the Mano hand shape detected by the Hamer model is used to calculate the wrist joint axis angle of the Mano hand shape according to forward kinematics and inverse kinematics to form a new hand representation, and the new hand representation is used to replace the hand representation of the SMPLX model.

[0059] More specifically, the Hamer model is used to detect the Mano hand shape of the digital human. The Pose coefficient and global rotation angle of Mano's left hand are symmetric about the y-axis (i.e., the rotation in the x and z directions is reversed) to obtain the Pose coefficient correctly mapped to SMPLX. The right-hand coordinate system is consistent with the SMPLX coordinate system and no transformation is required. For the Mano global rotation angle, it is equivalent to the forward kinematics of the rotation from the root node (SMPLX global rotation angle) to the wrist joint in the SMPLX model. For the adopted SMPLX model, most of its joint rotation angles are reasonable under visualization, so the problem is constrained to only modify the wrist joint rotation R hand_local , the formula is as follows:

[0060]

[0061] Where R SMPLX_cum Represents the accumulated rotation from the root node to the wrist joint, expressed as the multiplication form of the rotation matrix, R Mano_hand_global Expressed as a global rotation matrix in Mano form, Represented as R SMPLX_cum The inverse transformation of Multiply by R Mano_hand_global , we can get the new R hand_local Taking the SMPLX left-hand link as an example, its forward kinematic path is as follows:

[0062] The original global rotation angle of the left wrist joint is R hand_org =R Pelvis *R Spine1 *R Spine2 *R Spine3 *R Left_Collar *R Left_Shoulder *R Left_Elbow *R Left_Wrist , the wrist joint angle after Mano correction is R hand_local =(R Pelvis *R Spine1 *R Spine2 *R Spine3 *R Left_Collar *R Left_Shoulder *R Left_Elbow ) -1 *R Mano_hand_global , where R Pelvis , R Spine1 , R Spine2 , R Spine3 , R Left_Collar , R Left_Shoulder , R Left_Elbow , R Left_WristThey are the rotation matrices of the pelvic joint, spine joint 1, spine joint 2, spine joint 3, left clavicle joint, left shoulder joint, left elbow joint, and left wrist joint. Then, the joint angles of the Mano hand are replaced with the corresponding joint angles of SMPLX.

[0063] The comparison of the effects of the SMPLX model before and after optimization is shown in the figure below. Figure 5 As shown, from Figure 5 It can be seen that compared with the normal image obtained by the original SMPLX model, the normal image obtained by the SMPLX model optimized by this method has higher hand accuracy.

[0064] S3, based on the redirected key point sequence, a color coding representation image sequence and an eye gaze image sequence are obtained, and based on the optimized SMPLX model, a depth image sequence, a semantic image sequence, and a normal image sequence are obtained.

[0065] In the embodiment, a neural semantic image rendering method is used to use the redirected N-frame key point sequence L 1 : N As input, generate a color-coded representation image sequence CC of N frames respectively 1:N ={CCBR 1 ,CCBR i ,...,CCBR N}, where CCBR i ∈R H×W×3 And represents the color coding representation image, the eye fixation image sequence E 1:N ={E 1 ,E i ,...,E N}, where E i ∈R H ×W×3 , and represents the eye gaze image. Using the SMPLX coefficient sequence after the SMPLX coefficient migration, the depth image sequence DEPTH of N frames is generated respectively 1:N ={DEPTH 1 ,DEPTH i ,...,DEPTH N}, where DEPTH i ∈R H×W×3 And represents the depth image, semantic image sequence SEG 1:N ={SEG 1 ,SEG i ,...,SEG N}, where SEG i ∈R H×W×3 And represents the semantic image, the normal image sequence NORMAL 1:N={NORMAL 1 ,NORMAL i ,...,NORMAL N}, where NORMAL i ∈R H×W×3 And represents the normal image. Specifically, for the neural image rendering pipeline, Figure 2 As shown, it includes the following key sub-steps:

[0066] 1. SMPLX coefficient drawing

[0067] Use SMPLX coefficients to draw and get the depth image sequence DEPTH 1:N , semantic image sequence SEG 1:N , and the normal image sequence NORMAL 1:N ,like Figure 3 shown.

[0068] 2. Calculation of torso key points

[0069] According to the SMPLX coefficient, the torso key points are calculated and interpolated. The torso key points contained in the key point sequence are interpolated to increase the number of torso key points.

[0070] Since increasing the number of bone points can improve the replay performance, interpolation is performed between 8 body bone points, where the number and color of interpolation between joint points are fixed. After interpolation, a total of 79 body posture key points are obtained.

[0071] 3. Image drawing overlay and control

[0072] In this step, the actual drawing process of the renderer and the external interface of the graphics drawing process are mainly defined. For example, the specific primitive drawing order of eye drawing and hand drawing is implemented. The confidence of the key points is used as the transparency, and the head image drawing, body image drawing, and eye image drawing are completed in sequence, and superimposed to obtain N frames of color-coded representation image sequence CCBR obtained by head image drawing and body image drawing. 1:N , eye gaze image sequence E obtained by eye image drawing 1:N ,like Figure 3 shown.

[0073] S4, introduces temporal position coding for enhancing temporal consistency into the image generation model. The temporal position coding includes the index information of the current video and the index information of the video frame in the current video. At the same time, multiple frames of digital human images are continuously generated based on the multimodal features formed by all image sequences, and audio is added to the continuous digital human images to obtain digital human videos.

[0074] In the embodiment, Figure 6As shown, an image generation model is constructed based on the StyleUNet network, and the StyleUNet network includes an image generator and a discriminator, wherein the image generator is used to generate a video frame sequence according to an input image sequence and a noise sequence controlled by temporal position encoding, wherein the input image sequence is composed of a color coding representation image sequence, an eye gaze image sequence, a depth image sequence, a semantic image sequence, and a normal image sequence.

[0075] Specifically, the color coding of N frames represents the image sequence CCBR 1:N , N frames of eye gaze image sequence E 1:N , N frames of depth image sequence DEPTH 1:N , N frames of semantic image sequence SEG 1:N And the normal image sequence NORMAL of N frames 1:N Splicing is performed on the channel dimension and uniformly represented as the input image sequence X 1:N ={X 1 ,X i ,...,X N}, where X i ∈R H×W×15 , and introduce the time sequence position encoding as the noise variable Z t ∈N(0,1), the input image sequence X 1:N and the noise variable Z t Input to the image generator to get a generated video sequence of N frames in

[0076] The image generator G consists of three parts: a mapping network, an image encoder, and a generation network. The mapping network is used to transform the noise variable Z t Transformed into a latent code W∈R for controlling style features 64 , the image feature encoder is used to encode multi-level image features, each layer of features is connected to the upper layer features, and the generation network is used to generate video frames based on image features and potential codes. In addition, the image generator adopts the UNet architecture as a whole. The image encoder part contains 3 layers and is sent to the generation network through jump connections. The image generator performs wavelet transform on the input image and then inputs it into the backbone network, where the backbone network refers to the part of the Unet architecture that removes the wavelet transform and inverse wavelet transform, and finally performs inverse wavelet transform to restore it to the original image size.

[0077] Among them, as the noise variable Z t The temporal position encoding of is obtained as follows:

[0078] NOISE(VI,FI)=PE(VI / VN)+PE(FI / FN)

[0079] PE(p)=(sin(p*2 0 ),cos(p*2 0 ),sin(p*2 1 ),cos(p*2 1 ),...,sin(p*2 k ),cos(p*2 k ))

[0080] Among them, VI represents the index information of the current video, FI represents the index information of the video frame in the current video, VN is the total number of videos, FN is the total number of frames in the current video where the video frame is located, NOISE(VI,FI) represents temporal position coding, PE(p) represents the coding of variable p, in PE(VI / VN), variable p=VI / VN, and in PE(FI / FN), variable p=FI / FN.

[0081] In the embodiment, the discriminator D is used to determine the authenticity of the input video frame, wherein the input video frame includes a real video frame V t The combination of color coding representation image sequence, eye gaze image sequence, depth image sequence, semantic image sequence and normal image sequence (V t ,CCBR t ,E t ,DEPTH t ,SEG t ,NORMAL t ), and generate video frames The combination of the generated digital human image and the color coding representation image sequence, eye gaze image sequence, depth image sequence, semantic image sequence and normal image sequence

[0082] Construct the adversarial learning loss function of the StyleUNet network, including generation loss and discrimination loss, where the generation loss includes the reconstruction loss L based on the difference between the generated digital human image and the real video frame rec , based on the image perceptual loss L between the image features of the generated digital human image and the image features of the real video frame sen , and the discriminator-based adversarial loss L adv ; The discriminant loss includes the discriminant loss L based on the discriminator D ;

[0083] Among them, the reconstruction loss L rec Using LI loss, image perception loss L sen The image features used can be the image features extracted by inputting the video frame into the VGG network. adv It is expressed as:

[0084]

[0085] Among them, fake 1,i Representation combination The result of the true or false discrimination input to the discriminator is the discrimination score.

[0086] Discriminative loss L D It is expressed as:

[0087]

[0088] Among them, real 1,i Indicates (V t ,CCBR t ,E t ,DEPTH t ,SEG t ,NORMAL t ) is input into the discriminator to determine the authenticity of the result, that is, the discrimination score.

[0089] In the embodiment, normal samples and difficult samples are constructed to perform two-stage adversarial training on the StyleUNet network using the above loss function to optimize network parameters, including:

[0090] Construct normal samples and difficult samples, clean and segment the original collected digital human training video data, extract key points, remove background, draw neural semantic images, and build a sample annotation system. Specifically, use PY-QT to build a data annotation system that can load image frames, such as Figure 5 As shown, the system reads monocular video frames and action clip sequences processed as above, and can perform frame review in frame order. For difficult samples, the reviewer uses the system to calibrate the start frame and end frame of the character motion clip. For erroneous samples, the key points of the erroneous sample clips are manually corrected by the system. For difficult samples, their indexes are used as input for subsequent sample supersampling. Specifically, normal samples refer to video frames in digital human materials that tend to be static and have no obvious hand and body movements. Difficult samples refer to video frames in video materials with hand and body movements, especially the intermediate state of character motion clips. The number of samples is less than that of normal samples, and motion blur is prone to occur. The model is more difficult to learn correctly, so it is called a difficult sample. The erroneous sample is a frame with a large difference between the video hand image result and the key point after the character key point is visualized.

[0091] In the pre-training stage, normal sample pairs are used to perform multiple iterative adversarial training on the StyleUNet network until the StyleUNet network as a whole fits to a better overall effect.

[0092] In the fine-tuning stage, normal samples are combined with difficult samples for training. Oversampling is used to increase the amount of difficult sample data several times in the normal training data, and adversarial training is performed on the pre-trained StyleUNet network. This can meet the high-quality reenactment of all actions and expressions of digital human videos.

[0093] After the above two-stage training, the trained image generator is used as an image generation model for subsequent digital human videos.

[0094] In the embodiment, the obtained image generation model is used to continuously generate digital human images, and audio is added to the continuous digital human images to obtain digital human videos.

[0095] like Figure 7 As shown, the embodiment also provides a multimodal feature fusion digital human video generation device 70 based on temporal position coding, including a key point extraction module 71, a model optimization module 72, a drawing module 73, and a video generation module 74, wherein the key point extraction module 71 is used to generate a key point sequence based on a monocular video and divide it into different action segments after preprocessing, and align the action segments based on the coefficient migration of SMPLX to obtain a redirected key point sequence; the model optimization module 72 is used to fit a new coefficient using the facial Faceverse coefficient detected by the Faceverse model, and use the new coefficient to replace the facial Flame coefficient of the SMPLX model, and use the Mano hand shape detected by the Hamer model to fit the new coefficient. The hand representation is obtained by replacing the hand representation of the SMPLX model with the new hand representation to obtain the optimized SMPLX model; the drawing module 73 is used to obtain a color-coded representation image sequence and an eye gaze image sequence based on the redirected key point sequence, and to obtain a depth image sequence, a semantic image sequence, and a normal image sequence based on the optimized SMPLX model; the video generation module 74 is used to introduce a temporal position coding for enhancing temporal consistency into the image generation model, and the temporal position coding includes the index information of the current video and the index information of the video frame in the current video, and to continuously generate multiple frames of digital human images based on the multimodal features formed by all image sequences, and to add audio to the continuous digital human images to obtain digital human videos.

[0096] It should be noted that the multimodal feature fusion digital human video generation device based on temporal position coding provided in the above embodiment should be illustrated by the division of the above functional modules when performing multimodal feature fusion digital human video generation. The above functional allocation can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the multimodal feature fusion digital human video generation device provided in the above embodiment and the multimodal feature fusion digital human video generation method embodiment belong to the same concept. The specific implementation process is detailed in the multimodal feature fusion digital human video generation method embodiment, which will not be repeated here.

[0097] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein executable code is stored in the memory, and when the one or more processors execute the executable code, the method for generating a digital human video by fusion of multimodal features is implemented, specifically including the following steps:

[0098] S1, generates a key point sequence based on the monocular video and divides it into different action segments after preprocessing, aligns the action segments based on the coefficient migration of SMPLX, and obtains the redirected key point sequence;

[0099] S2, use the facial Faceverse coefficients detected by the Faceverse model to fit the new coefficients, and use the new coefficients to replace the facial Flame coefficients of the SMPLX model, use the Mano hand shape detected by the Hamer model to fit the new hand representation, and use the new hand representation to replace the hand representation of the SMPLX model to obtain the optimized SMPLX model;

[0100] S3, based on the redirected key point sequence, a color coding representation image sequence and an eye gaze image sequence are obtained, and based on the optimized SMPLX model, a depth image sequence, a semantic image sequence, and a normal image sequence are obtained;

[0101] S4, introduces temporal position coding for enhancing temporal consistency into the image generation model. The temporal position coding includes the index information of the current video and the index information of the video frame in the current video. At the same time, multiple frames of digital human images are continuously generated based on the multimodal features formed by all image sequences, and audio is added to the continuous digital human images to obtain digital human videos.

[0102] The computing device provided in the embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the multimodal feature fusion digital human video generation method based on temporal position coding described in S1-S4 above. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0103] Based on the same inventive concept, the embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned multimodal feature fusion digital human video generation method based on temporal position coding is implemented, which specifically includes the following steps:

[0104] S1, generates a key point sequence based on the monocular video and divides it into different action segments after preprocessing, aligns the action segments based on the coefficient migration of SMPLX, and obtains the redirected key point sequence;

[0105] S2, use the facial Faceverse coefficients detected by the Faceverse model to fit the new coefficients, and use the new coefficients to replace the facial Flame coefficients of the SMPLX model, use the Mano hand shape detected by the Hamer model to fit the new hand representation, and use the new hand representation to replace the hand representation of the SMPLX model to obtain the optimized SMPLX model;

[0106] S3, based on the redirected key point sequence, a color coding representation image sequence and an eye gaze image sequence are obtained, and based on the optimized SMPLX model, a depth image sequence, a semantic image sequence, and a normal image sequence are obtained;

[0107] S4, introduces temporal position coding for enhancing temporal consistency into the image generation model. The temporal position coding includes the index information of the current video and the index information of the video frame in the current video. At the same time, multiple frames of digital human images are continuously generated based on the multimodal features formed by all image sequences, and audio is added to the continuous digital human images to obtain digital human videos.

[0108] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0109] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating digital human video by multimodal feature fusion based on temporal position coding, characterized in that: The following steps are involved: Generate key point sequences based on monocular video and divide them into different action segments after preprocessing. Align the action segments based on SMPLX coefficient migration to obtain the redirected key point sequence. The new coefficients are fitted using the facial Faceverse coefficients detected by the Faceverse model, and the facial Flame coefficients of the SMPLX model are replaced by the new coefficients. The new hand representation is fitted using the Mano hand shape detected by the Hamer model, and the hand representation of the SMPLX model is replaced by the new hand representation to obtain the optimized SMPLX model. Based on the redirected key point sequence, a color coding representation image sequence and an eye gaze image sequence are obtained. Based on the optimized SMPLX model, a depth image sequence, a semantic image sequence, and a normal image sequence are obtained. A temporal position coding is introduced into the image generation model to enhance temporal consistency. The temporal position coding includes the index information of the current video and the index information of the video frame in the current video. At the same time, multiple frames of digital human images are continuously generated based on the multimodal features formed by all image sequences, and audio is added to the continuous digital human images to obtain digital human videos.

2. The method for generating digital human video by multimodal feature fusion based on temporal position coding according to claim 1, characterized in that: Preprocessing the key point sequence includes: performing linear interpolation processing and time series smoothing processing on the key point sequence.

3. The method for generating digital human video by multimodal feature fusion based on temporal position coding according to claim 1, characterized in that: The action clips are aligned based on the coefficient migration of SMPLX to obtain the redirected key point sequence, including: Calculate the SMPLX coefficient sequences in the source action segment and the target action segment respectively, and then replace the posture coefficients of the target action segment except the root node with the posture coefficients of the source action segment except the root node to obtain a new SMPLX coefficient sequence of the target action segment; The SmoothNet model is used to smooth the posture coefficient sequence in the new SMPLX coefficient sequence of the target action clip, and the key point sequence is calculated according to the smoothed new SMPLX coefficient sequence to obtain the redirected key point sequence.

4. The method for generating digital human video by multimodal feature fusion based on temporal position coding according to claim 1, characterized in that: The facial Faceverse coefficients detected by the Faceverse model are used to fit new coefficients, and the facial Flame coefficients of the SMPLX model are replaced with the new coefficients, including: using the facial Faceverse coefficients detected by the Faceverse model to calculate facial key points, fitting new coefficients to the facial key points, and replacing the facial Flame coefficients of the SMPLX model with the new coefficients; The Mano hand shape detected by the Hamer model is fitted with a new hand representation, and the hand representation of the SMPLX model is replaced by the new hand representation to obtain an optimized SMPLX model, including: using the Mano hand shape detected by the Hamer model, calculating the wrist joint axis angle of the Mano hand shape according to forward kinematics and inverse kinematics to form a new hand representation, and replacing the hand representation of the SMPLX model with the new hand representation.

5. The method for generating digital human video by multimodal feature fusion based on temporal position coding according to claim 1, characterized in that: The color coding representation image sequence and the eye gaze image sequence are obtained based on the redirected key point sequence drawing, including: inputting the redirected key point sequence into a neural network, drawing the redirected key point sequence based on a neural semantic image drawing method, and generating a color coding representation image sequence and an eye gaze image sequence.

6. The method for generating digital human video by multimodal feature fusion based on temporal position coding according to claim 1, characterized in that: The temporal position encoding is obtained by: NOISE(VI,FI)=PE(VI / VN)+PE(FI / FN) PE(p)=(sin(p*2) 0 ),cos(p*2 0 ),sin(p*2 1 ),cos(p*2 1 ),...,sin(p*2 k ),cos(p*2 k )) Among them, VI represents the index information of the current video, FI represents the index information of the video frame in the current video, VN is the total number of videos, FN is the total number of frames in the current video where the video frame is located, NOISE(VI,FI) represents temporal position coding, PE(p) represents the coding of variable p, in PE(VI / VN), variable p=VI / VN, and in PE(FI / FN), variable p=FI / FN.

7. The method for generating digital human video by multimodal feature fusion based on temporal position coding according to claim 1, characterized in that: The image generation model is built based on the StyleUNet network. The specific construction process is as follows; The StyleUNet network includes an image generator and a discriminator. The input of the image generator is the concatenation of a color coding representation image sequence, an eye gaze image sequence, a depth image sequence, a semantic image sequence, and a normal image sequence. It is used to generate a digital human image according to the input and temporal position coding. The discriminator is used to determine whether the input image is the digital human image output by the image generator and the authenticity of the real image. Construct the adversarial learning loss function of the StyleUNet network, including generation loss and discrimination loss, where the generation loss includes the reconstruction loss L based on the difference between the generated digital human image and the real video frame rec , based on the image perceptual loss L between the image features of the generated digital human image and the image features of the real video frame sen , and the discriminator-based adversarial loss L adv ; The discriminant loss includes the discriminant loss L based on the discriminator D ; The loss function is used to perform adversarial learning on the StyleUNet network to optimize the network parameters, and the image generator with optimized parameters is used as the image generation model.

8. A multi-modal feature fusion digital human video generation device based on temporal position coding, characterized in that: include: The key point extraction module is used to generate a key point sequence based on the monocular video and divide it into different action segments after preprocessing. The action segments are aligned based on the coefficient migration of SMPLX to obtain the redirected key point sequence; A model optimization module, which is used to fit new coefficients using the facial Faceverse coefficients detected by the Faceverse model, and replace the facial Flame coefficients of the SMPLX model with the new coefficients, fit new hand representations using the Mano hand shape detected by the Hamer model, and replace the hand representation of the SMPLX model with the new hand representation to obtain an optimized SMPLX model; A drawing module, which is used to draw a color coding representation image sequence and an eye gaze image sequence based on the redirected key point sequence, and to draw a depth image sequence, a semantic image sequence, and a normal image sequence based on the optimized SMPLX model; The video generation module is used to introduce temporal position coding for enhancing temporal consistency into the image generation model. The temporal position coding includes the index information of the current video and the index information of the video frame in the current video. At the same time, multiple frames of digital human images are continuously generated based on the multimodal features formed by all image sequences, and audio is added to the continuous digital human images to obtain digital human videos.

9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the multimodal feature fusion digital human video generation method based on temporal position coding according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the method for generating digital human video by multimodal feature fusion based on temporal position coding as described in any one of claims 1 to 7 is implemented.

Images