Multimodal feature fusion digital human video generation method and device based on time sequence position coding

Abstract

The application discloses a kind of multi-modal feature fusion digital person video generation method and device based on timing position coding, comprising: constructing key point based on monocular video;Using the face Faceverse coefficient detected by Faceverse model, the face Flame coefficient of SMPLX model is fitted and replaced, the Mano hand shape detected using Hamer model, the hand representation of SMPLX model is fitted and replaced, to obtain the optimized SMPLX model;Color coding representation image and eye gaze image are obtained based on key point drawing, while based on the depth image, semantic image and normal image obtained by drawing the optimized SMPLX model;In image generation model, timing position coding for enhancing timing consistency is introduced, while based on the multi-modal feature formed by all images, a plurality of digital person images are continuously generated, and audio is added to obtain digital person video, which has wide application prospect in many fields.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of digital human generation, and particularly relates to a multi-modal feature fusion digital human video generation method and device based on time sequence position coding. BACKGROUND

[0002] With the rapid development of AI, AI digital human generation tasks are applied to various fields, including entertainment, education, medical treatment, and virtual reality. The generation of traditional graphics digital humans usually requires a long model construction and rendering process, involving complex three-dimensional modeling, texture mapping, and lighting calculation. These steps not only consume time but also require a high demand for computing resources. In contrast, the time period required for AI digital human modeling is significantly shortened, and it is more efficient, which makes it more and more favored in practical applications.

[0003] At present, the generation technology of AI digital humans has made significant progress. The diffusion-based method mainly uses some pre-trained diffusion models, such as the Stable Diffusion model. These models can generate high-quality, detailed digital human images through a step-by-step denoising process that converts random noise into clear images. Diffusion-based methods usually generate higher-quality images because their generation process allows more detailed feature learning and more complex image structure presentation. However, the inference speed of this process is slow, especially in application scenarios that require real-time feedback, which can become a bottleneck. The GAN-based method mainly uses some pre-trained GAN models for digital human generation. The core idea of GAN is to train two networks in an adversarial manner - the image generator and the discriminator. The image generator is responsible for generating images, while the discriminator is used to determine whether the generated images are real.

[0004] Compared with diffusion models, GAN models usually have an advantage in inference speed, which makes GAN-based models more effective in real-time generation tasks of AI digital humans. For example, in the scenarios of virtual anchors, online education assistants, intelligent customer service, etc., it is crucial to quickly generate and render high-quality digital human images. GAN models perform well in this regard, enabling fast response while maintaining high image quality. The StyleGAN model maps the input random noise to the latent style space, decouples features through the latent style space, and generates images with higher quality and stronger editability.

[0005] However, the above-mentioned network is mainly used for single-frame image generation and does not consider the temporal consistency between video frames. Generating digital human videos through a single-frame image generation model can easily cause video frame jitter. In addition, generating digital human images based on a single modality can result in low image quality. SUMMARY

[0006] In view of the above, the purpose of the present application is to provide a multi-modal feature fusion digital human video generation method and device based on timing position coding, which uses key points and SMPLX coefficient features extracted from a human video for modeling, relies on multi-modal feature fusion information to generate a specific identity, high-fidelity, and timing-continuous digital human video consistent with the input expression action, and can reduce the development difficulty of digital human related applications and reduce the workload of artificial modeling, and has a wide application prospect in many fields.

[0007] To achieve the above application purpose, the embodiment provides a multi-modal feature fusion digital human video generation method based on timing position coding, comprising the following steps:

[0008] After generating a key point sequence based on a monocular video and preprocessing, the key point sequence is divided into different action segments, and the action segments are aligned based on SMPLX coefficient migration to obtain a redirected key point sequence;

[0009] The face Faceverse coefficients detected by the Faceverse model are used to fit new coefficients, and the new coefficients are used to replace the face Flame coefficients of the SMPLX model, the Mano hand shape detected by the Hamer model is used to fit a new hand representation, and the new hand representation is used to replace the hand representation of the SMPLX model, to obtain an optimized SMPLX model;

[0010] Based on the redirected key point sequence, color-coded representation image sequences and eye gaze image sequences are drawn, and based on the optimized SMPLX model, depth image sequences, semantic image sequences, and normal image sequences are drawn;

[0011] A timing position coding for enhancing timing consistency is introduced into the image generation model, the timing position coding includes index information of a current video and index information of a video frame in the current video, and based on multi-modal features formed by all image sequences, a plurality of frames of digital human images are continuously generated, and audio is added to the continuous digital human images to obtain a digital human video.

[0012] Preferably, the preprocessing of the key point sequence includes linear interpolation processing and timing smoothing processing of the key point sequence.

[0013] Preferably, the action segments are aligned based on SMPLX coefficient migration to obtain a redirected key point sequence, comprising:

[0014] The SMPLX coefficient sequence in the source action segment and the target action segment is calculated respectively, and then the pose coefficient in the source action segment except the root node is replaced by the pose coefficient in the target action segment except the root node to obtain a new SMPLX coefficient sequence of the target action segment;

[0015] The pose coefficient sequence in the new SMPLX coefficient sequence of the target action segment is smoothed using the SmoothNet model, and a key point sequence is calculated according to the smoothed new SMPLX coefficient sequence to obtain a redirected key point sequence.

[0016] Preferably, the face Faceverse coefficient detected by the Faceverse model is used to fit the new coefficient, and the face Flame coefficient of the SMPLX model is replaced by the new coefficient, including: using the face Faceverse coefficient detected by the Faceverse model to calculate the face key point, fitting the new coefficient to the face key point, and replacing the face Flame coefficient of the SMPLX model with the new coefficient.

[0017] The Mano hand shape detected by the Hamer model is used to fit the new hand representation, and the hand representation of the SMPLX model is replaced by the new hand representation to obtain an optimized SMPLX model, including: using the Mano hand shape detected by the Hamer model, calculating the wrist joint axis angle of the Mano hand shape according to the forward kinematics and the inverse kinematics, forming the new hand representation, and replacing the hand representation of the SMPLX model with the new hand representation.

[0018] Preferably, the color-coded representation image sequence and the eye gaze image sequence are drawn based on the redirected key point sequence, including: inputting the redirected key point sequence into a neural network, drawing the redirected key point sequence based on a neural semantic image drawing method, and generating the color-coded representation image sequence and the eye gaze image sequence.

[0019] Preferably, the time sequence position encoding is obtained by the following way:

[0020] NOISE(VI,FI)=PE(VI / VN)+PE(FI / FN)

[0021] PE(p)=(sin(p*2 0 ),cos(p*2 0 ),sin(p*2 1 ),cos(p*2 1 ),...,sin(p*2 k ),cos(p*2 k ))

[0022] Among them, VI represents the index information of the current video, FI represents the index information of the video frame in the current video, VN is the total number of videos, FN is the total number of frames in the current video where the video frame is located, NOISE(VI,FI) represents temporal position coding, PE(p) represents the coding of the variable p, in PE(VI / VN), the variable p = VI / VN, and in PE(FI / FN), the variable p = FI / FN.

[0023] Preferably, the image generation model is constructed based on the StyleUNet network, and the specific construction process is as follows:

[0024] The StyleUNet network consists of an image generator and a discriminator. The image generator input is the splicing result of a color coding representation image sequence, an eye gaze image sequence, a depth image sequence, a semantic image sequence, and a normal image sequence. It is used to generate a digital human image based on the input and temporal position encoding. The discriminator is used to determine whether the input image is the digital human image output by the image generator or a real image.

[0025] Construct the adversarial learning loss function of the StyleUNet network, including generation loss and discrimination loss, where the generation loss includes the reconstruction loss L based on the difference between the generated digital human image and the real video frame rec , image perception loss L based on the difference between the image features of the generated digital human image and the image features of the real video frame sen , and the discriminator-based adversarial loss L adv ; The discriminant loss includes the discriminant loss L based on the discriminator D ;

[0026] The loss function is used to perform adversarial learning on the StyleUNet network to optimize the network parameters, and the image generator with optimized parameters is used as the image generation model.

[0027] To achieve the above-mentioned purpose, an embodiment of the present invention further provides a device for generating a digital human video by multimodal feature fusion based on temporal position coding, comprising:

[0028] The key point extraction module is used to generate a key point sequence based on the monocular video and divide it into different action segments after preprocessing. The action segments are aligned based on the coefficient migration of SMPLX to obtain the redirected key point sequence;

[0029] The model optimization module is used to fit new coefficients using the Faceverse coefficients detected by the Faceverse model, and replace the facial Flame coefficients of the SMPLX model with the new coefficients. The new hand representation is fit using the Mano hand shape detected by the Hamer model, and the new hand representation is used to replace the hand representation of the SMPLX model to obtain the optimized SMPLX model.

[0030] a drawing module configured to draw a color-coded representation image sequence and an eye gaze image sequence based on the redirected keypoint sequence, and draw a depth image sequence, a semantic image sequence, and a normal image sequence based on the optimized SMPLX model;

[0031] a video generation module configured to introduce a temporal position encoding for enhancing temporal consistency in the image generation model, the temporal position encoding including index information of a current video and index information of a video frame in the current video, and continuously generate multiple frames of digital human images based on multi-modal features formed by all image sequences, and add audio to the continuous digital human images to obtain a digital human video.

[0032] To achieve the above-mentioned purposes, the embodiments further provide a computing device, comprising a memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the above-mentioned digital human video generation method based on multi-modal feature fusion of temporal position encoding.

[0033] To achieve the above-mentioned purposes, the embodiments further provide a computer-readable storage medium having a program stored thereon, the program being executed by a processor to implement the above-mentioned digital human video generation method based on multi-modal feature fusion of temporal position encoding.

[0034] Compared with the prior art, the present application has at least the following beneficial effects:

[0035] By performing redirection of the extracted keypoint sequence based on SMPLX coefficient migration, the accuracy of the keypoint is improved, stable and accurate data is provided for subsequent adversarial learning and digital human video generation, and the ontology modeling of the digital human is realized. At the same time, the face coefficients and hand representation of the SMPLX model are optimized through the Faceverse model and the Hamer model to improve the accuracy of the three-dimensional human body model. On this basis, by fusing the features of the color-coded representation image, the eye gaze image, the depth image, the semantic image, and the normal image, and introducing a temporal position encoding for enhancing temporal consistency in the image sequence image generation model, the adversarial learning framework can pay more attention to the temporal features when the image generation model generates video frames, improve the fidelity of the temporal features, greatly improve the fidelity of the entire video frame, and can be generated continuously in time, greatly improving the generation efficiency of the digital human video. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0037] Figure 1 is a flowchart of the method for generating digital human video based on multi-modal feature fusion of timing position coding provided by the embodiments;

[0038] Figure 2 is a hierarchical diagram of the semantic graph drawing of neural rendering provided by the embodiments;

[0039] Figure 3 is a schematic diagram of color-coded representation of image, eye gaze image, depth image, semantic image and normal image provided by the embodiments;

[0040] Figure 4 is a comparison diagram of the effects before and after optimization of the SMPLX model provided by the embodiments;

[0041] Figure 5 is a flowchart of the sample generation provided by the embodiments;

[0042] Figure 6 is a structural schematic diagram of the adversarial learning framework provided by the embodiments;

[0043] Figure 7 is a structural schematic diagram of the device for generating digital human video based on multi-modal feature fusion of timing position coding provided by the embodiments. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0045] The inventive concept of the present application is that some technical problems exist in the existing digital human technology, such as the timing consistency problem of generated video, the generation speed and the generation quality, which need to be improved. In order to solve these technical problems, the present application provides a multi-modal feature fusion digital human video generation scheme based on timing position coding, and performs timing enhancement, and introduces a difficult sample learning method to further improve the high fidelity of digital human video, so as to better meet the requirements of user demand and application scenarios.

[0046] S1, the monocular video sequence is divided into different action segments after key point sequence generation and preprocessing, and the action segments are aligned based on the coefficient migration of SMPLX to obtain the redirected key point sequence.

[0047] In the embodiment, input N-frame monocular video sequence V 1 : N ={V 1 ,V i ,…,V N}, wherein V i ∈R H×W×3 , V i represents the i-th video frame, HxW is the video frame size, 3 represents the channel number, and the monocular video sequence is used to generate N-frame key point sequence L 1 : N ={L 1 ,L i ,…,L N}, wherein L i ∈R J ×3 , J is the number of key points, 3 is the three-dimensional coordinates of the key points, and L i represents the i-th frame corresponding key point sequence, and in actual extraction, L i ∈R 560×3 , specifically including 478 face key points, 8 torso key points, and 21 hand key points and 21 eye key points. Specifically, the Faceverse 3DMM model, the SMPLX model, and the Mediapipe tool can be used to extract face key points, torso and hand key points, and eye key points during key point extraction.

[0048] In the embodiment, in view of the possible key point missing and jittering problem after key point extraction, the key point sequence needs to be preprocessed, including: using a key point correction tool to manually correct the key points or using a linear interpolation method to correct the key points, and then using a one euro filter algorithm to perform temporal smoothing processing on the face, body, and hand key points, so as to smooth the small amplitude jitter while preserving the original data samples.

[0049] In the embodiment, in view of the subsequent real-time driving application requirement, the digital human needs to perform limb movement continuously in different action segments, therefore, the preprocessed key point sequence is divided into different action segments according to the action, and the redirection method based on the coefficient migration of SMPLX is used to align the action segments, to redirect the posture of the source action to the target action, and to obtain the redirected key point sequence. The specific process is as follows:

[0050] The SMPLX coefficient sequence in the source action segment and the target action segment is calculated respectively, and then the pose coefficient in the source action segment except the root node is replaced by the pose coefficient in the target action segment except the root node to obtain a new SMPLX coefficient sequence of the target action segment;

[0051] The pose coefficient sequence in the new SMPLX coefficient sequence of the target action segment is smoothed using the SmoothNet model, and a key point sequence is calculated according to the smoothed new SMPLX coefficient sequence to obtain a redirected key point sequence.

[0052] S2, the face Faceverse coefficient detected by the Faceverse model is used to fit a new coefficient, and the new coefficient is used to replace the face Flame coefficient of the SMPLX model, the Mano hand shape detected by the Hamer model is used to fit a new hand representation, and the new hand representation is used to replace the hand representation of the SMPLX model, to obtain an optimized SMPLX model.

[0053] In the embodiment, the SMPLX coefficient is optimized to solve the problem of low accuracy of the face and hand representation detected by the SMPLX model, including:

[0054] (1) The face Faceverse coefficient detected by the Faceverse model is used to fit a new coefficient, and the new coefficient is used to replace the face Flame coefficient of the SMPLX model. Specifically, the face Faceverse coefficient detected by the Faceverse model is used to calculate face key points, a new coefficient is fitted for the face key points, and the new coefficient is used to replace the face Flame coefficient of the SMPLX model; wherein the Faceverse model is one of 3D deformable parameterized models (3DMM).

[0055] More specifically, the face Faceverse coefficient is detected by the Faceverse model, the Faceverse coefficient is split into identity coefficient Id, expression coefficient Exp, texture coefficient Tex, rotation coefficient Angles, light coefficient Light, translation coefficient Trans and eye rotation coefficient Eye, wherein Id∈R 150 , Exp∈R 52 , Tex∈R 251 , Angles∈R 3 , Light∈R 3 , Trans∈R 3 , Eye∈R 4. The global rotation matrix Rot, the left eye rotation matrix Rot_LeftEye and the right eye rotation matrix Rot_RightEye are calculated according to the rotation coefficient Angles and the eye rotation coefficient Eye. The face key points are linearly combined by the identity base vector and the expression base vector:

[0056] V shape = IdBase * Id + ExpBase * Exp + MeanShape

[0057] wherein V shape ∈R 478 , represents the face key points in the canonical space, IdBase is the identity base vector of Faceverse, ExpBase is the expression base vector of Faceverse, and MeanShape is the average key point of Faceverse; the projected face key points V t are calculated using the rigid transformation V shape = Rot * V t + Trans, wherein V t ∈R 478 . According to the expression dictionary matrix D of the Flame model, the expression coefficient C of the Flame is inversely calculated by the formula V t = D * C and SparseCoder (a class in the python sklearn library), wherein C ∈R 100 . The expression coefficient C is replaced with the face Flame coefficient of SMPLX to obtain the optimized Flame coefficient;

[0058] (2) The Mano hand shape detected by the Hamer model is used to fit the new hand representation, and the new hand representation is used to replace the hand representation of the SMPLX model to obtain the optimized SMPLX model. Specifically, the Mano hand shape detected by the Hamer model is used to calculate the wrist joint axis angle of the Mano hand shape according to the forward kinematics and the inverse kinematics, and a new hand representation is formed. The new hand representation is used to replace the hand representation of the SMPLX model.

[0059] More specifically, the Mano hand shape of the digital human is detected using the Hamer model, the Pose (attitude) coefficients of the Mano left hand and the global rotation angles are symmetrical to the y-axis (that is, the x, z direction rotation is reversed), and the correct mapping to the Pose coefficients of SMPLX is obtained. The coordinate system of the right hand is consistent with the coordinate system of SMPLX, and no transformation is required. For the Mano global rotation angle, it is equivalent to the forward kinematics of the rotation from the root node (SMPLX global rotation angle) to the wrist joint in the SMPLX model. For the SMPLX model adopted, most of the joint rotation angles are reasonable under visualization, so the problem is constrained to only modify the wrist joint rotation R hand_local , which is expressed as follows:

[0060]

[0061] where R SMPLX_cum represents the rotation accumulation from the root node to the wrist joint, expressed in the form of the multiplication of rotation matrices, R Mano_hand_global is expressed as a global rotation matrix in the form of Mano, is expressed as the inverse transformation of R SMPLX_cum , and R is multiplied by R Mano_hand_global , that is, the new R hand_local is obtained. Taking the SMPLX left hand link as an example, the forward kinematics path is as follows:

[0062] The original global rotation angle of the left hand wrist joint is R hand_org = R Pelvis *R Spine1 *R Spine2 *R Spine3 *R Left_Collar *R Left_Shoulder *R Left_Elbow *R Left_Wrist , and the corrected wrist joint rotation angle of Mano is R hand_local = (R Pelvis *R Spine1 *R Spine2 *R Spine3 *R Left_Collar *R Left_Shoulder *R Left_Elbow ) -1 *R Mano_hand_global , where R Pelvis , R Spine1 , R Spine2 , R Spine3 , R Left_Collar , R Left_Shoulder , R Left_Elbow , R Left_Wristare rotation matrix representations of the angles of the pelvic joint, the spine joint 1, the spine joint 2, the spine joint 3, the left clavicle joint, the left shoulder joint, the left elbow joint, and the left wrist joint, respectively. Then, the joint angles of the Mano hand model are replaced by the corresponding joint angles of the SMPLX.

[0063] The effect comparison chart of the SMPLX model before and after optimization is shown in Figure 5 As can be seen from Figure 5 , compared with the normal image obtained by the original SMPLX model, the normal image obtained by the SMPLX model after optimization has higher hand accuracy.

[0064] S3, based on the redirected key point sequence, color coding representation image sequences and eye gaze image sequences are drawn, and based on the optimized SMPLX model, depth image sequences, semantic image sequences, and normal image sequences are drawn.

[0065] In the embodiment, the neural semantic image drawing method is used, and the N frame key point sequence L 1 : N is used as input to generate N frame color coding representation image sequences CC 1:N ={CCBR 1 ,CCBR i ,...,CCBR N}, wherein CCBR i ∈R H×W×3 and CCBR 1:N represents a color coding representation image, eye gaze image sequences E 1 ={E i ,E N ,...,E i}, wherein E H ∈R ×W×3 and E 1:N represents an eye gaze image. Using the SMPLX coefficient sequence after the coefficient migration of SMPLX, N frame depth image sequences DEPTH 1 ={DEPTH i ,DEPTH N ,...,DEPTH i} are generated, wherein DEPTH H×W×3 ∈R 1:N and DEPTH 1 represents a depth image, semantic image sequences SEG i ={SEG N ,SEG i ,...,SEG H×W×3}, wherein SEG 1:N ∈R and SEG represents a semantic image, and normal image sequences NORMAL={NORMAL 1 ,NORMAL i ,...,NORMAL N}, where NORMAL i ∈R H×W×3 And represents the normal image. Specifically, for the neural image rendering pipeline, such as Figure 2 As shown, it includes the following key sub-steps:

[0066] 1. SMPLX coefficient drawing

[0067] Use SMPLX coefficients to draw and get the depth image sequence DEPTH 1:N , semantic image sequence SEG 1:N , and the normal image sequence NORMAL 1:N ,like Figure 3 shown.

[0068] 2. Torso key point calculation

[0069] According to the SMPLX coefficient, the torso key points are calculated and interpolated. The torso key points contained in the key point sequence are interpolated to increase the number of torso key points.

[0070] Since increasing the number of skeleton points can improve the replay performance, interpolation is performed between 8 body skeleton points. The number and color of the interpolation between joint points are fixed. After interpolation, a total of 79 body posture key points are obtained.

[0071] 3. Image drawing overlay and control

[0072] In this step, the actual rendering process of the renderer and the external interface of the graphics rendering process are mainly defined. For example, the specific primitive drawing order of eye rendering and hand rendering is implemented. The confidence of the key points is used as the transparency, and the head image rendering, body image rendering, and eye image rendering are completed in sequence, and superimposed to obtain N frames of color-coded representation image sequence CCBR obtained by head image rendering and body image rendering. 1:N , eye gaze image sequence E obtained by eye image rendering 1:N ,like Figure 3 shown.

[0073] S4, introduces temporal position coding to enhance temporal consistency in the image generation model. The temporal position coding includes the index information of the current video and the index information of the video frame in the current video. At the same time, based on the multimodal features formed by all image sequences, multiple frames of digital human images are continuously generated, and audio is added to the continuous digital human images to obtain digital human videos.

[0074] In the embodiment, Figure 6As shown, the image generation model is constructed based on a StyleUNet network, and the StyleUNet network includes an image generator and a discriminator, wherein the image generator is used to generate a video frame sequence according to an input image sequence and a noise sequence controlled by a time sequence position coding, and the input image sequence is spliced by a color coding representation image sequence, an eye gaze image sequence, a depth image sequence, a semantic image sequence and a normal image sequence.

[0075] Specifically, the color coding representation image sequence CCBR of N frames 1:N , the eye gaze image sequence E of N frames 1:N , the depth image sequence DEPTH of N frames 1:N , the semantic image sequence SEG of N frames 1:N , and the normal image sequence NORMAL of N frames 1:N are spliced in the channel dimension, and are uniformly represented as an input image sequence X 1:N ={X 1 ,X i ,...,X N}, wherein X i ∈R H×W×15 , and a time sequence position coding is introduced as a noise variable Z t ∈N(0,1), the input image sequence X 1:N and the noise variable Z t are input to the image generator to obtain a generated video sequence of N frames , wherein

[0076] The image generator G includes a mapping network, an image encoder and a generation network. The mapping network is used to convert the noise variable Z t into a latent code W 64 for controlling style features, the image feature encoder is used to encode multi-level image features, each layer of features is connected with the upper layer of features, and the generation network is used to generate video frames according to the image features and the latent code. In addition, the image generator as a whole adopts a UNet architecture, and the image encoder part includes 3 layers in total, which are sent to the generation network through a skip connection. The image generator inputs the input image after wavelet transform into a backbone network, wherein the backbone network refers to the part of the Unet architecture except the wavelet transform and the inverse wavelet transform, and finally performs inverse wavelet transform to restore the original image size.

[0077] , wherein the time sequence position coding of the noise variable Z t is obtained by the following method:

[0078] NOISE(VI,FI)=PE(VI / VN)+PE(FI / FN)

[0079] PE(p) = (sin(p*2 0 ), cos(p*2 0 ), sin(p*2 1 ), cos(p*2 1 ),..., sin(p*2 k ), cos(p*2 k ))

[0080] wherein VI represents index information of a current video, FI represents index information of a video frame in the current video, VN represents a total number of videos, FN represents a total number of frames of the current video in which the video frame is located, NOISE(VI, FI) represents a time sequence position encoding, and PE(p) represents encoding of a variable p, wherein the variable p = VI / VN in PE(VI / VN), and the variable p = FI / FN in PE(FI / FN).

[0081] In the embodiment, the discriminator D is configured to determine whether an input video frame is real or fake, wherein the input video frame includes a real video frame V t and a combination of a color-coded representation image sequence, an eye gaze image sequence, a depth image sequence, a semantic image sequence, and a normal image sequence (V t , CCBR t , E t , DEPTH t , SEG t , NORMAL t ), and generate a video frame , i.e., a generated digital human image, and a combination of a color-coded representation image sequence, an eye gaze image sequence, a depth image sequence, a semantic image sequence, and a normal image sequence

[0082] The adversarial learning loss function of the StyleUNet network is constructed, including a generation loss and a discrimination loss, wherein the generation loss includes a reconstruction loss L rec based on a difference between the generated digital human image and the real video frame, an image perception loss L sen based on an image feature of the generated digital human image and an image feature of the real video frame, and an adversarial loss L adv based on the discriminator; and the discrimination loss includes a discrimination loss L D based on the discriminator.

[0083] wherein the reconstruction loss L rec adopts a LI loss, the image perception loss L sen adopts an image feature, which can be an image feature extracted by inputting the video frame into a VGG network. The adversarial loss L adv based on the discriminator is expressed as:

[0084]

[0085] Among them, fake 1,i Representation combination The true or false discrimination result input to the discriminator is the discrimination score.

[0086] Discriminative loss L D Expressed as:

[0087]

[0088] Among them, real 1,i Indicates (V t ,CCBR t ,E t ,DEPTH t ,SEG t ,NORMAL t ) is input into the discriminator to determine the authenticity of the result, that is, the discrimination score.

[0089] In the embodiment, normal samples and difficult samples are constructed and the StyleUNet network is trained in two stages using the above loss function to optimize network parameters, including:

[0090] Construct normal samples and difficult samples, clean and segment the original collected digital human training video data, extract key points, remove background, render neural semantic images, and build a sample annotation system. Specifically, use PY-QT to build a data annotation system that can load image frames, such as Figure 5 As shown, the system reads monocular video frames and action clip sequences that have been processed as described above, and can perform frame review in frame order. For difficult samples, the reviewer uses the system to calibrate the start and end frames of the character's motion clip. For erroneous samples, the key points of the erroneous sample clip are manually corrected through the system. For difficult samples, their indexes are used as input for subsequent sample oversampling. Specifically, normal samples refer to video frames in digital human materials that tend to be static and have no obvious hand and body movements. Difficult samples refer to video frames in video materials with hand and body movements, especially the intermediate states of character motion clips. The number of samples is less than that of normal samples, and motion blur is prone to occur. It is difficult for the model to learn correctly, so they are called difficult samples. Error samples are frames where the video hand image results and key points are significantly different after the character key points are visualized.

[0091] In the pre-training stage, normal samples are used to perform multiple iterative adversarial training on the StyleUNet network until the StyleUNet network as a whole fits to a better overall effect.

[0092] The fine-tuning stage trains the normal sample combined with the difficult sample. The pre-trained StyleUNet network is subjected to adversarial training by increasing the number of difficult sample data in the normal training data by several times through super sampling. In this way, the digital human video can meet the high-quality replay of all actions and expressions.

[0093] After the above two-stage training, the trained image generator is used as an image generation model for subsequent digital human video.

[0094] In the embodiment, the obtained image generation model is used to continuously generate digital human images, and audio is added to the continuous digital human images to obtain a digital human video.

[0095] As shown in Figure 7 The embodiment also provides a multi-modal feature fusion digital human video generation device 70 based on time sequence position coding. The device includes a key point extraction module 71, a model optimization module 72, a rendering module 73, and a video generation module 74. The key point extraction module 71 is used to generate a key point sequence based on a monocular video, and the key point sequence is preprocessed and divided into different action segments. The action segments are aligned based on SMPLX coefficient migration to obtain a redirected key point sequence. The model optimization module 72 is used to fit new coefficients using face coefficients detected by the Faceverse model, replace face Flame coefficients of the SMPLX model with the new coefficients, fit a new hand representation using a Mano hand shape detected by the Hamer model, replace a hand representation of the SMPLX model with the new hand representation, and obtain an optimized SMPLX model. The rendering module 73 is used to render color-coded representation image sequences and eye gaze image sequences based on the redirected key point sequence, and render depth image sequences, semantic image sequences, and normal image sequences based on the optimized SMPLX model. The video generation module 74 is used to introduce time sequence position coding for enhancing time sequence consistency in an image generation model. The time sequence position coding includes index information of a current video and index information of a video frame in the current video. The video generation module 74 is also used to continuously generate multiple frames of digital human images based on multi-modal features formed by all image sequences, and add audio to the continuous digital human images to obtain a digital human video.

[0096] It should be noted that the multi-modal feature fusion digital human video generation device based on time sequence position coding provided in the above embodiment should be illustrated based on the division of the above functional modules when performing multi-modal feature fusion digital human video generation. The above functions can be completed by different functional modules according to needs, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the multi-modal feature fusion digital human video generation device and the multi-modal feature fusion digital human video generation method provided in the above embodiment belong to the same concept, and the specific implementation process is described in detail in the multi-modal feature fusion digital human video generation method embodiment, which will not be repeated here.

[0097] Based on the same inventive concept, the embodiment also provides a computing device comprising a memory and one or more processors, the memory having stored therein executable code, the one or more processors, when executing the executable code, being configured to implement the multi-modal feature fusion digital human video generation method described above, specifically comprising the following steps:

[0098] S1, generating a key point sequence based on a monocular video and performing preprocessing, dividing the key point sequence into different action segments, aligning the action segments based on coefficient migration of SMPLX, and obtaining a redirected key point sequence;

[0099] S2, using the face Faceverse coefficients detected by the Faceverse model to fit new coefficients, and using the new coefficients to replace the face Flame coefficients of the SMPLX model, using the Mano hand shape detected by the Hamer model to fit a new hand representation, and using the new hand representation to replace the hand representation of the SMPLX model, to obtain an optimized SMPLX model;

[0100] S3, based on the redirected key point sequence, drawing color-coded representation image sequences and eye gaze image sequences, and based on the optimized SMPLX model, drawing depth image sequences, semantic image sequences, and normal image sequences;

[0101] S4, introducing a time sequence position coding for enhancing time sequence consistency in the image generation model, the time sequence position coding comprising index information of a current video and index information of a video frame in the current video, simultaneously based on multi-modal features formed by all image sequences, continuously generating multiple frames of digital human images, and adding audio to the continuous digital human images to obtain a digital human video.

[0102] The computing device provided by the embodiment, in addition to including a processor and a memory, also includes an internal bus, a network interface, a memory and other hardware required by business. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to realize the multi-modal feature fusion digital human video generation method based on time sequence position coding described in S1-S4. Of course, in addition to the software implementation mode, the present application does not exclude other implementation modes, such as logic devices or software and hardware combined mode and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but also can be hardware or logic device.

[0103] Based on the same inventive concept, the embodiment also provides a computer readable storage medium having a program stored thereon, which, when executed by a processor, implements the multi-modal feature fusion digital human video generation method based on time sequence position coding described above, and specifically includes the following steps:

[0104] S1, generating a key point sequence based on a monocular video and dividing it into different action segments after preprocessing, aligning the action segments based on the coefficient migration of SMPLX to obtain a redirected key point sequence;

[0105] S2, using the face coefficients detected by the Faceverse model to fit new coefficients, and using the new coefficients to replace the face coefficients of the SMPLX model, using the Mano hand shape detected by the Hamer model to fit a new hand representation, and using the new hand representation to replace the hand representation of the SMPLX model, to obtain an optimized SMPLX model;

[0106] S3, based on the redirected key point sequence, drawing to obtain a color-coded representation image sequence and an eye gaze image sequence, and based on the optimized SMPLX model, drawing to obtain a depth image sequence, a semantic image sequence, and a normal image sequence;

[0107] S4, introducing a time sequence position coding for enhancing time sequence consistency in the image generation model, the time sequence position coding including index information of the current video and index information of the video frame in the current video, simultaneously based on multi-modal features formed by all image sequences, continuously generating multiple frames of digital human images, and adding audio to the continuous digital human images to obtain a digital human video.

[0108] In the embodiment, the computer readable medium includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data.

[0109] The above detailed description of the specific embodiments of the present application has described the technical solutions and beneficial effects of the present application, and it should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modifications, supplements and equivalent replacements made within the principle range of the present application shall be included in the protection range of the present application.

Claims

1. A method for generating digital human videos by multimodal feature fusion based on temporal position coding, characterized in that: The following steps are involved: Generate key point sequences based on monocular video and divide them into different action segments after preprocessing. Align the action segments based on SMPLX coefficient migration to obtain the redirected key point sequence. The Faceverse coefficients detected by the Faceverse model are used to fit new coefficients, and the new coefficients are used to replace the facial Flame coefficients of the SMPLX model. The Mano hand shape detected by the Hamer model is used to fit a new hand representation, and the new hand representation is used to replace the hand representation of the SMPLX model to obtain the optimized SMPLX model. Based on the redirected key point sequence, a color-coded representation image sequence and an eye gaze image sequence are obtained. At the same time, based on the optimized SMPLX model, a depth image sequence, a semantic image sequence, and a normal image sequence are obtained. A temporal position coding method is introduced into the image generation model to enhance temporal consistency. The temporal position coding includes the index information of the current video and the index information of the video frame in the current video. At the same time, multiple frames of digital human images are continuously generated based on the multimodal features formed by all image sequences, and audio is added to the continuous digital human images to obtain digital human videos. The temporal position encoding is obtained as follows: NOISE(VI,FI)=PE(VI / VN)+PE(FI / FN) PE(p)=(sin(p*2) 0 ),cos(p*2 0 ),sin(p*2 1 ),cos(p*2 1 ),...,sin(p*2 k ),cos(p*2 k )) Wherein, VI represents the index information of the current video, FI represents the index information of the video frame in the current video, VN represents the total number of videos, FN represents the total number of frames in the current video where the video frame is located, NOISE(VI,FI) represents temporal position encoding, PE(p) represents encoding of the variable p, in PE(VI / VN), the variable p = VI / VN, and in PE(FI / FN), the variable p = FI / FN; Among them, the image generation model is built based on the StyleUNet network, and the specific construction process is as follows; The StyleUNet network includes an image generator and a discriminator. The image generator is used to generate a video frame sequence based on the input image sequence and the noise sequence controlled by temporal position coding, where the input image sequence is composed of a color coding representation image sequence, an eye gaze image sequence, a depth image sequence, a semantic image sequence and a normal image sequence; the discriminator D is used to determine the authenticity of the input video frame, where the input video frame includes a real video frame V t Combined with color coding representation image sequence, eye gaze image sequence, depth image sequence, semantic image sequence and normal image sequence, and generate video frames a combination of a color-coded representation image sequence, an eye gaze image sequence, a depth image sequence, a semantic image sequence, and a normal image sequence; Construct the adversarial learning loss function of the StyleUNet network, including generation loss and discrimination loss, where the generation loss includes the reconstruction loss L based on the difference between the generated digital human image and the real video frame rec , image perception loss L based on the difference between the image features of the generated digital human image and the image features of the real video frame sen , and the discriminator-based adversarial loss L adv ; The discriminant loss includes the discriminant loss L based on the discriminator D ; The loss function is used to perform adversarial learning on the StyleUNet network to optimize the network parameters, and the image generator with optimized parameters is used as the image generation model.

2. The method for generating digital human videos by multimodal feature fusion based on temporal position coding according to claim 1, characterized in that: The preprocessing of the key point sequence includes: performing linear interpolation processing and time series smoothing processing on the key point sequence.

3. The method for generating digital human videos by multimodal feature fusion based on temporal position coding according to claim 1, characterized in that: Align the action clips based on SMPLX coefficient migration to obtain the redirected key point sequence, including: Calculate the SMPLX coefficient sequences in the source action segment and the target action segment respectively, then replace the posture coefficients of the target action segment except the root node with the posture coefficients of the source action segment except the root node to obtain a new SMPLX coefficient sequence of the target action segment; The SmoothNet model is used to smooth the posture coefficient sequence in the new SMPLX coefficient sequence of the target action clip, and the key point sequence is calculated based on the smoothed new SMPLX coefficient sequence to obtain the redirected key point sequence.

4. The method for generating digital human videos by multimodal feature fusion based on temporal position coding according to claim 1, characterized in that: Using the facial Faceverse coefficients detected by the Faceverse model to fit new coefficients, and using the new coefficients to replace the facial Flame coefficients of the SMPLX model, including: using the facial Faceverse coefficients detected by the Faceverse model to calculate facial key points, fitting new coefficients to the facial key points, and using the new coefficients to replace the facial Flame coefficients of the SMPLX model; The Mano hand shape detected using the Hamer model is fitted with a new hand representation, and the hand representation of the SMPLX model is replaced with the new hand representation to obtain an optimized SMPLX model, including: using the Mano hand shape detected using the Hamer model, calculating the wrist joint axis angle of the Mano hand shape based on forward kinematics and inverse kinematics to form a new hand representation, and replacing the hand representation of the SMPLX model with the new hand representation.

5. The method for generating digital human videos by multimodal feature fusion based on temporal position coding according to claim 1, characterized in that: A color-coded representation image sequence and an eye gaze image sequence are obtained based on the redirected key point sequence, including: inputting the redirected key point sequence into a neural network, drawing the redirected key point sequence based on a neural semantic image drawing method, and generating a color-coded representation image sequence and an eye gaze image sequence.

6. A device for generating digital human videos by multimodal feature fusion based on temporal position coding, the device being used to execute the method according to any one of claims 1 to 5, characterized in that: include: The key point extraction module is used to generate a key point sequence based on the monocular video and divide it into different action segments after preprocessing. The action segments are aligned based on the coefficient migration of SMPLX to obtain the redirected key point sequence; The model optimization module is used to fit new coefficients using the Faceverse coefficients detected by the Faceverse model, and replace the facial Flame coefficients of the SMPLX model with the new coefficients. The new hand representation is fit using the Mano hand shape detected by the Hamer model, and the new hand representation is used to replace the hand representation of the SMPLX model to obtain the optimized SMPLX model. A rendering module is used to render a color-coded representation image sequence and an eye gaze image sequence based on the redirected key point sequence, and to render a depth image sequence, a semantic image sequence, and a normal image sequence based on the optimized SMPLX model; The video generation module is used to introduce temporal position coding to enhance temporal consistency in the image generation model. The temporal position coding includes the index information of the current video and the index information of the video frame in the current video. At the same time, based on the multimodal features formed by all image sequences, multiple frames of digital human images are continuously generated, and audio is added to the continuous digital human images to obtain digital human videos.

7. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the multimodal feature fusion digital human video generation method based on temporal position coding according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by the processor, the method for generating digital human videos by multimodal feature fusion based on temporal position coding according to any one of claims 1 to 5 is implemented.

Images