Virtual portrait video feature prediction model training method and virtual portrait video generation method

Abstract

The present invention provides a virtual portrait video feature prediction model training method and a virtual portrait video generation method, which relate to the field of computer vision technology. The virtual portrait video feature prediction model obtained by training the initial video feature prediction model can learn the ability to extract fine-grained action features, achieve fine control of complex actions, enhance the expressiveness of virtual characters in fine-grained action synthesis in subsequently generated virtual portrait videos, and reduce the possibility of distortion, blur or distortion of virtual portraits.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a virtual portrait video feature prediction model training method and a virtual portrait video generation method. Background Art

[0002] Existing virtual portrait generation methods are mainly divided into two categories: virtual character generation methods based on motion capture and virtual portrait generation methods based on deep learning.

[0003] The virtual character generation method based on motion capture generates virtual character animation by capturing the action data of real people. That is, markers or sensors are installed on the real person, and the position and posture data of the joint points are obtained and digitized through light capture or inertial capture equipment, so as to obtain a complete three-dimensional motion trajectory and realize the three-dimensional reconstruction of the human skeleton. Then the captured posture data of the joint points are matched and bound with the reconstructed human skeleton to drive the virtual character to make corresponding actions. After the action of the virtual character is generated, the physical processes such as lighting, shadows, texture mapping, reflection and refraction are rendered to generate realistic visual effects. This type of method requires the construction of a professional capture environment and professional operators and software. The real person needs to wear a large number of equipment or markers during motion capture, which will also affect the amplitude and naturalness of the movement. The subsequent data cleaning, noise reduction, binding and rendering work still require a lot of manual intervention, and the production cycle is long.

[0004] The deep learning-based virtual portrait generation method aims to generate a coherent and natural virtual portrait video using a single input image. The key point is not only to ensure that the generated result is consistent with the appearance of the input image, but also to require the virtual character's movements to be natural and smooth. In order to ensure the consistency of the appearance of the virtual portrait, it is usually necessary to introduce a special identity document (ID) retention mechanism, such as using a reference network (reference net) or an appearance encoder (Appearance Encoder) to encode the details in the input image (such as facial features, clothing texture, etc.), generate a high-dimensional reference image feature, and embed the reference image feature in the subsequent generation process for learning. In order to achieve motion control, high-level drive control signals such as action skeletons or posture key points are usually provided to guide the video generation model to generate action video sequences that conform to the drive control signals. With the reference image features and drive control signals, the video generation model can be driven to generate multiple frames of continuous virtual portrait videos. In order to ensure the temporal consistency between video sequence frames, some temporal modules are usually introduced, such as using a temporal convolutional network to capture the temporal dependency between input sequences, or using optical flow constraints between consecutive frames to make the generated video sequence consistent in action.

[0005] Although existing deep learning-based generation methods have simplified hardware dependence and have achieved good results, they are difficult to generate natural and flexible detailed movements, especially in high-precision detail processing, which is prone to distortion, blur or distortion. In addition, the explicit two-dimensional skeleton key point information representation is sparse, making it difficult to capture complex action features and difficult to describe with simple key point information, which increases the difficulty of learning detailed features for video generation models. Summary of the invention

[0006] The present invention provides a virtual portrait video feature prediction model training method and a virtual portrait video generation method, which are used to solve the defects existing in the related technology.

[0007] The present invention provides a virtual portrait video feature prediction model training method, comprising: Acquire a sample video sequence in a video sample and a reference portrait in the video sample; Based on the feature extraction module in the initial video feature prediction model, the initial potential features of the sample video sequence and the reference portrait are extracted, and the sample fine-grained action features and posture features of the sample video sequence are extracted; Adding sample noise to the initial latent feature to obtain a noise latent feature, and generating a prediction feature based on a generation module in the initial video feature prediction model by using the noise latent feature, the sample fine-grained motion feature and the posture feature; Based on the predicted features and the sample noise or the initial potential features, the initial video feature prediction model is trained to obtain a virtual portrait video feature prediction model; The predicted features are used to generate a virtual portrait video.

[0008] According to a virtual portrait video feature prediction model training method provided by the present invention, the feature extraction module based on the initial video feature prediction model extracts the initial potential features of the sample video sequence and the reference portrait, and extracts the sample fine-grained action features and posture features of the sample video sequence, and also includes: Based on the feature extraction module, extracting image appearance features from the reference portrait; The generating module based on the initial video feature prediction model generates prediction features by using the noise potential features, the sample fine-grained motion features and the posture features, and further includes: Based on the generation module, the prediction feature is generated by using the noise potential feature, the sample fine-grained motion feature, the posture feature and the image appearance feature.

[0009] According to a virtual portrait video feature prediction model training method provided by the present invention, the posture feature is extracted by the feature extraction module from the action skeleton sequence of the sample video sequence; and / or, The feature extraction module is also used to map the dimension of the sample fine-grained action feature to be the same as the dimension of the image appearance feature.

[0010] According to a virtual portrait video feature prediction model training method provided by the present invention, the generation module is based on the noise potential feature, the sample fine-grained action feature, the posture feature and the image appearance feature to generate the prediction feature, including: Based on the cross-attention mechanism, the sample fine-grained action features and the image appearance features are embedded in the generation module, and based on the generation module, the posture features are used to predict the noise in the noise latent features or the noise-free latent features corresponding to the noise latent features to generate the predicted features.

[0011] According to a virtual portrait video feature prediction model training method provided by the present invention, the initial video feature prediction model also includes a fine-grained splicing module; The fine-grained splicing module is used to splice the sample fine-grained action features and the image appearance features to obtain fine-grained splicing features; The method of embedding the sample fine-grained action features and the image appearance features into the generation module based on the cross attention mechanism includes: Based on the cross-attention mechanism, the fine-grained splicing features are embedded into the generation module.

[0012] According to a virtual portrait video feature prediction model training method provided by the present invention, the sample fine-grained action features include at least one of sample facial fine-grained features and sample hand fine-grained features; The sample facial fine-grained features include at least one of eye fine-grained features, lip fine-grained features and expression fine-grained features; The sample hand fine-grained features include at least one of posture features, shape features and orientation features of both hands.

[0013] The present invention also provides a method for generating a virtual portrait video, comprising: Obtaining a target video and a target portrait, and determining a virtual portrait video feature prediction model; the virtual portrait video feature prediction model is trained based on the virtual portrait video feature prediction model training method described above; Slice the target video according to the sequence length of the sample video sequence to obtain a plurality of video segments, and input each video segment and the target portrait into the virtual portrait video feature prediction model to obtain the target video features output by the virtual portrait video feature prediction model; The target video features corresponding to the multiple video clips are input into a time series prediction model to obtain a virtual portrait video output by the time series prediction model.

[0014] According to a method for generating a virtual portrait video provided by the present invention, the target video is segmented according to the sequence length of the sample video sequence to obtain a plurality of video segments, including: The target video is segmented with overlaps according to the sequence length of the sample video sequence to obtain the multiple video segments.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for training a virtual portrait video feature prediction model or the method for generating a virtual portrait video as described above is implemented.

[0016] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for training a virtual portrait video feature prediction model or the method for generating a virtual portrait video as described above is implemented.

[0017] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned virtual portrait video feature prediction model training methods or virtual portrait video generation methods.

[0018] The virtual portrait video generation method, device, electronic device and storage medium provided by the present invention first obtain a sample video sequence in a video sample and a reference portrait in the video sample; then extract the initial potential features of the sample video sequence and the reference portrait based on the feature extraction module in the initial video feature prediction model, and extract the sample fine-grained action features and posture features of the sample video sequence; thereafter, add sample noise to the initial potential features to obtain noise potential features, and based on the generation module in the initial video feature prediction model, use the noise potential features, sample fine-grained action features and posture features to generate prediction features; finally, based on the prediction features, and the sample noise or initial potential features, train the initial video feature prediction model to obtain a virtual portrait video feature prediction model. After training, the initial video feature prediction model can learn the ability to extract fine-grained action features, realize fine control of complex actions, enhance the expressiveness of virtual characters in fine-grained action synthesis in the subsequently generated virtual portrait video, and reduce the possibility of distortion, blur or distortion of the virtual portrait. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of the virtual portrait video feature prediction model training method provided by the present invention.

[0021] Figure 2 It is a schematic diagram of the overall process of processing the initial video feature prediction model provided by the present invention.

[0022] Figure 3 It is a flow chart of the method for generating a virtual portrait video provided by the present invention.

[0023] Figure 4 It is a partial schematic diagram of a certain frame of a virtual portrait in a virtual portrait video generated by an existing method.

[0024] Figure 5 It is a partial schematic diagram of a virtual portrait of the same frame in a virtual portrait video generated by the virtual portrait video generation method provided in an embodiment of the present invention.

[0025] Figure 6 It is a structural schematic diagram of a virtual portrait video feature prediction model training device provided by the present invention.

[0026] Figure 7It is a structural schematic diagram of the virtual portrait video generating device provided by the present invention.

[0027] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Figure 1 FIG. 1 is a flow chart of a method for training a virtual portrait video feature prediction model provided in an embodiment of the present invention. Figure 1 As shown, the method includes: S11, obtaining a sample video sequence in a video sample and a reference portrait in the video sample; S12, extracting initial potential features of the sample video sequence and the reference portrait based on a feature extraction module in the initial video feature prediction model, and extracting sample fine-grained motion features and posture features of the sample video sequence; S13, adding sample noise to the initial latent feature to obtain a noise latent feature, and generating a prediction feature based on a generation module in the initial video feature prediction model by using the noise latent feature, the sample fine-grained motion feature and the posture feature; S14, training the initial video feature prediction model based on the prediction feature and the sample noise or the initial potential feature to obtain a virtual portrait video feature prediction model; The predicted features are used to generate a virtual portrait video.

[0030] Specifically, the virtual portrait video feature prediction model training method provided in the embodiment of the present invention is executed by a virtual portrait video feature prediction model training device, which can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0031] The execution subject is used to train a virtual portrait video feature prediction model, through which the target video features for generating a virtual portrait video can be determined according to an input video clip and a target image.

[0032] First, step S11 is performed to obtain a sample video sequence in a video sample and a reference portrait in the video sample. The video sample may be an original video containing various human body movements of real people. After the original video is acquired, the original video may be preprocessed to obtain a video sample. The preprocessing operation may include adjusting the frame size, cropping, and the like. Then, a video clip is randomly sampled from the video sample as a sample video sequence, and a frame is randomly sampled from the video sample as a reference portrait to control the appearance consistency between the reference portrait and the real person in the sample video sequence.

[0033] Then, step S12 is executed to extract the initial latent features of the sample video sequence and the reference portrait using the feature extraction module in the initial video feature prediction model, and to extract the sample fine-grained motion features and posture features of the sample video sequence.

[0034] Here, the initial video feature prediction model may include a feature extraction module and a generation module, and various features may be extracted through the feature extraction module, including initial potential features of the sample video sequence and the reference portrait, and sample fine-grained motion features and posture features of the sample video sequence.

[0035] The feature extraction module may include an encoder of a variational autoencoder (VAE), a preprocessor, and a posture feature extraction module. The preprocessor may include DWPose, openpose, etc. The posture feature extraction module may include a PoseNet module, etc. The PoseNet module may include multiple convolutional layers.

[0036] The sample video sequence and the reference portrait both have corresponding initial latent features. The initial latent feature of the sample video sequence can be the representation of the sample video sequence in the latent space, which can be expressed as latent_video(n,f,c,h,w), where n is the batch size during training, f is the number of frames in the sample video sequence, c is the number of channels per frame in the sample video sequence, h is the height of each frame in the sample video sequence, and w is the width of each frame in the sample video sequence. The initial latent feature of the reference portrait can be the representation of the reference portrait in the latent space, which can be expressed as latent_ref(n,1,c,h,w).

[0037] The initial latent features of the sample video sequence and the reference portrait can both be extracted through the VAE encoder (Encoder), and the initial latent features of the sample video sequence and the reference portrait can be extracted through the encoders of two VAEs respectively.

[0038] The sample fine-grained motion features of the sample video sequence refer to the implicit features of the parts of the sample video sequence where the real person can produce detailed motions, such as face and hands, etc. The sample fine-grained motion features can be extracted from the sample video sequence by a preprocessor.

[0039] Posture features can be extracted from sample video sequences through a posture feature extraction module.

[0040] Thereafter, step S13 is executed, and the initial video feature prediction model may include a latent feature splicing module, and the sample video sequence and the initial latent features of the reference portrait may be spliced ​​by the latent feature splicing module to obtain latent splicing features to maintain the appearance consistency of the real person. The latent splicing features may be expressed as latent_merge(n,f+1,c,h,w).

[0041] Then, sample noise is added to the latent concatenation features to obtain noise latent features. The sample noise can be Gaussian noise or noise of other distributions, which is not specifically limited here.

[0042] Then, the generation module in the initial video feature prediction model is used to take the noise potential feature as input, and the prediction feature is generated by combining the sample fine-grained motion feature and posture feature. The prediction feature can be the first prediction result obtained by the generation module predicting the input noise, or the second prediction result obtained by the generation module predicting the noise-free potential feature.

[0043] If the predicted feature is the first prediction result, the noise potential feature can be denoised by the inverse process of adding sample noise to obtain a denoised result, and the denoised result is input into the decoder to generate a virtual portrait video.

[0044] If the prediction feature is the second prediction result, the second prediction result can be directly input into the decoder to generate a virtual portrait video.

[0045] Finally, step S14 is performed to train the initial video feature prediction model using the predicted features and the sample noise or the initial potential features, thereby obtaining a virtual portrait video feature prediction model.

[0046] Here, the process of training the initial video feature prediction model can be understood as a diffusion learning process of regarding the initial video feature prediction model as a diffusion model.

[0047] If the predicted feature is the first prediction result, the loss function can be calculated using the first prediction result and the sample noise, and the initial video feature prediction model can be iteratively trained using the loss function until the loss function converges or reaches a preset number of iterations, thereby obtaining a virtual portrait video feature prediction model. The preset number of iterations can be set as needed and is not specifically limited here.

[0048] If the predicted feature is the second prediction result, the second prediction result and the initial potential feature can be used to calculate the loss function, and the initial video feature prediction model can be iteratively trained using the loss function until the loss function converges or reaches a preset number of iterations to obtain a virtual portrait video feature prediction model.

[0049] It can be understood that in the initial video feature prediction model, the posture feature extraction module and the generation module in the feature extraction module are both learnable modules. When training the initial video feature prediction model, the structural parameters of the posture feature extraction module and the generation module need to be updated, and the structural parameters of the VAE and the preprocessor need to be frozen and not updated.

[0050] The virtual portrait video feature prediction model training method provided in the embodiment of the present invention first obtains a sample video sequence in a video sample and a reference portrait in the video sample; then, based on the feature extraction module in the initial video feature prediction model, extracts the initial potential features of the sample video sequence and the reference portrait, and extracts the sample fine-grained action features and posture features of the sample video sequence; thereafter, adds sample noise to the initial potential features to obtain noise potential features, and based on the generation module in the initial video feature prediction model, uses the noise potential features, sample fine-grained action features and posture features to generate prediction features; finally, based on the prediction features, and the sample noise or initial potential features, the initial video feature prediction model is trained to obtain a virtual portrait video feature prediction model. After training, the initial video feature prediction model can learn the ability to extract fine-grained action features, realize fine control of complex actions, enhance the expressiveness of virtual characters in fine-grained action synthesis in the subsequently generated virtual portrait video, and reduce the possibility of distortion, blur or distortion of the virtual portrait.

[0051] On the basis of the above embodiment, the feature extraction module based on the initial video feature prediction model extracts the initial potential features of the sample video sequence and the reference portrait, and extracts the sample fine-grained action features and posture features of the sample video sequence, and further includes: Based on the feature extraction module, extracting image appearance features from the reference portrait; The generating module based on the initial video feature prediction model generates prediction features by using the noise potential features, the sample fine-grained motion features and the posture features, and further includes: Based on the generation module, the prediction feature is generated by using the noise potential feature, the sample fine-grained motion feature, the posture feature and the image appearance feature.

[0052] Specifically, the feature extraction module in the initial video feature prediction model may also include a visual sensor, such as a CLIP (Contrastive Language-Image Pre-training) visual encoder. The visual sensor may be used to extract the image appearance features in the reference portrait. The image appearance features may be represented as feat_id(n,f,1,1024), where 1024 is the dimension of the image appearance features.

[0053] When the initial video feature prediction model is trained, the structural parameters of the visual sensor are also frozen and not updated.

[0054] Furthermore, when generating prediction features, the generation module can be used to generate prediction features using noise latent features, sample fine-grained motion features, posture features, and image appearance features. Here, the noise latent features can be used as input and combined with the sample fine-grained motion features, posture features, and image appearance features to generate prediction features.

[0055] In the embodiment of the present invention, the image appearance features are introduced to further improve the accuracy of the generated prediction features.

[0056] Based on the above embodiment, the posture feature is extracted by the feature extraction module from the action skeleton sequence of the sample video sequence; and / or, The feature extraction module is also used to map the dimension of the sample fine-grained action feature to be the same as the dimension of the image appearance feature.

[0057] Specifically, the posture feature can be extracted from the action skeleton sequence of the sample video sequence by the feature extraction module, and the initial video feature prediction model can also include an action skeleton sequence extraction module, through which the action skeleton sequence can be extracted from the sample video sequence. Furthermore, the action skeleton sequence can be input into the posture feature extraction module of the feature extraction module, and the posture feature can be extracted from the action skeleton sequence of the sample video sequence by the posture feature extraction module.

[0058] The feature extraction module may also include a mapping module, which may be a mapper for mapping the dimension of the sample fine-grained motion feature to be the same as the dimension of the image appearance feature, thereby facilitating the subsequent feature splicing and application of the generation module.

[0059] Here, the mapping modules are all learnable modules, and when the initial video feature prediction model is trained, the structural parameters of the mapping modules need to be updated.

[0060] In the embodiment of the present invention, the accuracy of the posture features can be improved by extracting posture features from the action skeleton sequence. The dimension of the sample fine-grained action features is mapped to the same dimension as the image appearance features by the feature extraction module, which can facilitate the application of subsequent feature splicing and generation modules.

[0061] Based on the above embodiment, the sample fine-grained action feature includes at least one of a sample face fine-grained feature and a sample hand fine-grained feature; The sample facial fine-grained features include at least one of eye fine-grained features, lip fine-grained features and expression fine-grained features; The sample hand fine-grained features include at least one of posture features, shape features and orientation features of both hands.

[0062] Specifically, the sample fine-grained action feature may include at least one of a sample fine-grained facial feature and a sample fine-grained hand feature. The sample fine-grained facial feature may include at least one of a fine-grained eye feature, a fine-grained lip feature, and a fine-grained expression feature. If the sample fine-grained facial feature may include a fine-grained eye feature, a fine-grained lip feature, and a fine-grained expression feature, the first mapping result obtained after the sample fine-grained facial feature passes through the first mapping module may be expressed as feat_face(n,f,3,1024).

[0063] The fine-grained features of the sample hands can be extracted through parameterized models such as MANO in the preprocessor, and can include at least one of the pose features, shape features, and orientation features of the hands. The pose features represent the joint rotation and position of the hands in 3D space, the shape features are used to control the individual differences of the hands (such as finger length, palm size), and the orientation features are used to control the overall direction of the hands.

[0064] If the sample hand fine-grained features may include posture features, shape features, and orientation features of both hands, the second mapping result obtained after the sample hand fine-grained features pass through the second mapping module may be expressed as feat_hand(n,f,2,1024).

[0065] In the embodiment of the present invention, by refining the fine-grained action features of the samples, fine control of complex actions can be further achieved.

[0066] On the basis of the above embodiment, the generation module uses the noise potential feature, the sample fine-grained action feature, the posture feature and the image appearance feature to generate the prediction feature, including: Based on the cross-attention mechanism, the sample fine-grained action features and the image appearance features are embedded in the generation module, and based on the generation module, the posture features are used to predict the noise in the noise latent features or the noise-free latent features corresponding to the noise latent features to generate the predicted features.

[0067] Specifically, when generating prediction features, the cross-attention mechanism can be used to embed sample fine-grained motion features and image appearance features into the generation module. The generation module can be U-Net, which can include a down sampling module and an up sampling module. The sample fine-grained motion features and image appearance features can be embedded in the down sampling module and the up sampling module respectively, so that fine-grained motion can be finely and accurately controlled, ensuring detail consistency and high-precision control during the generation process.

[0068] Furthermore, the generation module can be used to use the posture features to predict the noise in the noise potential features or the noise-free potential features corresponding to the noise potential features to generate prediction features. Here, the posture features can be fused with the input of the first convolutional layer in the downsampling module to assist the subsequent operation process of the generation model.

[0069] Based on the above embodiment, the initial video feature prediction model may further include a fine-grained splicing module, through which the sample fine-grained action features and image appearance features may be spliced ​​to obtain fine-grained splicing features. The fine-grained splicing features may be represented as feat_merge (n, f, 1+3+2, 1024).

[0070] Furthermore, using the cross-attention mechanism, the concatenated fine-grained concatenated features can be embedded into the down-sampling module and up-sampling module of the generation module respectively.

[0071] In the cross-attention mechanism, the query vector (Q) can be obtained by the noisy latent feature map, and the value vector (V) and key vector (K) can be obtained by fine-grained concatenated feature maps.

[0072] like Figure 2As shown, it is a schematic diagram of the overall process of processing the initial video feature prediction model. The feature extraction module of the initial video feature prediction model includes a first encoder of the first VAE, a second encoder of the second VAE, a visual sensor, a preprocessor, a posture feature extraction module, a first mapping module, a second mapping module, a fine-grained splicing module (C1), a potential feature splicing module (C2) and an action skeleton sequence extraction module.

[0073] The input of the feature extraction module includes a sample video sequence and a reference portrait. The initial latent features of the sample video sequence are extracted by the encoder of the first VAE, and the initial latent features of the reference portrait are extracted by the encoder of the second VAE. The initial latent features of the sample video sequence and the reference portrait are spliced ​​by the latent feature splicing module to obtain the latent splicing features.

[0074] The image appearance features of the reference portrait are extracted by a visual sensor, and the sample facial fine-grained features and the sample hand fine-grained features in the sample video sequence are respectively extracted by a preprocessor. The dimensions of the sample facial fine-grained features are mapped to be the same as the dimensions of the image appearance features by a first mapping module to obtain a first mapping result. The dimensions of the sample hand fine-grained features are mapped to be the same as the dimensions of the image appearance features by a second mapping module to obtain a second mapping result.

[0075] The first mapping result, the second mapping result and the image appearance feature are spliced ​​together through a fine-grained splicing module to obtain a fine-grained splicing feature.

[0076] The action skeleton sequence is extracted from the sample video sequence by the action skeleton sequence extraction module, and the posture feature is extracted from the action skeleton sequence by the posture feature extraction module.

[0077] Sample noise is added to the initial latent feature to obtain the noise latent feature, and the latent splicing feature, posture feature and fine-grained splicing feature are input into the generation module to obtain the predicted feature.

[0078] like Figure 3 As shown, based on the above embodiment, an embodiment of the present invention further provides a method for generating a virtual portrait video, including: S21, obtaining a target video and a target portrait, and determining a virtual portrait video feature prediction model; the virtual portrait video feature prediction model is trained based on the virtual portrait video feature prediction model training method provided in the above embodiments; S22, segmenting the target video according to the sequence length of the sample video sequence to obtain a plurality of video segments, and inputting each video segment and the target portrait into the virtual portrait video feature prediction model to obtain the target video features output by the virtual portrait video feature prediction model; S23, inputting the target video features corresponding to the multiple video clips into a time series prediction model to obtain a virtual portrait video output by the time series prediction model.

[0079] Specifically, the virtual portrait video generation method provided in the embodiment of the present invention has a virtual portrait video generation device as its execution subject, which can be configured in a computer, which can be a local computer or a cloud computer, and the local computer can be a computer, a tablet, etc., which is not specifically limited here. The execution subject is used to apply the virtual portrait video feature prediction model, combined with the time series prediction model to generate a virtual portrait video, in which the virtual portrait in the virtual portrait video is consistent with the target portrait, and the action is consistent with the action of the real portrait in the target video, that is, the real portrait in the target video is converted into the target portrait, so as to realize the generation of the virtual portrait video.

[0080] First, step S21 is executed to obtain a target video and a target portrait. The target video can provide a desired action posture of a virtual portrait for generating a virtual portrait video, and the target portrait can provide a desired virtual portrait for generating a virtual portrait video.

[0081] Determine a virtual portrait video feature prediction model, which is trained using the virtual portrait video feature prediction model training method provided in the above embodiments, and has the ability to extract fine-grained action features, posture features, and potential features of video clips and target portraits in video clips with the same sequence length as the sample video sequence, and can also combine fine-grained action features, posture features, and potential features of video clips and target portraits to generate target video features.

[0082] Then, step S22 is performed to segment the target video according to the sequence length of the sample video sequence to obtain a plurality of video segments. Here, the length of each video segment is equal to the sequence length of the sample video sequence, for example, 40 frames.

[0083] Each video clip and the target person portrait are input into the virtual person portrait video feature prediction model, and the virtual person portrait video feature prediction model can obtain and output the target video feature. The target video feature can be the video noise predicted by the virtual person portrait video feature prediction model, or it can be the video feature predicted by the virtual person portrait video feature prediction model.

[0084] Finally, step S23 is executed to input the target video features corresponding to the multiple video clips into the time series prediction model, and the virtual portrait video is obtained and output through the time series prediction model.

[0085] The time series prediction model may include a time series prediction unit and a decoder. The time series prediction unit may generate virtual portrait video segment features corresponding to each video segment according to the target video features corresponding to each video segment, and the decoder may obtain each virtual portrait video segment of the virtual portrait video. The virtual portrait video segments may be spliced ​​to obtain the virtual portrait video.

[0086] Here, the decoder may be a decoder of VAE.

[0087] The virtual portrait video generation method provided in the embodiment of the present invention first obtains the target video and the target portrait, and determines the virtual portrait video feature prediction model; then the target video is segmented according to the sequence length of the sample video sequence to obtain multiple video clips, and each video clip and the target portrait are input into the virtual portrait video feature prediction model to obtain the target video features output by the virtual portrait video feature prediction model; finally, the target video features corresponding to the multiple video clips are input into the time series prediction model to obtain the virtual portrait video output by the time series prediction model. The method adopts the virtual portrait video feature prediction model, which can realize the fine control of complex actions, enhance the expressiveness of the virtual character in the generated virtual portrait video in fine-grained action synthesis, and reduce the possibility of distortion, blur or distortion of the virtual portrait.

[0088] On the basis of the above embodiment, the target video is segmented according to the sequence length of the sample video sequence to obtain multiple video segments, including: The target video is segmented with overlaps according to the sequence length of the sample video sequence to obtain the multiple video segments.

[0089] Specifically, when a longer video sequence is generated through a time series prediction model, there will be discontinuity and jump problems between the generated virtual video segments. Therefore, an overlapping reasoning strategy is adopted in the embodiment of the present invention. The target video is first segmented with overlap according to the sequence length of the sample video sequence to obtain multiple video segments. There are overlapping frames between each adjacent video segment, and the overlapping frames can be set as needed, for example, it can be expressed as m (m < n).

[0090] Then, the time series prediction model generates the i+1th virtual portrait video clip s i+1 When , the i-th virtual portrait video clip s can be used i Generate the last m frames to initialize s i+1 The first m frames of s0 are not replaced. That is, these m frames are not de-noised from the noise, but are inferred from the video features after de-noising by the virtual portrait video feature prediction model, which can ensure smooth transition of overlapping actions. At the same time, these m frames will continue to participate in s i+1In the timing prediction, other frames will refer to the information of the previous m frames, thereby effectively avoiding the jump and jitter problems in action generation and ensuring the smoothness and natural transition of actions in long time sequences.

[0091] Figure 4 is a partial schematic diagram of a certain frame of a virtual portrait in a virtual portrait video generated by an existing method, Figure 5 It is a partial schematic diagram of a virtual portrait of the same frame in a virtual portrait video generated by the virtual portrait video generation method provided in an embodiment of the present invention. Figure 5 Compared to Figure 4 , the effect of hand detail processing has been greatly improved.

[0092] In addition, the virtual portrait video generated by the virtual portrait video generation method provided in the embodiment of the present invention has also been greatly improved in the effect of facial detail processing.

[0093] like Figure 6 As shown, based on the above embodiment, an embodiment of the present invention provides a virtual portrait video feature prediction model training device, comprising: A first acquisition module 61, used to acquire a sample video sequence in a video sample and a reference portrait in the video sample; A feature extraction module 62, configured to extract initial latent features of the sample video sequence and the reference portrait based on the feature extraction module in the initial video feature prediction model, and to extract sample fine-grained motion features and posture features of the sample video sequence; A generating module 63, configured to add sample noise to the initial latent feature to obtain a noise latent feature, and generate a prediction feature based on the generating module in the initial video feature prediction model by using the noise latent feature, the sample fine-grained motion feature and the posture feature; A training module 64 is used to train the initial video feature prediction model based on the prediction feature and the sample noise or the initial potential feature to obtain a virtual portrait video feature prediction model; The predicted features are used to generate a virtual portrait video.

[0094] On the basis of the above-mentioned embodiment, in the virtual portrait video feature prediction model training device provided in the embodiment of the present invention, the feature extraction module is specifically used for: Based on the feature extraction module, extracting image appearance features from the reference portrait; The generating module based on the initial video feature prediction model generates prediction features by using the noise potential features, the sample fine-grained motion features and the posture features, and further includes: Based on the generation module, the prediction feature is generated by using the noise potential feature, the sample fine-grained motion feature, the posture feature and the image appearance feature.

[0095] On the basis of the above-mentioned embodiment, in the virtual portrait video feature prediction model training device provided in the embodiment of the present invention, the posture feature is extracted by the feature extraction module from the action skeleton sequence of the sample video sequence; and / or, The feature extraction module is also used to map the dimension of the sample fine-grained action feature to be the same as the dimension of the image appearance feature.

[0096] On the basis of the above-mentioned embodiment, in the virtual portrait video feature prediction model training device provided in the embodiment of the present invention, the generation module is specifically used for: Based on the cross-attention mechanism, the sample fine-grained action features and the image appearance features are embedded in the generation module, and based on the generation module, the posture features are used to predict the noise in the noise latent features or the noise-free latent features corresponding to the noise latent features to generate the predicted features.

[0097] On the basis of the above-mentioned embodiment, in the virtual portrait video feature prediction model training device provided in the embodiment of the present invention, the initial video feature prediction model further includes a fine-grained splicing module; The fine-grained splicing module is used to splice the sample fine-grained action features and the image appearance features to obtain fine-grained splicing features; The generating module is also specifically used for: Based on the cross-attention mechanism, the fine-grained splicing features are embedded into the generation module.

[0098] On the basis of the above-mentioned embodiment, in the virtual portrait video feature prediction model training device provided in the embodiment of the present invention, the sample fine-grained action feature includes at least one of a sample facial fine-grained feature and a sample hand fine-grained feature; The sample facial fine-grained features include at least one of eye fine-grained features, lip fine-grained features and expression fine-grained features; The sample hand fine-grained features include at least one of posture features, shape features and orientation features of both hands.

[0099] Specifically, the functions of each module in the virtual portrait video feature prediction model training device provided in the embodiment of the present invention correspond one-to-one to the operating procedures of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further details will be given in the embodiment of the present invention.

[0100] like Figure 7As shown, based on the above embodiment, an embodiment of the present invention provides a virtual portrait video generating device, including: The second acquisition module 71 is used to acquire the target video and the target portrait, and determine a virtual portrait video feature prediction model; the virtual portrait video feature prediction model is trained based on the virtual portrait video feature prediction model training method provided in the above embodiments; A feature prediction module 72 is used to segment the target video according to the sequence length of the sample video sequence to obtain multiple video segments, and input each video segment and the target portrait into the virtual portrait video feature prediction model to obtain the target video features output by the virtual portrait video feature prediction model; The time series prediction module 73 is used to input the target video features corresponding to the multiple video clips into the time series prediction model to obtain the virtual portrait video output by the time series prediction model.

[0101] On the basis of the above-mentioned embodiment, in the virtual portrait video generating device provided in the embodiment of the present invention, the feature prediction module is specifically used for: The target video is segmented with overlaps according to the sequence length of the sample video sequence to obtain the multiple video segments.

[0102] Specifically, the functions of each module in the virtual portrait video generating device provided in the embodiment of the present invention correspond one-to-one to the operation flow of each step in the above method embodiment, and the effects achieved are also consistent. Please refer to the above embodiment for details, which will not be repeated in the embodiment of the present invention.

[0103] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the virtual portrait video feature prediction model training method or the virtual portrait video generation method provided in the above embodiments.

[0104] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0105] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the virtual portrait video feature prediction model training method or the virtual portrait video generation method provided in the above embodiments provided by the above methods.

[0106] In another aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the virtual portrait video feature prediction model training method or the virtual portrait video generation method provided in the above embodiments. The computer-readable storage medium may be a non-transitory computer-readable storage medium or a transient computer-readable storage medium, which is not specifically limited here.

[0107] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual portrait video feature prediction model training method, characterized in that: include: Acquire a sample video sequence in a video sample and a reference portrait in the video sample; Based on the feature extraction module in the initial video feature prediction model, the initial potential features of the sample video sequence and the reference portrait are extracted, and the sample fine-grained motion features and posture features of the sample video sequence are extracted; Adding sample noise to the initial latent feature to obtain a noise latent feature, and generating a prediction feature based on a generation module in the initial video feature prediction model by using the noise latent feature, the sample fine-grained motion feature and the posture feature; Based on the predicted features and the sample noise or the initial potential features, the initial video feature prediction model is trained to obtain a virtual portrait video feature prediction model; The predicted features are used to generate a virtual portrait video.

2. The virtual portrait video feature prediction model training method according to claim 1, characterized in that: The feature extraction module based on the initial video feature prediction model extracts the initial potential features of the sample video sequence and the reference portrait, and extracts the sample fine-grained action features and posture features of the sample video sequence, and also includes: Based on the feature extraction module, extracting image appearance features in the reference portrait; The generating module based on the initial video feature prediction model generates prediction features by using the noise potential features, the sample fine-grained motion features and the posture features, and further includes: Based on the generation module, the prediction feature is generated by using the noise potential feature, the sample fine-grained motion feature, the posture feature and the image appearance feature.

3. The virtual portrait video feature prediction model training method according to claim 2, characterized in that: The posture feature is extracted by the feature extraction module from the action skeleton sequence of the sample video sequence; and / or, The feature extraction module is also used to map the dimension of the sample fine-grained action feature to be the same as the dimension of the image appearance feature.

4. The virtual portrait video feature prediction model training method according to claim 2, characterized in that: The generating module generates the prediction feature by using the noise potential feature, the sample fine-grained motion feature, the posture feature and the image appearance feature, including: Based on the cross-attention mechanism, the sample fine-grained action features and the image appearance features are embedded in the generation module, and based on the generation module, the posture features are used to predict the noise in the noise latent feature or the noise-free latent feature corresponding to the noise latent feature to generate the predicted feature.

5. The virtual portrait video feature prediction model training method according to claim 4, characterized in that: The initial video feature prediction model also includes a fine-grained splicing module; The fine-grained splicing module is used to splice the sample fine-grained action features and the image appearance features to obtain fine-grained splicing features; The method of embedding the sample fine-grained action features and the image appearance features into the generation module based on the cross attention mechanism includes: Based on the cross-attention mechanism, the fine-grained splicing features are embedded into the generation module.

6. The method for training a virtual portrait video feature prediction model according to any one of claims 1 to 5, characterized in that: The sample fine-grained action feature includes at least one of a sample face fine-grained feature and a sample hand fine-grained feature; The sample facial fine-grained features include at least one of eye fine-grained features, lip fine-grained features and expression fine-grained features; The sample hand fine-grained features include at least one of posture features, shape features and orientation features of both hands.

7. A method for generating a virtual portrait video, characterized in that: include: Obtain a target video and a target portrait, and determine a virtual portrait video feature prediction model; The virtual portrait video feature prediction model is trained based on the virtual portrait video feature prediction model training method according to any one of claims 1 to 6; Slice the target video according to the sequence length of the sample video sequence to obtain a plurality of video segments, and input each video segment and the target portrait into the virtual portrait video feature prediction model to obtain the target video features output by the virtual portrait video feature prediction model; The target video features corresponding to the multiple video clips are input into a time series prediction model to obtain a virtual portrait video output by the time series prediction model.

8. The method for generating a virtual portrait video according to claim 7, characterized in that: The step of segmenting the target video according to the sequence length of the sample video sequence to obtain a plurality of video segments includes: The target video is segmented with overlaps according to the sequence length of the sample video sequence to obtain the multiple video segments.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the virtual portrait video feature prediction model training method as described in any one of claims 1-6, or the virtual portrait video generation method as described in any one of claims 7-8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for training a virtual portrait video feature prediction model as described in any one of claims 1 to 6, or the method for generating a virtual portrait video as described in any one of claims 7 to 8 is implemented.

Images