Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Synthetic video generation method based on three-dimensional face reconstruction and video key frame optimization

A technology of video key frame and 3D face, which is applied in neural learning methods, neural architecture, 3D image processing, etc., can solve the problems of low video frame quality, unnaturalness, and inability to obtain synthesized human face speech video, etc., to achieve high Quality portraits and backgrounds, the effect of labor cost reduction

Pending Publication Date: 2021-08-17
GUANGDONG UNIV OF TECH
View PDF19 Cites 27 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Although this method can also maintain the attribute characteristics of the original image and the identity characteristics of the person in the target image, the stability of this method to obtain a realistic synthetic face is not high, and it is impossible to obtain Synthetic face speech video, and the face background of the synthetic face video generated by this method is blurred, the video frame quality is low and unnatural

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Synthetic video generation method based on three-dimensional face reconstruction and video key frame optimization
  • Synthetic video generation method based on three-dimensional face reconstruction and video key frame optimization
  • Synthetic video generation method based on three-dimensional face reconstruction and video key frame optimization

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0069] A synthetic video generation method based on 3D face reconstruction and video keyframe optimization, such as figure 1 shown, including the following steps:

[0070] The convolutional neural network is used to optimize and fit the parameters of the three-dimensional face deformation model to the input face image, so as to realize the parametric reconstruction of the face model;

[0071] Use the parameters of the target video and face model to train the speech-to-expression and head pose mapping network H, and use the trained speech-to-expression and head pose mapping network H to obtain facial expression and head pose parameters from the input audio, so The target video includes a video frame and audio corresponding to the video frame, and the video frame includes a face image;

[0072] Replace the parameters in the parameterized face image according to the acquired facial expression and head posture parameters, synthesize and render the face image of each frame, and ge...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a synthetic video generation method based on three-dimensional face reconstruction and video key frame optimization. The synthetic video generation method comprises the following steps: optimizing and fitting each parameter of a three-dimensional face deformation model for an input face image by adopting a convolutional neural network; training a voice-to-expression and head posture mapping network by using the parameters of the target video and the face model, and acquiring facial expression and head posture parameters from the input audio by using the trained voice-to-expression and head posture mapping network; synthesizing a human face and rendering the synthesized human face to generate a vivid human face video frame; training a rendering network based on a generative adversarial network by using the parameterized face image and the face image in the video frame, wherein the rendering network is used for generating a background for each frame of face image; and performing face background rendering and video synthesis based on video key frame optimization. The background transition of each frame of the output synthesized face video is natural and vivid, and the usability and practicability of the synthesized face video can be greatly enhanced.

Description

technical field [0001] The present invention relates to the fields of three-dimensional face reconstruction and face synthesis migration in deep learning, and more specifically, relates to a synthetic video generation method based on three-dimensional face reconstruction and video key frame optimization. Background technique [0002] With the improvement of our country's social level, the popularization of mobile smart terminals and the rapid development of mobile Internet technology, video has become an indispensable part of people's life for learning, entertainment and work. Compared with traditional graphic representations, video Being able to combine hearing and vision, the production threshold is lower. At present, most of the applications of synthetic video are still in the entertainment field, such as face-changing photos in Meitu Xiuxiu, AR avatar production in iPhones, iSwap Faces and other applications. Most of these applications are essentially based on deep learn...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06T17/00G06T15/00G06K9/00G06K9/62G06N3/04G06N3/08
CPCG06T17/00G06T15/005G06N3/08G06T2200/04G06V40/161G06V40/168G06N3/045G06F18/214
Inventor 杨志景李为杰温瑞冕徐永宗李凯凌永权
Owner GUANGDONG UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com