Face changing and replaying method and device based on three-dimensional face decomposition

Abstract

The invention provides a face changing and replaying method and device based on three-dimensional face decomposition, which have the characteristic that the same set of process can simultaneously achieve face changing and expression replaying of any person, and the method comprises the following steps: decomposing a given 2D image by applying three-dimensional deformation model fitting to obtain three-dimensional decomposition parameters: an ID shape, an expression and a posture; training an image conversion model, wherein the unified input of the model is a target image background Isur, a manipulated 3D face projection Ishp and a 3D replay face appearance image Iapp, and the output of the model is a generated face changing or replay image, the training loss of the model comprises the reconstruction loss Lrec of the constraint generation image similar to the target image in the training data and the identity loss Lid of the constraint generation image similar to the input image on the ID; weighing the two losses to form a final loss; optimizing the final loss to obtain a trained model; and performing three-dimensional face decomposition and face attribute recombination transformation on test data, and inputting the test data into the trained model to generate a face changing video and an expression replaying video.

Description

technical field [0001] The invention relates to the field of image processing, in particular to a method and device for changing faces and replaying based on three-dimensional face decomposition. Background technique [0002] The development of high-quality image generation and translation models has inspired many interesting applications of face manipulation, such as face swapping, face reenactment, and face attribute editing, etc. Facial processing technology has attracted much attention for its potential applications in entertainment, visual effects, online conferencing, virtual avatars, and more. Active research on these techniques not only improves the realism of synthetic faces but also helps to advance the development of forgery detection techniques. [0003] Face swapping and expression reenactment, such as figure 1 shown. For expression reenactment, related works include Face2face and NeuralTexture, both of which only manipulate facial expressions. A more genera...

AI Technical Summary

Problems solved by technology

It also has a variant that uses a compromise mouth region reenactment method to approximate expression ghosting in videos, but this method suffers from inconsistencies in mouth and jaw movements

In addition to lacking general applicability to both tasks, many current works struggle to preserve source person face shapes for face swapping

This is mainly because they directly generate synthetic images based on target facial shape representations, which in most works use facial keypoints and lack factorization capabilities.

Images