Image resynthesis using forward warping, gap discriminators, and coordinate-based inpainting

Abstract

The invention relates to image processing and, in particular, to image resynthesis for synthesizing new views of a person or an object based on an input image, to resolve tasks such as predicting views of a person or an object from new viewpoints and in new poses. Technical result consists in improved accuracy of image resynthesis based on at least one input image. An image resynthesis system, a system for training a gap filling module to be used in the image resynthesis system, an image resynthesis method, a computer program product and a computer-readable medium are provided. The image resynthesis system comprises a source image input module, a forward warping module configured to predict, for each source image pixel, a corresponding position in a target image, the forward warping module being configured to predict a forward warping field which is aligned with the source image, and a gap filling module configured to fill in the gaps resulting from the application of the forward warping module. The image resynthesis method comprises the steps of: inputting a source image, predicting, for each source image pixel, a corresponding position in a target image, wherein a forward warping field which is aligned with the source image is predicted, predicting a binary mask of gaps which result from the forward warping, filling in the gaps based on said binary mask of gaps by generating a texture image by means of predicting a pair of coordinates in the source image for each pixel in the texture image, and mapping the whole texture back to a new pose using backward warping.

Description

technical field [0001] The present invention relates generally to image processing and, more particularly, to image resynthesis for synthesizing new views of a person or object based on input images using machine learning techniques. Background technique [0002] Recently, there has been growing interest in learning-based image resynthesis. In this context, the task of machine learning is to learn to synthesize new views, eg of a particular type of person or object, based on one or more input images of the person or object. In extreme cases, only one input view is available. In this sense, new views correspond to new camera positions and / or new body poses of the person. In image resynthesis, the quality of the target view is measured and is not concerned with the quality of the intermediate representation, which usually corresponds implicitly or explicitly to the model (eg, 3D reconstruction) of the scene. Optimizing directly for target view quality usually means higher t...

AI Technical Summary

Problems solved by technology

In particular, a disadvantage of available prior art solutions relates to the use of backwarping in image resynthesis, where, for each pixel in the destination image, the position of a pixel in the source image will be copied in backwarping was predicted

Images