Multi-view Neural Human Prediction with an Implicit Differentiable Renderer for Facial Expression, Body Pose Shape, and Clothing Performance Capture

Abstract

The neural human performance capture framework (MVS-PERF) captures a person's skeletal, body shape and clothing displacements, and appearance from a calibrated multi-view image set. MVS-PERF addresses the ambiguity of predicting absolute positions in monocular human mesh recovery and mediates the volumetric representation from NeRF to animation-friendly performance capture. MVS-PERF includes three modules: a module that extracts feature maps from multi-view images and fuses them into features, a module that regresses the features onto naked human parameter vectors to generate an SMPL-X skin-tight body mesh including skeletal pose, body shape and facial expression, and a module that leverages neural radiation and deformation fields and infers clothing as displacements on a naked body using differentiable rendering. A clothed human body mesh is obtained by adding the interpolated displacement vectors to the SMPL-X skin-tight body mesh vertices. The obtained luminance field is used for free-view volumetric rendering of the input subject.