Method and device for generating dynamic human free viewpoint video based on neural network

Abstract

The present application proposes a method and device for generating a dynamic human body free-viewpoint video based on a neural network, which relates to the technical field of computer vision and computer graphics, wherein the method includes: reconstructing a pre-scan model of a single human body; Describe a single human body shot to obtain a sequence of RGB images; deform the pre-scan model so that the deformed pre-scan model matches each frame of RGB image; sample the pre-scan model, and define a hidden code at each sampling point, And jointly optimize the hidden code and network parameters based on the neural network; obtain any rendering perspective, and generate free-view video based on the arbitrary rendering perspective. As a result, a sequence of RGB images is captured based on multiple RGB cameras, and a temporally continuous and dynamic free-viewpoint video is generated based on this sequence, resulting in more realistic and dynamic rendering results.

Description

technical field [0001] The present application relates to the technical fields of computer vision and computer graphics, and in particular, to a method and device for generating a dynamic human free-viewpoint video based on a neural network. Background technique [0002] Dynamic human free-view video generation is a key problem in the fields of computer graphics and computer vision. High-quality human free-view video has broad application prospects and important application value in the fields of film and television entertainment, human body digitization and so on. However, high-quality, high-fidelity human free-view video acquisition usually relies on expensive laser scanners or multi-camera array systems to model the human body. Although the effect is more realistic, it also has some obvious shortcomings: first, The equipment is complex, and these methods often require the construction of multi-camera arrays; second, the speed is slow, and it often takes at least 10 minut...

AI Technical Summary

Problems solved by technology

However, high-quality, high-fidelity human body free-viewpoint video acquisition usually relies on expensive laser scanners or multi-camera array systems to model the human body. Although the effect is more realistic, there are also some obvious shortcomings: first, The equipment is complicated, and these methods often require the construction of multi-camera arrays; second, the speed is slow, and it often takes at least 10 minutes to several hours to reconstruct a 3D human body model, and the reconstruction sequence is even slower

Images