Virtual viewpoint hole filling method based on deep background modeling

Abstract

The invention discloses a virtual viewpoint hole filling method based on deep background modeling. The method comprises the following steps: inputting a deep video and a texture video of a reference viewpoint; respectively carrying out background modeling on the deep video and the texture video of the reference viewpoint to obtain a depth background image and a texture background image; respectively mapping the depth video and the depth background image of the reference viewpoint and the texture video and the texture background image of the reference viewpoint by using a three-dimensional mapping method to obtain a depth video, a depth background image, a texture video and a texture background image of the virtual viewpoint; and finally, respectively carrying out depth hole filling on thedepth video and the depth background image of the virtual viewpoint, fusing the texture background image of the virtual viewpoint with the texture video of the virtual viewpoint to obtain a fused image of each frame of image in the texture video of the virtual viewpoint; and filling the residual cavity area in each frame of fusion image to obtain a first texture video of the virtual viewpoint. According to the method, the defect of time inconsistency of depth videos is overcome, and the virtual viewpoint perception quality after filling is good.

Description

technical field [0001] The invention relates to the field of hole filling in virtual viewpoint rendering, in particular to a virtual viewpoint hole filling method based on depth background modeling. Background technique [0002] With the vigorous development of computer vision and 3D video technology, multi-view video (MVV) and free view video (FVV) technologies emerged as the times require, allowing viewers to freely choose the viewing position and viewing angle. However, the current technology cannot guarantee continuous and synchronous capture of viewpoint information at any position and angle, and transmit such a large amount of data in real time, so it is necessary to use limited viewpoints to synthesize virtual viewpoints. [0003] The core of FVV is the virtual viewpoint rendering technology. The depth map-based rendering method (DIBR) using the multi-viewpoint plus depth video format (MVD) data format only needs to use a few or even one reference viewpoint to draw al...

AI Technical Summary

Problems solved by technology

However, the above methods all construct the background from the perspective of probability. The default assumption is that the background with a higher probability of occurrence in the time domain is the background, which is contrary to some scenarios in practical applications.

In addition, due to the poor quality of the depth map, the edge contour of the foreground may be unstable in the time domain. The background constructed by this method may produce certain artifacts in these areas, resulting in unnatural filling results.

Images