Virtual viewpoint synthesis method and system, medium, equipment and terminal

Abstract

The invention belongs to the technical field of three-dimensional display, and discloses a virtual viewpoint synthesis method and system, a medium, equipment and a terminal, and the method comprises the steps: synthesizing a virtual view through a reference color map and a depth map through 3D-Warping; performing hole filling on the virtual view with holes, artifacts and overlapping through a hole filling module; and carrying out overall optimization of image quality on the image after hole filling through an image optimization module, and outputting a virtual view. In the first stage of the virtual viewpoint repair model, multi-scale fused partial convolution is proposed on the basis of single-scale partial convolution, context information is better associated, the model detail repair capability is enhanced, and the resource transmission cost is reduced. According to the method, the local chromatic aberration can be remained due to the limitation of partial convolution after hole filling, the local chromatic aberration caused by partial convolution can be effectively solved while the problems of image overlapping and artifacts are removed by using the second-stage image quality optimization network, and the overall quality of the optimized image is improved.

Description

technical field [0001] The invention belongs to the technical field of three-dimensional display, and in particular relates to a virtual viewpoint synthesis method, system, medium, device and terminal. Background technique [0002] At present, with the continuous progress of 3D display technology, ordinary 2D video is gradually developing into 3D video. As a key technology of 3D display, depth image based rendering technology (Depth Image Based Rendering, DIBR) has the advantages of fast rendering speed, small amount of transmitted data, and less storage space. In addition, due to the quality of the depth map and the occlusion between objects, the drawn images still have problems such as holes, overlaps, and artifacts, which lead to the overall degradation of the virtual viewpoint image quality. To this end, many improved DIBR algorithms have been proposed for virtual viewpoint synthesis, including preprocessing the depth map before DIBR to reduce the number of holes, using...

AI Technical Summary

Problems solved by technology

[0005] Through the above analysis, the problems and defects of the existing technology are: (1) The existing traditional hole filling algorithm lacks semantic perception of image details

(2) Using the existing partial convolutional network for hole filling will lead to local chromatic aberration and blurring of the repaired image

(3) The existing overlapping and artifact removal methods will consume more storage resources on the one hand, and have certain limitations on the other hand, and the specific implementation is relatively complicated

Images