Method for synthesizing virtual viewpoint image based on implicit neural scene representation

Abstract

The invention discloses a method for synthesizing a virtual viewpoint image by using implicit neural scene representation on the basis of multi-view three-dimensional cross-view loss, and is suitable for the field of computer vision. The method comprises the following steps: acquiring an image data set which needs to generate a virtual viewpoint; preprocessing the training image data set, and performing feature point extraction and matching on the input training image data set based on a feature matching algorithm Sift in the preprocessing stage; processing the obtained training image data and the extracted feature point information, and inputting the processed training image data and extracted feature point information into a multi-layer perceptron network for training; inputting test image data into the trained multi-layer perceptron network, and then obtaining a tested rendered image through volume rendering; and generating a virtual viewpoint image based on the trained multilayer perceptron network. Therefore, the data volume of the neural network during training fitting scene representation is reduced, and centralized sampling is performed in combination with the image depth information, so that the operation speed and performance of the neural scene representation can be improved, and a high-quality virtual viewpoint image is generated.

Description

technical field [0001] The invention relates to a method for realizing the synthesis and roaming of virtual viewpoint images by using implicit neural scene representation on the basis of multi-view stereo cross-view loss, which is suitable for the field of computer vision. Background technique [0002] With the development of science and technology and the continuous improvement of living standards, panoramic video, interactive video, free-view video and other new video methods different from traditional two-dimensional video have gradually entered the public's field of vision. Most of the current so-called free-view video generation methods are to set up multiple cameras in the scene to shoot at the same time, which is inefficient. And in a general environment, it is impossible to place a camera at any viewpoint for shooting in a large-scale scene. Therefore, synthesizing a virtual viewpoint image at any position through a small number of input viewpoint images is a researc...

AI Technical Summary

Problems solved by technology

However, the implicit neural scene representation is very slow in training and rendering, and it is necessary to further improve the training speed of the network

Images