A new viewpoint synthesis method for scene reconstruction based on generating multi-plane images

Abstract

A new viewpoint synthesis method based on generating multi-plane image scene reconstruction uses Transformer as the backbone of the network and reconstructs the scene through multi-plane images, overcoming the problem of limited receptive field of previous convolutional network-based generation of multi-plane scene representation. Transformer can capture long-distance dependent features and overcome the problem of stacking ghosting when viewpoint extrapolation. At the same time, through long-distance dependent features, some de-occluded parts can also be well inferred; the introduction of Transformer's ability to capture global features, combined with the local features of the convolutional network, retains the inherent inductive bias of convolution, which is conducive to the network retaining as many required features as possible, making the network have better generalization; hybrid Transformer and convolutional network can enable multi-plane scene representation to be applied to more real scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a new viewpoint synthesis method based on generating multi-plane image scene reconstruction. Background Art

[0002] Reconstructing scenes from sparse input views has been a long-standing goal in the field of imaging, and this work plays a very important role in novel viewpoint synthesis. However, the input views are too sparse and have occlusion problems, which leads to many challenges in scene reconstruction. In addition, the traditional voxel representation of scenes is costly, takes a long time to render, is inefficient, and is difficult to achieve real-time inference rendering, and has low usability in actual engineering applications. In recent years, the implicit scene representation of neural radiance fields has made great breakthroughs, but this representation requires dense input images, has high training costs, and has low generalization. Explicit scene representation based on multi-plane images has the potential to overcome the above challenges. This new scene representation is not only low-cost, but can also be combined with image-based rendering (IBR), greatly reducing rendering time and even potentially achieving real-time rendering, which has a wide range of engineering applications.

[0003] A multi-plane image (MPI) consists of a set of fronto-parallel, semi-transparent images within the reference camera's frustum. Each layer consists of four values: RGB (red, green, and blue) and transparency (alpha). MPI is a powerful scene description that can handle diffuse surfaces and non-Lambertian effects, such as transparent and specular areas. Existing methods for generating MPIs are mostly based on convolutional networks (CNNs) or multi-layer perceptron networks (MLPs). The limited receptive field of convolutional networks leads to stacking artifacts when using MPIs for viewpoint extrapolation. This also presents problems when synthesizing new viewpoints on small objects or specular surfaces. Multi-layer perceptron networks primarily introduce implicit ideas from neural radiance fields into MPI-based scene representations. This can address viewpoint-dependent visual effects, such as the rainbow effect in CD reflections and specular highlights. It also provides some improvements in rendering small structures. However, stacking artifacts still occur when the extrapolation distance is too wide, and rendering errors can occur in areas such as glints, sharp highlights, and refraction in test tubes.

[0004] The main problems with the existing technology are:

[0005] 1. Most existing networks are based on convolutional networks and multi-layer perceptron networks. Due to the limited receptive field of existing networks, all generated multi-plane scene representations will produce stacked ghosting when rendering new viewpoint images over a wide range of viewpoints, and the reconstruction quality of the de-occluded parts is poor.

[0006] 2. Existing technologies still have many problems in reconstructing the effects of flashes, highlights, refractions, and other areas that depend on viewpoints. The implicit ideas introduced require more input viewpoints to optimize network parameters, which will require more computational loss and reduce rendering quality. Summary of the Invention

[0007] To address the challenges of the aforementioned background technologies, we propose a new viewpoint synthesis method for scene reconstruction based on the generation of multi-plane images. Combining CNNs with Transformer networks, this method retains the CNN's inductive bias and local feature capture, further improving prediction accuracy. Furthermore, the Transformer's attention mechanism captures global features, demonstrating superior performance compared to convolutional networks. By leveraging the strengths of both networks, the resulting multi-plane image scene representation is capable of better viewpoint extrapolation and restoration of challenging regions.

[0008] A new viewpoint synthesis method for scene reconstruction based on generating multi-plane images comprises the following steps:

[0009] Step 1: Obtain a dataset for training through light field sampling;

[0010] Step 2: Preprocess the dataset images in step 1, select the corresponding source view and target view, determine the hypothesized depth plane, warp the source view to the depth plane centered on the target view, and obtain the plane scan volume PSVs required by the network;

[0011] Step 3: The PSVs obtained in step 2 are fed into a Transformer-based encoder, where the Transformer has 12 layers. The features extracted from layers 3, 6, 9, and 12 are extracted and restored to multi-scale features through convolution.

[0012] Step 4: The multi-scale features obtained in step 3 are input into the convolution-based decoder. The low-dimensional and high-dimensional features are combined through residual connections to predict the contribution weights and alpha values ​​of PSVs from different viewpoints to the MPI. The RGB values ​​of each layer of the MPI are calculated according to the weight values.

[0013] Step 5: When synthesizing the new viewpoint image, the traditional MPI synthesis method is used. Given the MPI representation relative to the reference viewpoint, the inverse homography plane transformation is applied to the RGBA image of each plane. The transformed image is synthesized from back to front by the over operator on each layer of RGBA images to generate a new viewpoint image.

[0014] Step 6: Use the perceptual loss function of the classic VGG network to calculate the loss value of the rendered new viewpoint image and the real image, perform backpropagation processing on the network, and iteratively optimize the network hyperparameters to finally obtain a trained model;

[0015] Step 7: Based on the trained model, the real scene is reconstructed and rendered to obtain a new viewpoint image. The sparse viewpoint image of the real scene is input into the network trained in step 6. The network reconstructs the scene and renders the image of the target viewpoint.

[0016] The beneficial effects achieved by the present invention are:

[0017] (1) The present invention uses Transformer as the backbone of the network and reconstructs the scene through multi-plane images, overcoming the problem of limited receptive field of the previous convolutional network-based generation of multi-plane scene representation. Transformer can capture long-range dependent features and overcome the problem of stacking and ghosting when viewpoints are extrapolated. At the same time, through long-range dependent features, some de-occluded parts can also be well inferred.

[0018] (2) The present invention introduces the characteristics of Transformer to capture global features, and also combines the local features of the convolutional network, retaining the inherent inductive bias of convolution, which is conducive to the network retaining as many required features as possible, making the network have better generalization.

[0019] (3) Based on the light field sampling characteristics, the present invention reduces the input views as much as possible, that is, reconstructs the scene as much as possible under the condition of sparser input, and then renders the new view. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the projection described in an embodiment of the present invention.

[0021] Figure 2 2 is a structural diagram of an encoder-decoder described in an embodiment of the present invention.

[0022] Figure 3 It is the Transformer block described in the embodiment of the present invention. DETAILED DESCRIPTION

[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0024] A new viewpoint synthesis method based on scene reconstruction generated from multi-plane images uses the Transformer as the backbone of the network and reconstructs the scene from multi-plane images. This overcomes the problem of limited receptive field in previous methods based on convolutional networks to generate multi-plane scene representations. The Transformer can capture long-range dependent features and overcome the problem of stacked ghosting when viewpoints are extrapolated. At the same time, through long-range dependent features, some de-occluded parts can also be well inferred. This method introduces the Transformer's ability to capture global features, while also combining the local features of the convolutional network, retaining the inherent inductive bias of convolution, which helps the network retain as many required features as possible, making the network more generalizable. This hybrid Transformer and convolutional network method can enable the application of multi-plane scene representations to more real-world scenarios.

[0025] A new viewpoint synthesis method for scene reconstruction based on generating multi-plane images comprises the following steps:

[0026] Step 1: Obtain a dataset for training by light field sampling.

[0027] The classic light field acquisition method, a camera array, involves placing multiple cameras on a plane to sample the scene. This requires consideration of the spacing between cameras. Simply put, the larger the baseline, the more difficult it is to reconstruct the scene and perform viewpoint synthesis. Light field images also face the problem of mutual occlusion. Previous work has shown that the sampling intervals for occluded and unoccluded areas vary, but are both affected by factors such as the highest spatial frequency of the sampled light field, the camera focal length, and the nearest and farthest scene depths.

[0028] Previous work has shown that in the presence of occlusion, the sampling interval satisfies the following formula:

[0029]

[0030]

[0031] Among them, K x is the maximum spatial frequency of the sampled light field, which is limited by the maximum spatial frequency of the continuous light field and the pixel size, and satisfies the above formula, B x is the maximum spatial frequency of the continuous light field, f is the focal length of the camera, Z min 、Z max where Δx is the pixel size and the nearest and farthest scene depths are the closest and farthest depths, respectively. If an explicit scene representation like MPI is used, the sampling interval can be further increased. Assuming the scene is divided into D layers at equal depths, the sampling interval can be increased by a factor of D.

[0032] Step 2: Preprocess the dataset images in step 1, select the corresponding source view and target view, determine the hypothesized depth plane, warp the source view to the depth plane centered on the target view, and obtain the plane scan volumes (PSVs) required by the network.

[0033] The scene representation adopted is based on MPI. According to previous work, it is necessary to first determine the nearest and farthest depth planes (for example, within the range of 1 to 100 meters), then equidistantly layer the scene according to the depth, and then project the source viewpoint image onto the layered depth of the target view to obtain plane scan volumes (PSVs).

[0034] refer to Figure 1 , where x and x' are the images of a 3D point in the source and target views, C and C' are the source and target viewpoints, H is the homography matrix, K and K' are the intrinsic parameters of the source and target cameras, R and t are the rotation and translation, n is the plane normal vector, and d is the depth value. The corresponding relationship satisfies the following formula:

[0035]

[0036] Applying homography transformation to x of the source viewpoint can obtain x' of the target viewpoint, ie, x'=Hx.

[0037] Step 3: The PSVs obtained in step 2 are fed into a Transformer-based encoder, where the Transformer has 12 layers. The features extracted from layers 3, 6, 9, and 12 are extracted and restored to multi-scale features through convolution.

[0038] Step 4: Input the multi-scale features obtained in step 3 into the convolution-based decoder, and combine the low-dimensional and high-dimensional features through residual connections to predict the contribution weights of PSVs from different viewpoints to MPI (the RGB values ​​of each layer of MPI are calculated according to the weights) and alpha values.

[0039] Existing MPIs are mostly based on convolutional neural networks (CNNs), which are limited by the receptive field of the convolution operation. They also have limitations when handling large baselines and some non-Lambertian planes, and can also suffer from mispredictions or aliasing when handling occluded regions. The Transformer overcomes this limited receptive field and can capture long-range dependent features. Furthermore, some work has demonstrated that fusing Transformer and CNN features can achieve better results. Previous work has also demonstrated that convolutional networks and Transformers have complementary properties. The Transformer's loss space is flatter than that of CNNs, resulting in superior performance and generalization. However, the Transformer's loss space is non-convex, requiring a large training dataset. The Transformer aggregates features, while the CNN diversifies them. This means that the Transformer reduces high-frequency components, while the CNN amplifies them. This means that the CNN is more susceptible to high-frequency components, while the Transformer is less susceptible. Based on the complementary nature of Transformer and CNN, we plan to use a hybrid network architecture called Transformer-CNN to extract the scene features needed to generate MPI. On the one hand, we hope to improve the quality of existing MPI. At the same time, because Transformer can capture long-distance dependent features, we also hope to further perform viewpoint extrapolation.

[0040] Encoder and decoder structure reference Figure 2-3 , the PSV is subjected to image segmentation, linear projection, position embedding and other operations to convert it into a sequence input to the stacked transformer (each layer of transformer performs layer normalization and multi-head attention mechanism on the input, and then performs jump connection, further performs layer normalization and passes through the multi-layer perceptron, and then performs a jump connection and inputs to the next transformer). The 3rd, 6th, 9th, and 12th layer features are taken out in the encoder, and these features are restored into features of different scales through convolution. The features of different scales are then sent to the convolution-based decoder to obtain PSVs weight values ​​and alpha values. The weights of the predicted PSVs are weighted and summed on the PSVs to obtain an explicit hierarchical result of the RGB values. Combining RGB and alpha, MPI is obtained.

[0041] Step 5: When synthesizing the new viewpoint image, the traditional MPI synthesis method is used. For a given MPI representation relative to the reference viewpoint, a new viewpoint image is generated by applying a plane transformation (inverse homography) to the RGBA image of each plane, and then performing an "over" operator synthesis operation on each layer of RGBA images from back to front.

[0042] When synthesizing new viewpoint images, the traditional MPI synthesis method is adopted. For a given MPI representation relative to a reference viewpoint, a new viewpoint image is generated by applying a plane transformation (inverse homography) to the RGBA image of each plane, and then performing an "over" operator synthesis operation on each layer of RGBA images from back to front.

[0043] The plane transformation here is to reversely warp each layer of the MPI RGBA image to a target viewpoint. The MPI plane to be transformed, that is, the view frustum plane of the reference viewpoint, has a set shape of n·x+a=0, where n is the plane normal and x=[u s ,v s ,1] T is the source pixel homogeneous coordinate, and a is the plane offset. Since the source viewpoint MPI plane is parallel to the reference viewpoint camera, n=[0,0,1], a=-d s , where d s The rigid 3D transformation matrix mapping from the reference viewpoint camera to the target viewpoint camera is defined by the 3D rotation R and translation t. The internal parameters of the reference viewpoint camera and the target viewpoint camera are represented as k s and d s Then for each pixel [u t ,v t ] is obtained using the standard inverse homography matrix by finding the corresponding relationship [u s ,v s ] to obtain each target image [u t ,v t ]’s color value and alpha value. The conversion relationship between the source viewpoint and the target viewpoint is shown in the following formula:

[0044]

[0045] Among them, [u s ,v s ] is the pixel point in the reference image, [u t ,v t ] is the pixel point in the target image, k s and k tare the internal parameters of the reference viewpoint camera and the target viewpoint camera respectively, R and t are the rotation and translation from the reference viewpoint camera to the target viewpoint camera, a is the plane offset, and T represents the matrix transpose.

[0046] "over" operator synthesis: After obtaining the MPI scene representation, that is, after obtaining the RGBA values ​​of each layer, the image is synthesized using the standard "over" operator. The "over" operator is expressed as follows:

[0047]

[0048] Where T d (A) represents the transmittance, that is, the value retained by the d layer after the stratification before the d layer, A is the alpha value, C is the color value, c d is the color value of layer d, a i is the alpha value of layer i.

[0049] Step 6: Use the perceptual loss function of the classic VGG network to calculate the loss value of the rendered new viewpoint image and the real image, perform backpropagation processing on the network, and iteratively optimize the network hyperparameters to finally obtain a trained model.

[0050] The network is trained using synthetic viewpoint images as supervision, and the loss function samples the perceptual loss of the classic VGG network. The loss function is shown in the following formula. Where I represents the synthetic image, I g Represents a real image.

[0051] L=L VGG (I,I g )

[0052] Step 7: Based on the trained model, the real scene is reconstructed and rendered to obtain a new viewpoint image. The sparse viewpoint image of the real scene is input into the network trained in step 6. The network reconstructs the scene and renders the image of the target viewpoint.

[0053] Figure 2 In

[15] , the dilated convolution kernel is used to extract multi-scale features, which can replace the Transformer block operation in the encoder to complete similar feature extraction work.

[0054] Figure 2 In

[15] , the convolutional part in the decoder is replaced by the Transformer block, which has the same purpose.

[0055] Figure 3 In

[15] , other attention mechanisms, such as channel attention, spatial self-attention, etc., are used to replace the multi-head attention mechanism, with the same operation purpose.

[0056] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed in the present invention should be included in the protection scope recorded in the claims.

Claims

1. A new viewpoint synthesis method for scene reconstruction based on generating multi-plane images, characterized by: The method comprises the following steps: Step 1: Obtain a dataset for training through light field sampling; Step 2: Preprocess the dataset images in step 1, select the corresponding source view and target view, determine the hypothesized depth plane, warp the source view to the depth plane centered on the target view, and obtain the plane scan volume PSVs required by the network; Step 3: The PSVs obtained in step 2 are fed into the Transformer-based encoder, where the Transformer has 12 layers. The features extracted from layers 3, 6, 9, and 12 are extracted. For the features of layer 12, a deconvolution operation with a convolution kernel size of 2*2*2 is directly used. For the features of layers 3, 6, and 9, a deconvolution operation with a convolution kernel size of 2*2*2 is first used, and then a convolution operation with a convolution kernel size of 3*3*3 is used. Then, batch normalization and ReLU activation function are used to obtain multi-scale features of layers 3, 6, 9, and 12. Step 4: The multi-scale features obtained in step 3 are input into the convolution-based decoder. The low-dimensional and high-dimensional features are combined through residual connections to predict the contribution weights and alpha values ​​of PSVs from different viewpoints to the MPI. The RGB values ​​of each layer of the MPI are calculated according to the weight values. Step 5: When synthesizing the new viewpoint image, the traditional MPI synthesis method is used. Given the MPI representation relative to the reference viewpoint, the inverse homography plane transformation is applied to the RGBA image of each plane. The transformed image is synthesized from back to front by the over operator on each layer of RGBA images to generate a new viewpoint image. Step 6: Use the perceptual loss function of the classic VGG network to calculate the loss value of the rendered new viewpoint image and the real image, perform backpropagation processing on the network, and iteratively optimize the network hyperparameters to finally obtain a trained model; Step 7: Based on the trained model, the real scene is reconstructed and rendered to obtain a new viewpoint image. The sparse viewpoint image of the real scene is input into the network trained in step 6. The network reconstructs the scene and renders the image of the target viewpoint.

2. The novel viewpoint synthesis method for scene reconstruction based on generating multi-plane images according to claim 1, characterized in that: In step 1, in the case of occlusion, the sampling interval satisfies the following formula: Among them, K x is the maximum spatial frequency of the sampled light field, which is limited by the maximum spatial frequency of the continuous light field and the pixel size, and satisfies the above formula. x is the maximum spatial frequency of the continuous light field, f is the focal length of the camera, Z min 、Z max are the nearest and farthest scene depths, and Δx is the pixel size. If an explicit scene representation such as MPI is used, the sampling interval can be further expanded. Assuming that the scene is divided into D layers at equal intervals according to depth, the sampling interval can be expanded by D times.

3. The new viewpoint synthesis method for scene reconstruction based on generating multi-plane images according to claim 1, characterized in that: In step 2, the scene representation used is based on MPI. First, the nearest and farthest depth planes need to be determined. Then, the scene is equidistantly layered according to the depth. The source viewpoint image is then projected onto the layered depth of the target view to obtain the plane scan volume (PSVs). Let x and x' be the images of a 3D space point in the source view and the target view, C and C' are the source viewpoint and the target viewpoint respectively, H is the homography matrix, K and K' are the intrinsic parameters of the source camera and the target camera respectively, R and t are the rotation and translation respectively, n is the plane normal vector, and d is the depth value; the corresponding relationship satisfies the following formula: Applying homography transformation to x of the source viewpoint can obtain x' of the target viewpoint, ie, x'=Hx.

4. The new viewpoint synthesis method for scene reconstruction based on generating multi-plane images according to claim 1, characterized in that: In step 3, a transformer-CNN hybrid network architecture consisting of a transformer-based encoder and a CNN-based decoder network is used to extract the scene features required to generate MPI.

5. The new viewpoint synthesis method for scene reconstruction based on generating multi-plane images according to claim 1, characterized in that: In step 4, the features of different scales are fed into the convolution-based decoder to obtain the PSVs weight value and alpha value. The predicted PSVs weights are weighted and summed to obtain the explicit hierarchical result of the RGB value. Combining RGB and alpha, MPI is obtained. The input of the decoder here is the multi-scale feature obtained in step 3. The deep features of the network are convolved with a convolution kernel size of 3*3*3, and then restored to features consistent with the upper-level feature scale through batch normalization and ReLU activation function. New features are then formed through jump connection operations. The above operation is performed four times to complete the feature connection of all levels, and finally the PSVs weight value and alpha value are output.

6. The new viewpoint synthesis method for scene reconstruction based on generating multi-plane images according to claim 1, characterized in that: In step 5, the inverse homography plane transformation is to reversely warp each layer RGBA image of the MPI to a target viewpoint; The MPI plane to be transformed, that is, the view cone plane of the reference viewpoint, has a set shape of n·x+a=0, where n is the plane normal and x=[u s ,v s ,1] T is the homogeneous coordinate of the source pixel, and a is the plane offset; since the source viewpoint MPI plane is parallel to the reference viewpoint camera, n=[0,0,1], a=-d s , where d s is the depth of the source MPI plane; the rigid three-dimensional transformation matrix mapping from the reference viewpoint camera to the target viewpoint camera is defined by the three-dimensional rotation R and translation t. The internal parameters of the reference viewpoint camera and the target viewpoint camera are represented as k s and d s ; Then for each pixel [u t ,v t ] is obtained using the standard inverse homography matrix by finding the corresponding relationship [u s ,v s ] to obtain each target image [u t ,v t ]; the conversion relationship between the source viewpoint and the target viewpoint is shown in the following formula: Among them, [u s ,v s ] is the pixel point in the reference image, [u t ,v t ] is the pixel point in the target image, k s and k t are the internal parameters of the reference viewpoint camera and the target viewpoint camera, R and t are the rotation and translation from the reference viewpoint camera to the target viewpoint camera, a is the plane offset, and T represents the matrix transpose; Over operator synthesis: After obtaining the MPI scene representation, that is, after obtaining the RGBA values ​​of each layer, the image is synthesized through the standard over operator; the over operator is expressed as shown in the following formula: Where T d (A) represents the transmittance, that is, the value retained by the d layer after the stratification before the d layer, A is the alpha value, C is the color value, c d is the color value of layer d, a i is the alpha value of layer i.

7. The new viewpoint synthesis method for scene reconstruction based on generating multi-plane images according to claim 1, characterized in that: In step 6, the network is trained using the synthetic viewpoint images as supervision, and the loss function samples the perceptual loss of the classic VGG network; the loss function is shown in the following formula: L=L VGG (I,L g ) Among them, I represents the synthesized image, I g Represents a real image.

Images