A new view synthesis method for wide-baseline sparse camera arrays

Abstract

The application discloses a novel view synthesis method of wide-baseline sparse camera array, belongs to the technical field of novel view synthesis in computer vision and computer graphics, and comprises the following four steps: target sparse acquisition, camera pose calibration and depth map reconstruction, construction of plane-depth fusion scanning body, and generation of predicted view by using a convolutional neural network.In the target sparse acquisition, a wide-baseline sparse camera array is used, camera poses are calibrated by using the acquired sparse views, and a camera pose standardization method is provided; by using the standardized camera poses, a plane-depth fusion scanning body is designed; and finally, two convolutional neural networks are used to generate predicted views.The actual effect shows that, compared with existing novel view synthesis methods, the novel view synthesis method has the advantages of large input view angle, high generated image quality, and free generated view point position, and the view synthesis effect is better than that of traditional novel view synthesis methods.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of novel view synthesis in computer vision and computer graphics, and particularly relates to a novel view synthesis method of a wide-baseline sparse camera array. BACKGROUND

[0002] With the development of virtual reality technology and stereoscopic display technology, multi-view applications have been used in many aspects in recent years. However, the capture and synthesis of dense viewpoints in real scenes is still a bottleneck, especially the prediction of novel views when sparse collection. Obtaining dense views is one of the core problems of computer graphics and vision. In recent years, multi-view technology has been widely used, such as panoramic views of virtual reality (VR), full-parallax images required by three-dimensional (3D) light field display, or multi-angle broadcasting of competitions.

[0003] There are some methods to obtain dense views directly. For example, a dense camera array can obtain direct dense views, such as the multi-camera array of Stanford University, which uses 128 cameras to capture dense view images in a wide range. Lytro Illum and other light field cameras can take a plenoptic image at a time and decompose the plenoptic image into multiple images with different parallaxes. However, both of the above methods have their own shortcomings. The dense camera array is difficult to build due to its complex hardware structure. The total baseline of the light field camera is too short, resulting in a narrow field of view of the captured multi-parallax images. Synthesizing novel views from sparse images is an effective method to solve the above problems.

[0004] Image-based rendering (IBR) is a method that can map nearby views to synthesize new views without 3D reconstruction. IBR methods can synthesize real views within a short baseline, but as the baseline widens, the synthesized images will have errors and cracks due to depth discontinuity and uncertainty. Deep learning methods have become a general method for many computer vision tasks. Convolutional neural networks (CNNs) have shown excellent performance in target classification and image segmentation due to their excellent feature extraction capabilities. And CNNs have also been successfully used in view synthesis tasks, such as EPI reconstruction, DeepStereo, MPI, and NeRF. The EPI reconstruction method proposes an epipolar plane image to obtain a new view by angular domain interpolation. The appearance flow method and the DVM method estimate the correspondence between the input views and mix the mapped views into a new view. In DeepStereo, a plane-scan volume (PSV) is used to construct a selection tower and a color tower. The outputs of the two towers are used to predict new views. In the MPVN, dense view fusion, virtual view fusion, and other methods, the tower construction method is also applied. The disparity estimation network divides the new view synthesis process into two parts: disparity estimation and color estimation. The disparity estimator estimates the disparity map, and then sends the result and the warped image to the color predictor to synthesize the final result. The MPI method synthesizes new views by accumulating alpha-transparency maps. Recently, another method has been proposed, which synthesizes new views of complex scenes by optimizing the underlying continuous scene function using a sparse input set (such as NeRF, NeX, GRF, MINE, and DSNeRF).

[0005] Although there are many methods at present, there are still many limitations, such as narrow baseline, inflexible target viewpoint position, and the need to retrain for each scene. SUMMARY

[0006] The purpose of the present application is to propose a new view synthesis method based on a wide baseline sparse camera array of plane-depth fusion scanning volume, which can solve the shortcomings of the prior art. The method uses sparse input views taken by a wide baseline camera array to synthesize new views at any position within the camera array's field of view.

[0007] The new view synthesis method of a wide baseline sparse camera array of the present application comprises the following steps:

[0008] 1) Acquire sparse acquisition views of the object, comprising the following steps:

[0009] 1.1 Build a wide-baseline spherical camera array, each row and each column of cameras are on the same arc of a sphere, cameras are in converging structure, the focal point of each camera coincides with the center of the sphere, and the physical parameters of each camera are the same; when collecting, the target is placed near the focal point to ensure that the target is near the center of the image in the image taken by each camera;

[0010] 1.2 Use the camera array to take a synchronous shot of the target to obtain a sparse view of the target;

[0011] 2) Use the sparse view obtained in step 1.2 to calibrate the pose of the camera array and reconstruct the depth map, including the following steps:

[0012] 2.1 Use multi-view calibration on the sparse view to estimate the pose of each camera, the intrinsic matrix G of the camera - focal length f a , f b ; image principal point coordinates C a , C b ; distortion coefficients f1, f2; use traditional checkerboard calibration method for calibration;

[0013] 2.2 Use the multi-view calibration method to obtain the camera extrinsic parameters - rotation matrix R and translation vector T, and reconstruct part of the depth map d partial , since the depth map contains holes that cannot be reconstructed to have a depth value, so d partial is called a partial depth map;

[0014] 3) Coordinate system standardization of the camera extrinsic parameters obtained in step 2.2, including the following steps:

[0015] 3.1 Set the coordinates of point P in the current coordinate system as (X P , Y P , Z P ), denoted as S P , similarly, set the coordinates of camera C O as (X O , Y O , Z O ), denoted as S O , and the coordinates of camera C A as (X A , Y A , Z A ), denoted as S A , R A and T A represent the rotation matrix and translation vector from S O to S A , similarly, R P and T P represent the rotation matrix and translation vector from S O to SP The rotation matrix and translation vector of S A can be transformed to S P by the rotation matrix R AP and translation vector T AP , which can be expressed as follows:

[0016] T AP = T P - R P · R A -1 · T A

[0017] 3.2 Using the expression obtained in step 3.1, convert the camera extrinsic parameters obtained in step 2.2 to obtain the rotation matrix and vector of other cameras in the camera array when the specified camera pose is taken as the reference, at this time the pose of the reference camera becomes the identity matrix E and the 0 vector, and the pose of other cameras is obtained by rigid transformation of the reference camera, which is only related to the reference camera, that is, the standardization of the camera coordinate system is realized;

[0018] 4) According to the camera intrinsic parameters, extrinsic parameters and partial depth map obtained in step 2), use the standardized camera coordinate system obtained in step 3) to perform image mapping on the input image to generate the mapping view I t of the target camera, wherein the subscript t represents the position coordinate of the target camera in the camera array, and the specific steps include:

[0019] 4.1 In order to obtain the depth information corresponding to the target camera C t to be synthesized, first use the intrinsic matrix G j of the input camera C j obtained in step 2.1 to map the input image Ig j to the three-dimensional space according to the partial depth map obtained in step 2.2 to form a three-dimensional point set, wherein the subscript j represents the serial number of the input camera, and the number of input cameras is n, then the value range of j is [1, n];

[0020] 4.2 Use the rotation matrix R t and translation vector T t of the standardized target camera C t obtained in step 2.2 and step 3) to project the point set in the three-dimensional space in step 4.1 to the target camera C t to obtain the partial depth map d c corresponding to the target camera C t partialThen in the partial depth map d c partial The void locations in the image are filled with a series of depth values, resulting in a series of depth planes, which together form a set of depth map tensors d. tensor These fill depth values ​​are uniformly taken within ±25% of the median of the partial depth map;

[0021] 4.3 Using the target camera C t intrinsic parameter matrix G t and depth map tensor d tensor Each depth map d in i Target camera C t The location of each pixel in the captured image is mapped to three-dimensional space to obtain the three-dimensional coordinates of the pixel, where the subscript i represents the index of each depth map in the depth map tensor;

[0022] 4.4 After obtaining the target camera C t After determining the three-dimensional coordinates of the pixels in the captured image in space, the rotation matrix R is used... t Translation vector T t The three-dimensional coordinates are reprojected onto each input camera C. j On the image, the pixel value corresponding to each position is extracted using bicubic interpolation, thus obtaining a mapped view I of the target position. t i , among which, I t i This indicates that the depth map d is being retrieved. i At this time, a mapped view is obtained. This operation is performed on each depth map, and then the resulting mapped views are arranged into groups to obtain a set of mapped views I for the target location. t ;

[0023] 5) Based on the mapping view calculated in step 4), construct the planar-depth fusion scanning volume V. t Specifically, it includes the following steps:

[0024] 5.1 Following step 4), calculate the mapping from each input camera to the target camera C. t Mapped view j I t ,in, j I t This represents the mapping of the j-th input view to the target camera C. t A set of mapped views;

[0025] 5.2 Calculate the depth d for each of the n sets of mapped views. i Downmap view j I t i The mean and variance, wherej I t i This represents the i-th mapped view in a set of mapped views corresponding to the j-th input view. The calculated mean and variance are grouped together to form the planar-depth fusion scan volume V. t ;

[0026] 6) The planar-depth fusion scan volume V obtained in step 5) t Input into the feature extraction convolutional neural network Net F In the middle, generate feature mask F mask Then use F mask The color prediction convolutional neural network Net C The input is used to generate the target camera C. t Predictive view Specifically, the following steps are included:

[0027] 6.1 Constructing a Net using a fully convolutional autoencoder F An autoencoder consists of an encoder and a decoder. The encoder compresses the input data and extracts feature information at different scales. The decoder then decodes the feature information and restores the data size. The encoder and decoder have a symmetrical structure, and the output of the decoder is the feature mask F. mask ;

[0028] 6.2 Color Prediction Convolutional Neural Network Net C Adopting Net F The two networks have the same structure, but their network parameters are not shared, and their input and output structures are different. C The input consists of 4 parts, represented as follows:

[0029]

[0030] Wherein: F mask Represents Net F Output feature mask; This represents n input views, each according to its corresponding partial depth map d. j partial Map the view to the target location, then combine it with the mask F. mask The generated Hadamard product; {Ig1,…,Ig n} represents n input views; {P t Q t} is the target camera C t The equivalent two-dimensional coordinates in the camera array represent the relative position of the target camera within the camera array. (Net) C The output is the generated prediction view.

[0031] Compared with the prior new view synthesis method, the new view synthesis method has the advantages of large input view angle, high generated image quality, and free generated view point position, and the view synthesis effect is better than that of the traditional new view synthesis method. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Flowchart of the new view synthesis method of the wide-baseline sparse camera array;

[0033] Figure 2 Schematic diagram of the wide-baseline sparse camera array;

[0034] Figure 3 Schematic diagram of the coordinate system standardization process;

[0035] Figure 4 Schematic diagram of the plane-depth fusion scanning body V t Generation method flowchart;

[0036] Figure 5 Schematic diagram of a feature extraction convolutional neural network;

[0037] Figure 6 Schematic diagram of the input composition of a color prediction convolutional neural network. DETAILED DESCRIPTION

[0038] The present application will be described below with reference to the accompanying drawings.

[0039] As shown in the drawings, the new view synthesis method of the wide-baseline sparse camera array of the present application generally includes target sparse acquisition, camera pose calibration and depth map reconstruction, plane-depth fusion scanning body construction, and using a convolutional neural network to generate a predicted view, and specifically includes the following steps: Figure 1 1) Acquire sparse acquisition views of an object, including the following steps:

[0040] 1.1 Build a wide-baseline spherical camera array, as shown in the drawings, which is a wide-baseline sparse spherical camera array, as shown in the 4x4 camera array, the height of the camera array frame is 2.7 meters, the length is 2.8 meters, the included angle between every two adjacent cameras in the horizontal and vertical directions is 18°, i.e. the field of view angle of the camera array reaches 54°, each row and each column of cameras are on the same arc line of the sphere, the cameras adopt a converging structure, the focal point of each camera coincides with the sphere center, and the physical parameters of each camera are the same; during acquisition, the target to be photographed is placed near the focal point to ensure that the target is located near the center of the image in the image captured by each camera;

[0041] Figure 2 Figure 2

[0042] ​​​1.2 Take synchronized pictures of the target using the camera array, get the sparse views of the target, and take the camera C (1,1) 、 (1,4) 、 (4,1) 、 (4,4) as input;

[0043] 2) Use the sparse views obtained in step 1.2 to calibrate the pose of the camera array and reconstruct the depth map, including the following steps:

[0044] 2.1 Use multi-view calibration on the sparse views to estimate the pose of each camera, the intrinsic matrix G of the camera - focal length f a 、 f b ; principal point coordinates C a 、 C b ; distortion coefficients f1, f2; use traditional checkerboard calibration method for calibration;

[0045] 2.2 Use the multi-view calibration method to obtain the camera extrinsic parameters - rotation matrix R and translation vector T, and reconstruct the partial depth map d partial , since the depth map contains holes that cannot be reconstructed to have depth values, so d partial is called a partial depth map;

[0046] 3) Coordinate system standardization of the camera extrinsic parameters obtained in step 2.2, including the following steps:

[0047] 3.1 As shown in Figure 3 , let the coordinates of point P in the current coordinate system be (X P , Y P , Z P ), denoted as S P , similarly, let the coordinates of camera C O be (X O , Y O , Z O ), denoted as S O , and the coordinates of camera C A be (X A , Y A , Z A ), denoted as S A , R A and T A represent the rotation matrix and translation vector from S O to S A , similarly, R P and T P represent the rotation matrix and translation vector from S O to S Protation matrix and translation vector of the object, since multiple consecutive rotation operations of the object can be combined into one rotation operation by multiplying the rotation matrices, and consecutive translation operations of the object can be combined into one translation operation by adding the translation vectors, by S A transforming to S P the rotation matrix R AP and the translation vector T AP can be expressed as follows:

[0048] T AP = T P - R P · R A -1 · T A

[0049] 3.2 Using the expression obtained in step 3.1, the camera extrinsic parameters obtained in step 2.2 are converted to obtain the rotation matrix and vector of the other cameras in the camera array when the specified camera pose is taken as the reference, the pose of the reference camera becomes the identity matrix E and the 0 vector, and the pose of the other cameras is obtained by rigid transformation from the reference camera, which is only related to the reference camera, that is, the standardization of the camera coordinate system is achieved;

[0050] 4) According to the camera intrinsic parameters, extrinsic parameters and partial depth map obtained in step 2), the input image is mapped to generate the mapping view I t of the target camera using the standardized camera coordinate system obtained in step 3), wherein the subscript t represents the position coordinate of the target camera in the camera array, and the specific steps include:

[0051] 4.1 As shown in Figure 4 , in order to obtain the depth information corresponding to the target camera C t to be synthesized, first, the intrinsic matrix G j of the input camera C j obtained in step 2.1 is used to map the input image Ig j to the three-dimensional space according to the partial depth map obtained in step 2.2 to form a three-dimensional point set, wherein the subscript j represents the serial number of the input camera, and the number of input cameras is n, so the value range of j is [1, n], as shown in Figure 4 , four cameras with vertex positions are taken as input, that is, the value of n is 4;

[0052] 4.2 Using the rotation matrix R t and the translation vector T t of the standardized target camera C t obtained in steps 2.2 and 3), the point set in the three-dimensional space in step 4.1 is projected onto the target camera C t to obtain the target camera C tCorresponding partial depth map d c partial Then in the partial depth map d c partial The void locations in the image are filled with a series of depth values, resulting in a series of depth planes, which together form a set of depth map tensors d. tensor These fill depth values ​​are uniformly taken within ±25% of the median of the partial depth map;

[0053] 4.3 Using the target camera C t intrinsic parameter matrix G t and depth map tensor d tensor Each depth map d in i Target camera C t The location of each pixel in the captured image is mapped to three-dimensional space to obtain the three-dimensional coordinates of the pixel, where the subscript i represents the index of each depth map in the depth map tensor;

[0054] 4.4 After obtaining the target camera C t After determining the three-dimensional coordinates of the pixels in the captured image in space, the rotation matrix R is used... t Translation vector T t The three-dimensional coordinates are reprojected onto each input camera C. j On the image, the pixel value corresponding to each position is extracted using bicubic interpolation, thus obtaining a mapped view I of the target position. t i , among which, I t i This indicates that the depth map d is being retrieved. i At this time, a mapped view is obtained. This operation is performed on each depth map, and then the resulting mapped views are arranged into groups to obtain a set of mapped views I for the target location. t ;

[0055] 5) Based on the mapping view calculated in step 4), construct the planar-depth fusion scanning volume V. t Specifically, it includes the following steps:

[0056] 5.1 Following step 4), calculate the mapping from each input camera to the target camera C. t Mapped view j I t ,in, j I t This represents the mapping of the j-th input view to the target camera C. t A set of mapped views;

[0057] 5.2 Calculate the depth d for each of the n sets of mapped views. i Downmap view j It i The mean and variance, where j I t i This represents the i-th mapped view in a set of mapped views corresponding to the j-th input view. The calculated mean and variance are grouped together to form the planar-depth fusion scan volume V. t ;

[0058] 6) The planar-depth fusion scan volume V obtained in step 5) t Input into the feature extraction convolutional neural network Net F In the middle, generate feature mask F mask Then use F mask The color prediction convolutional neural network Net C The input is used to generate the target camera C. t Predictive view Specifically, the following steps are included:

[0059] 6.1 such as Figure 5 As shown, a Net is constructed using a fully convolutional autoencoder. F An autoencoder consists of an encoder and a decoder. The encoder compresses the input data and extracts feature information at different scales. The decoder then decodes the feature information and restores the data size. The encoder and decoder have a symmetrical structure, and the output of the decoder is the feature mask F. mask ;

[0060] 6.2 Color Prediction Convolutional Neural Network Net C Adopting Net F The two networks have the same structure, but their network parameters are not shared, and their input and output structures are different, such as... Figure 6 As shown, the color prediction convolutional neural network Net C The input consists of 4 parts, represented as follows:

[0061]

[0062] Wherein: F mask Represents Net F Output feature mask; This represents n input views, each according to its corresponding partial depth map d. j partial Map the view to the target location, then combine it with the mask F. mask The generated Hadamard product; {Ig1,…,Ig n} represents n input views; {P t Q t} is the target camera C tNet C The output is the generated predicted view

Claims

1. A method for synthesizing new views of a wide-baseline sparse camera array, comprising the following steps: 1) obtaining sparse views of an object, comprising the following steps: S1.1 building a wide-baseline spherical camera array, each row and each column of cameras are on the same arc line of the sphere, the cameras adopt a converging structure, the focal point of each camera coincides with the center of the sphere, and the physical parameters of each camera are the same; during the collection, the target to be photographed is placed near the focal point to ensure that the target is located near the center of the image in the image captured by each camera; S1.2 synchronously capturing the target using the camera array to obtain sparse views of the target; 2) using the sparse views obtained in step 1.2 to calibrate the pose of the camera array and reconstruct the depth map, comprising the following steps: S2.1 Use multi-view calibration to estimate the pose of each camera using sparse views, the intrinsic matrix G of the camera includes focal length , ; image principal point coordinates ; distortion coefficients ; calibrate using traditional checkerboard calibration method; S2.2 The method of using multi-view calibration to obtain camera extrinsic parameters, including rotation matrix R and translation vector T, and reconstructing partial depth map Since the depth map contains holes that cannot be reconstructed with depth values, it is called partial depth map; 3) standardizing the coordinate system of the camera extrinsic parameters obtained in step 2.2, comprising the following steps: S3.1 Let the coordinates of point P in the current coordinate system be , denoted as , let the coordinates of camera be , denoted as , let the coordinates of camera be , denoted as , and denote the rotation matrix and translation vector from to , and denote the rotation matrix and translation vector from to , the rotation matrix from to and the translation vector are denoted as follows: , ; S3.2 using the expression obtained in step 3.1 to convert the camera extrinsic parameters obtained in step 2.2 to obtain the rotation matrix and vector of the other cameras in the camera array when the pose of the specified camera is taken as the reference, at this time the pose of the reference camera becomes the unit matrix E and the 0 vector, the pose of the other cameras is obtained by rigid transformation from the reference camera and only related to the reference camera, that is, the standardization of the camera coordinate system is realized; 4) using the camera intrinsic, extrinsic, and partial depth map obtained from step 2), and the normalized camera coordinate system obtained from step 3), perform image mapping on the input image to generate the mapping view of the target camera wherein, the subscript t represents the position coordinates of the target camera in the camera array, comprising the following steps: S4.1 To obtain the target camera to be synthesized The corresponding depth information, first use the input camera Intrinsic matrix obtained in step 2.1 , the input image According to the partial depth map obtained in step 2.2, it is mapped into three-dimensional space to form a three-dimensional point set, where subscript j represents the serial number of the input camera, and the number of input cameras is n, then the value range of j is [1, n]; S4.2 Projecting the point set in 3D space in step 4.1 onto the target camera using the normalized rotation matrix and translation vector obtained in step 2.2 and step 3) These filled depth values are uniformly taken in the range of ±25% of the median of the partial depth map.​​​​​​​ S4.3 using the target camera intrinsic parameter matrix and depth map tensor Each depth map in , target camera The location of each pixel in the captured image is mapped to three-dimensional space to obtain the three-dimensional coordinates of the pixel, where the subscript i represents the index of each depth map in the depth map tensor; S4.4 After obtaining the target camera After obtaining the three-dimensional coordinates of the pixel points in the space of the photographed image, the three-dimensional coordinates are re-projected onto the image of each input camera using a rotation matrix and a translation vector and the pixel value corresponding to each position is extracted using a bicubic interpolation method, i.e. a mapping view of the target position is obtained , wherein represents that the depth map is obtained, the operation is performed on each depth map, and each obtained mapping view is arranged into a group, thereby obtaining a group of mapping views of the target position ; 5) From the mapping view computed in step 4), construct a plane-depth fused scan volume comprising the following steps: S5.1 Compute the mapping view of each input camera to the target camera according to step 4) where, represents the set of mapping views of the jth input view to the target camera where,​ S5.2 Calculate the mean and variance of each same depth in n groups of mapped views The mean and variance of each same depth in n groups of mapped views where The mean and variance of each same depth in n groups of mapped views ; 6) The planar-depth fusion scan volume obtained in step 5) Input into the feature extraction convolutional neural network In the middle, generate feature masks Then use Constructing a color prediction convolutional neural network Input, generate target camera Predictive view Specifically, it includes the following steps: S6.1 Constructing using a full convolutional autoencoder The autoencoder is composed of an encoding end and a decoding end. The encoding end performs size compression on input data and extracts feature information at different scales. The decoding end decodes the feature information and restores the data size. The encoding end and the decoding end adopt a symmetric structure, and the output of the decoding end is a feature mask . S6.2 Color prediction convolutional neural network The same structure is adopted as but the network parameters are not shared, and the input and output of the two networks are different, The input of the network is composed of 4 parts, represented as: ; wherein: represents an output characteristic mask; } represents n input views respectively mapped to the target location according to their corresponding partial depth maps mapped views, and combined with the mask generated Hadamard product; } represents n input views; } is the target camera the equivalent two-dimensional coordinates in the camera array, representing the relative position of the target camera in the camera array, The output of .

Images