Three-dimensional reconstruction and rendering method for scene containing transparent object

Abstract

The invention provides a three-dimensional reconstruction and rendering method for a scene containing a transparent object, relates to the technical field of computer vision, and aims to shoot the panorama of the transparent object and the surrounding environment, construct different types of transparent object data sets in the scene with bright color for network training, and finally obtain a reconstruction result of the whole scene. According to the method, a multi-module rendering network model is used, and a network is trained according to information of different angles in a learning picture through an angle network and a reflection network. Due to the fact that the scene comprises transparent objects, when the light propagation path is considered, it cannot only be assumed that the light path is propagated along a straight line, and special links such as refraction and reflection in the scene need to be considered. The method provided by the invention comprises the reconstruction of the refraction area and the reflection area at the same time, and realizes the reconstruction and rendering of the whole scene comprising the transparent object.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and three-dimensional reconstruction, and in particular to a three-dimensional reconstruction and rendering method of a scene containing transparent objects. Background Art

[0002] Transparent objects are involved in various fields in daily life, but the three-dimensional reconstruction and rendering of transparent objects has always been a difficult problem. Three-dimensional reconstruction is a core technology in the field of computer vision, among which multi-view stereo reconstruction is a technology that converts two-dimensional images or other types of data into three-dimensional models. This technology has a wide range of applications. The principle of traditional three-dimensional reconstruction methods assumes that the light on the surface of the object propagates normally in a straight line, and the surface of the reconstructed object has clear colors. However, for the surface of transparent objects, the propagation of light does not follow the principle of straight-line propagation, there is diffuse reflection, and the color observed at different angles is also different. There are also special light propagation modes such as refraction inside transparent objects, which is contrary to the principle of traditional three-dimensional reconstruction that light propagates in a straight line. As a result, the results of reconstructing transparent objects are defective, resulting in inaccurate accuracy. Therefore, a method for three-dimensional reconstruction of transparent object scenes is needed to solve this problem. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention provides a three-dimensional reconstruction and rendering method of a scene containing transparent objects;

[0004] A three-dimensional reconstruction and rendering method for a scene containing transparent objects comprises the following steps:

[0005] S1. Shoot a 360° circular scene containing transparent objects. The specific steps are:

[0006] First, place objects of different colors around the transparent object and use colorful pictures as the boundary area of ​​the scene; shoot under sufficient light conditions and shoot multiple groups of 360° ring scenes of transparent objects, centered around the transparent object;

[0007] S2: Preprocess the captured images and build a transparent object dataset. The specific steps are:

[0008] First, extract and match the feature points of all the photos taken to obtain the internal and external parameter information of the camera position of each picture. Sparse reconstruction is performed on all the photos taken to obtain the sparse point cloud results of the entire scene. The pose data of all the photos taken is calculated using the scene pose script to obtain the pose information; secondly, select several photos with obvious reflection features, extract the outline of the transparent objects in the photos, and remove the background as mask information;

[0009] Finally, the training set and validation set of the transparent object dataset were divided into two sets in a 1:1 ratio of the number of photos.

[0010] S3. Construct a multi-module network model for 3D reconstruction and rendering of transparent object scenes. The specific steps are:

[0011] The 3D reconstruction and rendering network of transparent object scenes consists of three main modules: the reconstruction and rendering module of the entire scene, the reconstruction and rendering module of the reflection area, and the reconstruction and rendering module of the refraction area inside the transparent object. The working principle of the network is that each ray emitted from the camera origin passes through each pixel in the corresponding captured image and is uniformly sampled, and the RGB color value C and weight ω of each sampling point are calculated. The transparency σ is determined by the weight ω of each sampling point. The rendering formula for color RGB is defined as:

[0012]

[0013] For the weight of each sampling point, the final weight parameter formula is calculated based on the opacity or density value and is defined as:

[0014] ω i =e -∑σδ (1-e -δσ )

[0015] Among them, σ and δ are the density value of the sampling points in the scene, that is, the transparency and the distance between two adjacent sampling points on each ray, and i is the i-th sampling point on each ray;

[0016] The scene contains transparent objects or highlight objects. For the reflection area, the reflection appearance of the transparent object is input into the reflection module of the network by the extracted mask, the scene of the reflection area is extracted separately, and the reflection area is reconstructed and rendered separately. The RGB color value and weight value of each sampling point are also calculated, which is the same as the color RGB rendering formula and weight parameter formula definition;

[0017] For the interior of transparent objects, due to the refraction of light, a separate model with a network offset at a certain angle from the camera is introduced to learn scene information. After reconstructing and rendering the refraction area inside the transparent object, the weighted sum is taken with the overall reconstruction and rendering results in step 3-2. For the module of the refraction network and the final sum of the reconstruction and rendering results of the original network, the angle network is defined as:

[0018]

[0019] Among them, α is the three-dimensional space coordinate information of each sampling point in the image, Camera parameters for each picture;

[0020] The panoramic reconstruction network module, refraction network module and reflection network module layers are all fully connected neural networks consisting of an input layer, an output layer and four hidden layers. There is a ReLU activation function after each hidden layer, and there is a color RGB value activation function at the end of each network module.

[0021] The ReLU activation function is defined as:

[0022] f(x)=max(x,0)

[0023] The activation function of the color RGB value is the Sigmoid activation function;

[0024] The corresponding Sigmoid activation function is defined as:

[0025]

[0026] S4: Train and test the 3D reconstruction and rendering network for scenes containing transparent objects, and obtain the rendering results for each round. The specific steps are:

[0027] First, set the parameters of network training and select the type of transparent object dataset for training; second, train the 3D reconstruction and rendering network containing transparent object scenes; then the network first samples the entire scene to obtain preliminary reconstruction and rendering results. For areas with reflections, extract them separately and reconstruct and render the reflection areas;

[0028] Verify the reconstruction and rendering results. After each round of verification, the network performs secondary sampling, and the samples with relatively large calculated weight values ​​are further sampled. The above training steps are repeated to obtain the verified images and the calculated index values ​​of each round.

[0029] Finally, the 3D reconstruction and rendering network of the scene containing transparent objects is tested. By inputting some images of the validation set, the 3D reconstruction and rendering results of the entire scene are obtained.

[0030] The beneficial effects of adopting the above technical solution are:

[0031] The present invention provides a three-dimensional reconstruction and rendering method for a scene containing transparent objects. Based on the phenomenon that light propagation is subject to refraction and reflection when transparent objects are included in multiple scenes, a network performs separate sampling and reconstruction of light and performs offset angle re-weighting processing on areas where refraction and reflection exist in the transparent objects. This method can simultaneously realize reconstruction and rendering of areas where partial refraction and reflection of the transparent objects in the scene exist, and ultimately realize three-dimensional reconstruction and rendering of the scene containing transparent objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1is a flow chart of a method for three-dimensional reconstruction and rendering of a scene containing transparent objects in an embodiment of the present invention;

[0033] Figure 2 is a flow chart of a network training process in an embodiment of the present invention;

[0034] Figure 3 A structural diagram of a network in an embodiment of the present invention;

[0035] Figure 4 The verification results of the three-dimensional reconstruction and rendering of the scene containing transparent objects using this network model in the embodiment of the present invention. Among them, (a)-is the transparent teacup corresponding to the transparent object dataset, (b)-is the transparent vase corresponding to the dataset, (c)-is the transparent ball corresponding to the transparent object dataset, (d)-is the corresponding mirror local scene, and (e)-is the transparent horse corresponding to the transparent object dataset. Among them, each picture in (a)(b)(c)(d)(e) represents the real result, fine sampling result, fine sampling normal reconstruction result, fine sampling depth map, fine sampling reflection area result, fine sampling reflection area depth map, forward propagation depth map, and mask depth map from left to right. DETAILED DESCRIPTION

[0036] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0037] A method for three-dimensional reconstruction and view rendering of a scene containing transparent objects comprises the following steps:

[0038] S1. Shooting a 360° annular scene containing transparent objects:

[0039] Shoot a scene with multiple transparent objects, place a transparent sphere, a transparent glass cup, and a transparent statue horse in the center of the scene, place objects of different colors around the transparent objects, and use colorful pictures as the boundary area of ​​the scene, finally forming a closed area to represent the overall scene;

[0040] It is necessary to ensure that there is sufficient light under the shooting conditions, and the viewing angle is centered around the transparent object. It is required that the reflection color of the transparent object surface in the pictures taken at different angles is different, and the texture outline of the transparent object surface in the pictures taken is clear. The final pictures taken are saved in 672×504 JPG format;

[0041] S2. Preprocess the captured images and set network parameters to obtain network input content:

[0042] First, extract and match the feature points of all the photos taken to obtain the internal and external parameter information of the camera position of each photo. Perform sparse reconstruction on all the photos taken to obtain the sparse point cloud results of the entire scene and save all the results; secondly, use the scene pose script to calculate the pose data of all the photos taken to obtain the pose information and save it; select several photos with obvious reflection features, extract the outline of the transparent object in the photo to obtain the reflection shell image in PNG format, and remove the background as mask information;

[0043] Finally, the training set and validation set of the transparent object dataset are divided into a 1:1 ratio of the number of photos to form a transparent object dataset. The transparent object dataset contains network input content, which includes training and validation files, camera parameters, sparse point clouds, and boundary calculation value files;

[0044] S3. Construct a multi-module network model for 3D reconstruction and rendering of transparent object scenes:

[0045] The 3D reconstruction and rendering network of transparent object scenes consists of three main modules: the reconstruction and rendering module of the entire scene, the reconstruction and rendering module of the reflection area, and the reconstruction and rendering module of the refraction area inside the transparent object. The network works by uniformly sampling each ray emitted from the camera origin, passing through each pixel in the corresponding captured image, and calculating the opacity σ and RGB color value C of each pixel. The opacity σ is determined by the weight ω of each sampling point. The rendering formula for color RGB is:

[0046]

[0047] For the weight of each sampling point, the final weight parameter is calculated based on the opacity or density value to determine the importance of each sampling point on the light. The weight parameter calculation formula is:

[0048] ω i =e -∑σδ (1-e -δσ )

[0049] Among them, σ and δ are the density value and opacity loss of the sampling point in the scene and the distance between the two adjacent sampling points on each ray, respectively, and i is the i-th sampling point on each ray;

[0050] The density value of σ for each sampling point is related to the spatial coordinates, and the pose parameters of the camera for each image determine the color value of each sampling point. The color of transparent objects is different in images taken from different perspectives, so the camera parameters determine the color value of each sampling point and are independent of the three-dimensional spatial coordinate information;

[0051] The scene contains transparent objects or highlight objects. For the reflection area, the reflection surface of the transparent object is input into the reflection module of the network by extracting the mask, and the scene of the reflection area is extracted separately, and the reflection area is reconstructed and rendered separately. The RGB color value and weight value of each sampling point are calculated in the same way as the rendering formula and weight parameter formula definition;

[0052] For the interior of transparent objects, due to the refraction of light, an angle network is introduced separately, which is at a certain angle with the light emitted by the camera, as the refraction module of the network. After reconstructing and rendering the refraction area inside the transparent object, it is weighted and summed with the overall reconstruction and rendering results in step 3-2. For the refractive index inside the transparent object, the angle network forms a certain angle with the light of the overall reconstruction network of the scene. The training network continuously learns the scene and verifies the results, and finally approaches the real refraction angle inside the transparent object. The network is defined as:

[0053]

[0054] Among them, α is the three-dimensional space coordinate information of each sampling point in the image, Camera parameters for each picture;

[0055] Each network module layer is a fully connected neural network consisting of an input layer, an output layer and four hidden layers. After each hidden layer, there is a ReLU activation function and an activation function of the color RGB value.

[0056] The ReLU activation function is defined as:

[0057] f(x)=max(x,0)

[0058] The activation function of the color RGB value is the Sigmoid activation function;

[0059] The Sigmoid activation function is defined as:

[0060]

[0061] S4: Train and test the 3D reconstruction and rendering network for scenes containing transparent objects, and obtain the verification results and rendering results for each round. The specific steps are:

[0062] First, set up the shell file for network training. In the file, set the aspect ratio, batch size, and number of sampling points during training, and select the type of transparent object dataset for training.

[0063] Secondly, the 3D reconstruction and rendering network of transparent object scenes is trained, and the training and verification method is adopted. Verification is performed after each round of network training, and the verified pictures and the calculated index values ​​of each round are obtained;

[0064] The network first samples the entire scene to obtain preliminary reconstruction and rendering results. After reconstructing and rendering the refractive area inside the transparent object, it adds a weighted sum to the overall reconstruction and rendering results. For areas with reflections, it extracts them separately and reconstructs and renders the reflection areas.

[0065] Verify the reconstruction and rendering results. After each round of verification, the network performs secondary sampling, and the samples with relatively large calculated weight values ​​are again sampled, and the above training steps are repeated to obtain the verified images and the calculated values ​​of indicators for each round, such as peak-to-noise ratio and structural similarity.

[0066] like Figure 4 The figure shows the obtained verification result diagram, where each picture represents the true result, fine sampling result, fine sampling normal reconstruction result, fine sampling depth map, fine sampling reflection area result, fine sampling reflection area depth map, forward propagation depth map, and mask depth map from left to right;

[0067] Finally, the 3D reconstruction and rendering network of the scene containing transparent objects is tested. By inputting some images of the validation set, the 3D reconstruction and rendering results of the entire scene are obtained.

[0068] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A three-dimensional reconstruction and rendering method for a scene containing transparent objects, characterized in that: The following steps are involved: S1, shooting a 360° annular scene containing transparent objects; S2, pre-processing the captured images, setting network parameters, and obtaining network input content; S3, construct a multi-module network model for 3D reconstruction and rendering of transparent object scenes, including a scene overall reconstruction module, a refraction module, and a reflection module; S4. Train and test the 3D reconstruction and rendering network of scenes containing transparent objects, and obtain the verification results and rendering results of each round.

2. A three-dimensional reconstruction and rendering method for a scene containing transparent objects according to claim 1, characterized in that: The S1 specifically includes the following steps: First, place objects of different colors around the transparent object and use brightly colored drawings as the boundary area of ​​the scene; Shoot under well-lit conditions and shoot multiple groups of 360° circular scenes of transparent objects, with the viewing angle centered around the transparent objects. It is required that the reflective colors on the surface of the transparent objects in pictures taken at different angles are different, and the texture contours of the surface of the transparent objects in the pictures must be clear.

3. The method for three-dimensional reconstruction and rendering of a scene containing transparent objects according to claim 1, characterized in that: The S2 specifically includes the following steps: First, feature points of all the photos taken are extracted and matched to obtain the internal and external parameter information of the camera position of each picture. All the photos taken are sparsely reconstructed to obtain the sparse point cloud results of the entire scene; secondly, the pose information of all the photos taken is calculated using the scene pose script; finally, 4 photos with obvious reflection features are selected, the outline of the transparent objects in the photos is extracted, and the background is removed as mask information; and the training set and verification set of the transparent object dataset are divided in a 1:1 ratio.

4. The method for 3D reconstruction and rendering of a scene containing transparent objects according to claim 1, characterized in that: The S3 specifically includes the following steps: The 3D reconstruction and rendering network of the transparent object scene consists of three main modules: the reconstruction and rendering module of the entire scene, the reconstruction and rendering module of the reflection area, and the reconstruction and rendering module of the refraction area inside the transparent object. The working principle of the network is that each ray emitted from the camera origin passes through each pixel in the corresponding captured image and is uniformly sampled, and the RGB color value C and weight ω of each sampling point are calculated. The transparency σ is determined by the weight ω of each sampling point. The rendering formula for color RGB is: For each sampling point weight, the final weight parameter is calculated based on the opacity or density value and is defined as: oh i =e -∑σδ (1-= -σδ ) Among them, σ and δ are the density value of the sampling points in the scene, that is, the transparency and the distance between two adjacent sampling points on each ray, and i is the i-th sampling point on each ray; The scene contains transparent objects or highlight objects. For the reflection area, the reflection surface of the transparent object is input into the reflection module of the network by the extracted mask, the scene of the reflection area is extracted separately, and the reflection area is reconstructed and rendered separately, and the RGB color value and weight value of each sampling point are calculated in the same way; For the interior of transparent objects, due to the refraction of light, a separate model with a network offset at a certain angle from the camera is introduced to learn scene information. After reconstructing and rendering the refraction area inside the transparent object, the weighted sum is taken with the overall reconstruction and rendering results in step 3-2. For the module of the refraction network and the final sum of the reconstruction and rendering results of the original network, the angle network is defined as: Among them, α is the three-dimensional space coordinate information of each sampling point in the image, Camera parameters for each picture; The panoramic reconstruction network module, refraction network module and reflection network module layers are all fully connected neural networks consisting of an input layer, an output layer and four hidden layers. There is a ReLU activation function after each hidden layer, and there is a color RGB value activation function at the end of each network module. The ReLU activation function is defined as: f(x)=max(x,0) The activation function of the color RGB value is the Sigmoid activation function; The corresponding Sigmoid activation function is defined as: 。 5. The method for 3D reconstruction and rendering of a scene containing transparent objects according to claim 1, characterized in that: The S4 specifically comprises the following steps: Set the network training parameters and select the type of transparent object dataset to be trained; train the 3D reconstruction and rendering network for scenes containing transparent objects; the network samples the entire scene once to obtain preliminary reconstruction and rendering results. For areas with reflections, extract them separately and reconstruct and render the reflection areas; The reconstruction and rendering results are verified. After each round of verification, the network performs secondary sampling, and the samples with relatively large calculated weight values ​​are again sampled, and the above training steps are repeated to obtain verified images and the calculated values ​​of indicators for each round, such as peak-to-noise ratio and structural similarity. Finally, the 3D reconstruction and rendering network containing transparent object scenes is tested, and the 3D reconstruction and rendering results of the overall scene are obtained by inputting part of the verification set images.

Images