Point cloud-based scene reconstruction and rendering method, storage medium and electronic device

Abstract

The present invention provides a method, storage medium and electronic device for scene reconstruction and rendering based on point cloud, constructs a data set by acquiring RGB images, camera parameters and depth map data of multiple different perspectives in a single scene; performs multi-perspective three-dimensional reconstruction on the point cloud with neural network features to obtain a feature-enhanced point cloud; uses the feature-enhanced point cloud to simulate the traditional volumetric radiation field for rendering, processes the target scene into a point cloud with a neural feature vector to simulate the neural radiation field, avoids sampling in an empty scene, can represent the scene with less memory, and achieves faster radiation field reconstruction. In addition, since the present invention uses the confidence of the point to determine whether a point is on the surface, it can ignore noise and discrete points in the point cloud, thereby obtaining a better rendering effect.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, computer graphics processing and three-dimensional reconstruction, and in particular to a method, storage medium and electronic device for scene reconstruction and rendering based on point cloud. Background Art

[0002] In recent years, modeling real scenes from image data and rendering photo-realistic images are important topics in computer vision and computer graphics. For 3D reconstruction, "MVSNet: DepthInference for Unstructured Multi-view Stereo" proposed by Y Yao et al. has shown great success. This is because learning-based methods can learn to use the global semantic information of the scene, including object materials, mirrors, and ambient lighting, to obtain stronger matching and more complete reconstruction effects. However, most of the MVS methods currently used apply dense multi-scale 3D CNN (Convolutional Neural Networks) to predict depth maps or voxel occupancy. 3D CNN requires huge memory related to the model resolution, which is a potential obstacle to this technology at present.

[0003] In addition, for scene reconstruction and rendering, Mildenhall et al. proposed the concept of "Nerf: Representing scenes as neural radiance fields for view synthesis", which made great contributions to the reconstruction and rendering of three-dimensional scenes. This method uses global MLPs to reconstruct the radiation field in the entire space by ray marching, but it also leads to a long reconstruction time.

[0004] Alex Yu et al.'s "pixelNeRF: Neural Radiance Fields from One or Few Images" and Qianqian Wang et al.'s "IBRNet: Learning Multi-View Image-Based Rendering" both use a relatively small set of inputs and aggregate multi-view 2D features at each sampled ray point, regressing volume rendering properties for subsequent rendering from radiation fields. However, due to dataset limitations, they do not apply well to multi-view rendering of new scenes, nor do they avoid sampling in empty spaces of the scene. Summary of the invention

[0005] In view of the above-mentioned problems, the present invention provides a point cloud-based scene reconstruction and rendering method, storage medium and electronic device to achieve faster and higher-quality multi-view scene reconstruction and rendering.

[0006] The method for scene reconstruction and rendering based on point cloud of the present invention comprises the following steps:

[0007] Step 1: Obtain multiple RGB images of different perspectives in a single scene, their corresponding camera parameters and real ground depth map data, and build a data set;

[0008] Step 2: Perform multi-view 3D reconstruction on the point cloud with neural network features to obtain a feature-enhanced point cloud. Each point has three attributes: coordinates, eigenvalues, and confidence:

[0009] Step 3: Use feature-enhanced point clouds to simulate traditional volumetric radiation fields for rendering

[0010] According to the three attributes of each 3D point in the feature-enhanced point cloud P, three MLP networks are constructed to regress the volume density of any rendering position and the radiance that changes with viewing angle, which is used to simulate the radiation field. When obtaining the volume density and radiance, K points in a sphere with a radius of R and the target rendering position x as the center will be obtained. The final result is then gathered from the K points, and finally the NeRF-based discrete volume rendering method is used to obtain the final high-precision rendered image.

[0011] The three attributes of each point in the feature-enhanced point cloud in step 2 are: coordinates, eigenvalues, and confidence. The specific acquisition steps are:

[0012] The RGB images in the dataset and the corresponding camera parameters are input into the pre-trained MVSNet to obtain the depth probability, feature map and depth map of each RGB image;

[0013] Perform trilinear sampling on the depth probability of each RGB image and its two adjacent RGB images in the dataset to obtain a value between 0 and 1. This value is used as the confidence of the point to evaluate whether the point is on the surface. The larger the value, the more likely it is to be on the surface.

[0014] Project the depth map obtained above into three-dimensional space to obtain the coordinates of each point;

[0015] The feature map obtained above is downsampled using a two-dimensional convolutional network with a step size of 2, and the feature layer before each downsampling is extracted and combined to form a three-layer feature pyramid F:

[0016]

[0017] Among them, the subscript i represents the i-th image, and the superscripts represent different layers in the three-layer feature pyramid;

[0018] For the input N RGB images, the eigenvalue C of each 3D point j Use the following formula to get it from the three-layer feature pyramid:

[0019]

[0020] Among them, j represents different layers in the three-layer feature pyramid.

[0021] The feature-enhanced point cloud P is expressed as follows:

[0022] P={(p i , f i , γ i )|i=1,...,T} (3)

[0023] Among them, i represents the i-th point in the point cloud, p represents the coordinates of the point, f represents the feature value of the point, γ represents the confidence, and T is the total number of point clouds;

[0024] According to the three attributes of each 3D point in the feature-enhanced point cloud P, three MLP networks are constructed to regress the volume density of any rendering position and the radiance that changes with the viewing angle, which is used to simulate the radiation field. When obtaining the volume density and radiance, K points in a sphere with a radius of R and a target rendering position x as the center are obtained, and then the final result is gathered from the K points. Finally, the discrete volume rendering method based on NeRF is used to obtain the final high-precision rendering image, which is specifically as follows:

[0025] The first MLP network is used to predict the feature vector f of the target rendering position x, which is in the form of: f i，x =F(f i , xp i ). (4)

[0026] Where i represents the i-th neighboring point of the target rendering position x, (xp i ) represents x and p i The relative position of

[0027] The second MLP network is used to obtain the radiance r of the target rendering position x: the feature values ​​of the neighboring points of the target rendering position x are added up using the inverse distance weighted method to obtain a single feature describing the target rendering position x, which is defined as:

[0028]

[0029] Among them, W i Defined as:

[0030]

[0031] The third MLP network regresses the radiance r associated with the line of sight d from this single feature:

[0032] r=R(f x , d) (7)

[0033] The third MLP network is used to regress the volume density of the neighboring points of the target rendering position x, and then the inverse distance weighting is performed to obtain the volume density σ of the target rendering position x:

[0034] σ i =T(f i，x ) (8)

[0035]

[0036] The obtained radiance r and volume density σ are used in the discrete volume rendering method proposed in NeRF to complete multi-view high-precision scene rendering. The formula is as follows:

[0037]

[0038]

[0039] Where c is the color of the final rendered position, M is the total number of points whose colors need to be calculated, j indicates that this is the jth point, τ represents the volume transmittance, σ and r are the volume density and radiance obtained in the above steps, Δ is the distance between adjacent points, and t is the tth point in the shading range when calculating the volume transmittance.

[0040] Also included is a storage medium on which a computer program executable by a processor is stored, and when the computer program is executed, the steps of any one of the above-mentioned point cloud-based scene reconstruction and rendering methods are implemented.

[0041] Also included is an electronic device, including a processor, a memory, an input unit and a display unit, wherein the memory stores a computer program that can be executed by the processor, and when the computer program is executed, the steps of any one of the above-mentioned point cloud-based scene reconstruction and rendering methods are implemented; the input unit is used to input a specified new perspective; and the display unit is used to display a new perspective image.

[0042] The present invention processes the target scene into a point cloud with neural feature vectors to simulate the neural radiation field, which avoids sampling in an empty scene and can achieve faster radiation field reconstruction. On the one hand, the scene can be represented with less memory, and on the other hand, pre-training can be performed across scenes. Compared with other methods, it has significant advantages in both reconstruction and rendering. In addition, since the present invention uses the confidence of the point to determine whether a point is on the surface, it can ignore the noise and discrete points in the point cloud, thereby obtaining a better rendering effect. DETAILED DESCRIPTION

[0043] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below through specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0044] The method for scene reconstruction and rendering based on point cloud of the present invention comprises the following steps:

[0045] Step 1: Obtain multiple RGB images of different viewing angles in a single scene, the corresponding camera parameters (including pose parameters) and the depth map data of the ground truth, and construct a data set. There is no limit on the number of images and shooting angles, but the final point cloud must be complete. The present invention uses the open source DTU data set.

[0046] Step 2: Perform multi-view 3D reconstruction on the point cloud with neural network features to obtain a feature-enhanced point cloud. Each point has three attributes: coordinates, eigenvalues, and confidence:

[0047] (1) Obtaining the coordinates and confidence of the point

[0048] The RGB images in the data set and the corresponding camera parameters are input into the pre-trained MVSNet to obtain the depth probability, feature map and depth map of each RGB image. The present invention adopts the existing MVSNet.

[0049] The depth probability describes the possibility of each point being on the surface. Tri-linearly sample the depth probability of each RGB image and its two adjacent RGB images in the dataset to obtain a value between 0 and 1, which is used as the confidence of the point to evaluate whether the point is on the surface. The larger the value, the more likely it is to be on the surface.

[0050] Project the obtained depth map into three-dimensional space to obtain the coordinates of each point;

[0051] (2) Obtaining the feature value of a point

[0052] First, a three-layer feature pyramid is used to extract multi-view image features: for the feature map obtained in step (1), a two-dimensional convolutional network with a step size of 2 is used for downsampling, and the feature layer before each downsampling is extracted and combined to form a three-layer feature pyramid F:

[0053]

[0054] Among them, the subscript i represents the i-th image, and the superscripts represent different layers in the three-layer feature pyramid.

[0055] Next, for the N input RGB images, the eigenvalue C of each 3D point j It can be obtained from the three-layer feature pyramid using the following formula:

[0056]

[0057] Among them, j represents different layers in the three-layer feature pyramid;

[0058] (3) The feature value, coordinates and confidence of each point are combined as the attributes of each 3D point in the point cloud, thereby constructing a feature-enhanced point cloud;

[0059] Step 3: Use feature-enhanced point clouds to simulate traditional volumetric radiation fields for rendering

[0060] (1) The feature-enhanced point cloud P is expressed as follows:

[0061] P={(p i , f i , γ i )|i=1,...,T} (3)

[0062] Among them, i represents the i-th point in the point cloud, p represents the coordinates of the point, f represents the feature value of the point, γ represents the confidence, and T is the total number of point clouds;

[0063] (2) According to the three attributes of each 3D point in the feature-enhanced point cloud P, three MLP networks are constructed to regress the volume density of any rendering position and the radiance that varies with viewing angle, which is used to simulate the radiation field. When obtaining the volume density and radiance, K points in a sphere with a radius of R and the target rendering position x as the center are obtained. The final result is then gathered from the K points. Finally, a discrete volume rendering method based on NeRF is used to obtain the final high-precision rendered image.

[0064] The first MLP network is used to predict the feature vector f of the target rendering position x, which is in the form of: f i，x =F(f i , xp i ). (4)

[0065] Where i represents the i-th neighboring point of the target rendering position x, (xp i ) represents x and p i This method can describe the relationship between the target rendering position x and the neighboring point i, which can achieve better results when obtaining the radiance in the next step.

[0066] The second MLP network is used to obtain the radiance r of the target rendering position x: first, the feature values ​​of the neighboring points of the target rendering position x are added up using the inverse distance weighting method to obtain a single feature describing the target rendering position x (using the inverse distance weighting method allows closer points to contribute more to the single feature), and the single feature is defined as:

[0067]

[0068] Among them, W i Defined as:

[0069]

[0070] Then, a third MLP network is used to regress the radiance r associated with the line of sight d from this single feature: r=R(f x , d) (7)

[0071] The third MLP network is used to regress the volume density: first, the volume density of the neighboring points of the target rendering position x is regressed using the third MLP network, and then the volume density σ of the target rendering position x is obtained by inverse distance weighting:

[0072] σ i =T(f i ,x) (8)

[0073]

[0074] (3) The radiance r and volume density σ obtained above are used in the discrete volume rendering method proposed in NeRF to complete high-precision scene rendering from multiple perspectives. The formula is as follows:

[0075]

[0076]

[0077] Where c is the color of the final rendered position, M is the total number of points whose colors need to be calculated, j indicates that this is the jth point, τ represents the volume transmittance, σ and r are the volume density and radiance obtained in the above steps, Δ is the distance between adjacent shading points, and t is the tth point in the shading range when calculating the volume transmittance.

[0078] This embodiment further provides a storage medium on which a computer program executable by a processor is stored. When the computer program is executed, the steps of the method for scene reconstruction and rendering based on point cloud in this embodiment are implemented.

[0079] This embodiment also provides an electronic device, including a processor, a memory, an input unit and a display unit, wherein the memory stores a computer program that can be executed by the processor, and when the computer program is executed, the steps of the point cloud-based scene reconstruction and rendering method in this embodiment are implemented; the input unit is used to input a specified new perspective; and the display unit is used to display a new perspective image.

[0080] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The patent protection scope of the present invention should be defined by the claims.

Claims

1. A method for scene reconstruction and rendering based on point cloud, characterized in that The steps include: Step 1: Obtain multiple RGB images of different perspectives in a single scene, their corresponding camera parameters and real ground depth map data, and build a dataset; Step 2: Perform multi-view 3D reconstruction on the point cloud with neural network features to obtain a feature-enhanced point cloud, where each point has three attributes: coordinates, eigenvalues, and confidence. Step 3: Use the feature-enhanced point cloud to simulate the traditional volumetric radiation field for rendering; According to the three attributes of each 3D point in the feature-enhanced point cloud P, three MLP networks are constructed to regress the volume density of any rendering position and the radiance that changes with the viewing angle, which is used to simulate the radiation field. When obtaining the volume density and radiance, K points in a sphere with a radius of R and the target rendering position x as the center are obtained. The final result is then gathered from the K points. Finally, the discrete volume rendering method based on NeRF is used to obtain the final high-precision rendering image. The three attributes of each point in the feature-enhanced point cloud in step 2 are: coordinates, eigenvalues, and confidence. The specific acquisition steps are: The RGB images in the dataset and the corresponding camera parameters are input into the pre-trained MVSNet to obtain the depth probability, feature map and depth map of each RGB image; Perform trilinear sampling on the depth probability of each RGB image and its two adjacent RGB images in the dataset to obtain a value between 0 and 1. This value is used as the confidence of the point to evaluate whether the point is on the surface. The larger the value, the more likely it is to be on the surface. Project the depth map obtained above into three-dimensional space to obtain the coordinates of each point; The feature map obtained above is downsampled using a two-dimensional convolutional network with a step size of 2, and the feature layer before each downsampling is extracted and combined to form a three-layer feature pyramid F: Among them, the subscript i represents the i-th image, and the superscripts represent different layers in the three-layer feature pyramid; For the N input RGB images, the eigenvalue C of each 3D point j Use the following formula to get it from the three-layer feature pyramid: Among them, j represents different layers in the three-layer feature pyramid.

2. The scene reconstruction and rendering method according to claim 1, characterized in that: The feature-enhanced point cloud P is expressed as follows: P={{p i ,f i ,γ i )|i=1,...,T} (3) Where i represents the i-th point in the point cloud, p represents the coordinates of the point, f represents the feature value of the point, γ represents the confidence, and T is the total number of point clouds; According to the three attributes of each 3D point in the feature-enhanced point cloud P, three MLP networks are constructed to regress the volume density of any rendering position and the radiance that changes with the viewing angle, which is used to simulate the radiation field. When obtaining the volume density and radiance, K points in a sphere with a radius of R and a target rendering position x as the center are obtained, and then the final result is gathered from the K points. Finally, the discrete volume rendering method based on NeRF is used to obtain the final high-precision rendering image, which is specifically as follows: The first MLP network is used to predict the feature vector f of the target rendering position x, which is in the form of: f i,x =F(f i,x -p i )· (4) Where i represents the i-th neighboring point of the target rendering position x, (xp i ) represents x and p i The relative position of The second MLP network is used to obtain the radiance r of the target rendering position x: the feature values ​​of the neighboring points of the target rendering position x are added up using the inverse distance weighted method to obtain a single feature describing the target rendering position x, which is defined as: Among them, W i Defined as: The third MLP network regresses the radiance r associated with the line of sight d from this single feature: r=R(f x ,d) (7) The third MLP network is used to regress the volume density of the neighboring points of the target rendering position x, and then the inverse distance weighting is performed to obtain the volume density σ of the target rendering position x: s i =T(f i，x ) (8) The obtained radiance r and volume density σ are used in the discrete volume rendering method proposed in NeRF to complete multi-view high-precision scene rendering. The formula is as follows: Where c is the color of the final rendered position, M is the total number of points whose colors need to be calculated, j indicates that this is the jth point, τ represents the volume transmittance, σ and r are the volume density and radiance obtained in the above steps, Δ is the distance between adjacent points, and t is the tth point in the shading range when calculating the volume transmittance.

3. A computer-readable storage medium having stored thereon a computer program executable by a processor, wherein when the computer program is executed, the steps of any one of the point cloud-based scene reconstruction and rendering methods described in claims 1-2 are implemented.

4. An electronic device, comprising a processor, a memory, an input unit and a display unit, wherein the memory stores a computer program that can be executed by the processor, and when the computer program is executed, the steps of any one of the point cloud-based scene reconstruction and rendering methods described in claims 1-2 are implemented; the input unit is used to input a specified new perspective; and the display unit is used to display a new perspective image.

Images