A temporally stable neural rendering method for hair point clouds

Abstract

The present invention discloses a method for neural rendering of hair point clouds with temporal stability. In view of the problem that the existing point cloud-based neural rendering network cannot render high-quality hair with temporal stability, the present invention proposes a depth peeling and temporal stability enhancement network. By layered projection of the input point cloud model, the feature information of different layers is obtained, and the results are fused to adapt to the semi-transparent characteristics of hair; the trained results are input into the temporal stability enhancement network, and the dependency relationship between the current frame and the previous frames is obtained by reprojecting the point clouds between adjacent frames, and the final result of the current frame is generated, thereby ensuring the temporal stability of the training results. The present invention infers the results of global illumination through the information of local illumination, and can ensure both rendering quality and rendering speed. It overcomes the shortcoming that the traditional method of using global illumination algorithms such as ray tracing to render hair is very time-consuming.

Description

Technical Field

[0001] The present invention relates to the fields of computer graphics and neural rendering, and in particular to a time-domain stable hair point cloud neural rendering method. Background Art

[0002] Hair plays a vital role in the realistic performance of computer-generated characters and animations. In the film industry, the synthetic animals in movies such as Godzilla vs. Kong and Zootopia have abundant hair around their bodies. It is precisely because of the hair generated by special effects that is very close to the real thing that the audience has a more immersive experience. Similarly, in the game industry, the high-quality hair of a large number of characters in Sekiro is very important for gamers to have a better gaming experience.

[0003] Therefore, how to generate high-quality hair is a very important issue. Traditional hair is generally expressed based on patches or curves, and rendered using rasterization or ray tracing, which can lead to poor shading effects or slow rendering speeds. And if an offline renderer is used to render a frame of high-quality hair, it usually takes half an hour to an hour, which is very expensive.

[0004] In recent years, with the rise of neural networks, computer graphics has also ushered in new developments. The concept of neural rendering has gradually become popular, aiming to use neural networks to store the geometric lighting information of the scene, and use the learning and migration capabilities of neural networks to obtain photo-realistic rendering results. The traditional rendering process includes scene construction, setting material parameters, camera parameters, arranging scene light sources, and rendering using global illumination algorithms.

[0005] Neural rendering can be well combined with traditional rendering processes. In neural rendering, scenes are expressed in different ways, such as implicit surfaces, meshes, and point clouds, which are adapted to different rendering requirements. For objects with very rich geometry, such as hair, a better way is to use point clouds to describe them. Point clouds can better express the relationship between the mutual occlusion of hair, and point clouds will not be limited by the resolution of the mesh. In addition, the scene's light source information and scene point cloud information can be input into the neural network as auxiliary parameters to obtain better rendering results. Summary of the invention

[0006] The purpose of the present invention is to provide a method for neural rendering of hair point clouds with temporal stability in view of the deficiencies of the prior art. The present invention can greatly improve the temporal stability of network rendering results and can improve the quality of hair rendering. At the same time, the rendering network in this article can also be used to infer global illumination results.

[0007] The objective of the present invention is achieved through the following technical solution: A method for neural rendering of hair point clouds with temporal stability, comprising the following steps:

[0008] Step 1: Use the ray tracing algorithm to generate a hair rendering dataset, including local illumination results and global illumination results. Build a basic neural rendering network according to the point cloud-based neural rendering method framework, including point cloud projection to obtain feature maps, use the U-Net network to extract and render feature maps, and use the perceptual loss function as a constraint on the network rendering results.

[0009] Step 2: Project the point cloud layer by layer into the network built in step 1 to obtain point cloud projection feature maps at different levels and rendering results of neural networks at different levels, and perform transparency fusion on the rendering results of each layer to obtain the final rendering result;

[0010] Step 3: For the rendering results after layered projection in step 2, during the network training process, the constraint relationship between adjacent frames is considered to reduce the projection error caused by mutual occlusion of point clouds, and obtain a stable rendering result in the time domain.

[0011] Step 4: Connect the rendering result obtained in step 3 to a small convolutional neural network to obtain the global illumination inference network.

[0012] Furthermore, step one is implemented by the following sub-steps:

[0013] (1.1) Use Blender to create a hair model consisting of hundreds of thousands of hairs, and generate 540 camera surround local illumination, global illumination rendering results and corresponding transparency masks for each model. The local illumination results are mainly used for the network to infer the global illumination results. The dataset generated by the global illumination results has rich hair details and lighting information, which is very important for exploring the capture of high-frequency details by neural networks;

[0014] (1.2) Use CUDA to rasterize and project the hair point cloud, and obtain a feature map of (32+3)*600*600, where 32 is the feature vector dimension of each point in the point cloud, and 3 is the unit direction vector corresponding to each projection point; input the feature map into the U-Net network, encoder and decoder, and obtain a (3+1)*600*600 RGBA network rendering result. The network loss function uses a perceptual loss function, which can constrain the network rendering result to be closer to human perception to a certain extent.

[0015] Furthermore, step 2 is implemented through the following sub-steps:

[0016] (2.1) Based on step 1, a depth peeling scheme based on curved surface is introduced for hair point cloud rendering, which can better meet the characteristics of hair point cloud and capture richer details. The details of step 2 are as follows: For a given viewing angle and the corresponding camera parameters are known, the point cloud is first projected to the viewing angle by rasterization, and the feature vector of the point cloud is written into the projection result map as a pseudo color value. After a single projection, the index, depth and feature vector corresponding to the current projection result are returned. The returned index will be used to delete the first projection point in the second projection, and then the above process is repeated to obtain the projection of each layer; when there is no point cloud inside the hair, directly using layered projection will cause the network to eventually fuse the hair point clouds before and after the model, resulting in incorrect rendering results. Therefore, starting from the second layer, the depth difference between the current layer and the previous layer will be used to judge the validity of each point to avoid the adverse effects of the invisible points on the other side of the object on the rendering. When the depth difference is too large, the depth value and transparency value of the corresponding invalid point are set to zero to ensure that the points behind these invalid points are still regarded as invalid points and no back-propagation gradient values ​​are obtained during training;

[0017] (2.2) The feature projection images of different layers are input into the rendering network, and the rendering results are transparently fused to obtain the final result.

[0018] Furthermore, step three is specifically as follows:

[0019] The method of steps 1 and 2 can produce high-quality rendering results, but the results lack temporal stability. When the network is used to generate a surrounding camera sequence, a flickering video sequence will be obtained. Therefore, continuous sequence frames need to be added during the training process. For adjacent frames I a and I b , since the corresponding camera parameters and depth map are known, I is obtained by reprojection a Each pixel and I b The mapping relationship between the two is compared, the color values ​​and depth values ​​of the two are compared, and the pixels with too large difference in color value and depth value are filtered out to obtain the reprojected reference image I a→b , the final result is generated in the following way: Let I a-1 with I a is the result of two adjacent frames obtained by transparency fusion, is the final result of the time domain stabilization of the previous frame, which contains the information of the long time series, then:

[0020]

[0021]

[0022] where w 1～4The mixed weights are w1+w2=1, w3+w4=1, I a-1→a Representative I a-1 To I a The reprojection error map of represent To I a The reprojection error map of isI a-1→a and The weighted fusion result of Represents the final temporal stabilization result of the current frame.

[0023] Furthermore, step four is specifically as follows:

[0024] The local illumination result obtained by ray tracing is used as the input of the network, and the result after transparency fusion is used as the output. Then a small convolutional neural network is connected in series to obtain the result of inferring global illumination from the local illumination of the hair.

[0025] The beneficial effect of the present invention is that, based on the point cloud-based neural rendering framework, the present invention optimizes the rendering of hair objects. First, a high-quality hair dataset generated by offline rendering is used, and the point cloud and corresponding camera parameters are used as network input. A high-quality new perspective rendering result is obtained. The method proposed in the present invention greatly improves the temporal stability of point cloud neural rendering, and ensures the rendering quality and rendering speed. At the same time, by modifying the corresponding point cloud-based deep peeling neural rendering network, the network can also be used to infer the results of global illumination of hair, and finally a very realistic global illumination result map can be rendered, which greatly reduces the hair rendering time. DETAILED DESCRIPTION

[0026] The time-domain stable hair point cloud neural rendering method of the present invention comprises the following steps:

[0027] Step 1: Use the ray tracing algorithm to generate a hair rendering dataset, including local illumination results and global illumination results. Build a basic neural rendering network according to the point cloud-based neural rendering method framework, including point cloud projection to obtain feature maps, use the U-Net network to extract and render feature maps, and use the perceptual loss function as a constraint on the network rendering results.

[0028] This step is the technical basis of the present invention and is divided into the following sub-steps.

[0029] 1) Use Blender to create a hair model consisting of hundreds of thousands of hairs, and generate 540 camera surround local illumination, global illumination rendering results and corresponding transparency masks for each model. The local illumination results are mainly used for the network to infer the global illumination results. The dataset generated by the global illumination results has rich hair details and lighting information, which is very important for exploring the capture of high-frequency details by neural networks.

[0030] The input of the neural rendering network includes the internal and external parameters of the camera corresponding to 540 viewing angles, the approximately 450,000 point cloud model obtained by sampling the corresponding hair model, the ray tracing rendering results of the corresponding viewing angles, and the transparency mask.

[0031] 2) Use CUDA to rasterize and project the discrete point cloud, write the feature vector of each point cloud as a pseudo color value into the feature projection, and then input the result into the neural rendering network. The structure of the feature map of the point cloud projection is C*H*W, where C is the feature vector dimension, H and W are the height and width of the image respectively, and the feature dimension can generally be taken as 32. Calculate the direction vector from the camera position to the position of each projected point and normalize it. Use zero vectors to fill the position without points to obtain a 3*H*W tensor, which is connected with the former to obtain a (C+3)*H*W tensor as the input of the neural rendering network. The neural rendering network uses RGB encoder and RGB decoder to output RGB values, and then connects the RGB values ​​with the input to obtain a (C+6)*H*W tensor input to the Alpha encoder and Alpha decoder to obtain the Alpha mask. The encoder and decoder use the up and down sampling method of Antialiasing CNN to ensure translation stability as much as possible, and the convolution uses gated convolution to filter invalid features.

[0032] Step 2: Project the point cloud in layers onto the network built in step 1 to obtain point cloud projection feature maps at different levels and rendering results of neural networks at different layers, and perform transparency fusion on the rendering results of each layer to obtain the final rendering result.

[0033] This step is one of the core technologies of the present invention. Step 2 is implemented by the following sub-steps:

[0034] 1) For the model point cloud and the internal and external parameters of the camera, CUDA is used for rasterization to obtain the point cloud projection results, including the index, depth and feature vector of the point at each pixel position. During the projection process, the point will be splatted according to the distance from the camera, that is, the depth, that is, the closer point will occupy more than one pixel to better fill the gap between the points. The returned index will be used to delete these points, that is, the peeling operation for the next layer of projection. The returned depth value only retains the result of the first layer for reprojection. The returned feature vector corresponds to the point. For the position not covered by the point, the default vector is uniformly filled. After one projection is completed, the point corresponding to the returned index is deleted from the point cloud and the next projection is performed. Starting from the second layer of projection, the depth difference between the current layer and the previous layer will be used to judge the validity of each point to avoid the point on the other side of the object that should not be visible from causing adverse effects on the rendering. When the depth difference is too large, the depth of the corresponding invalid point is set to zero, and the alpha value is set to zero to ensure that the points after these invalid points are still regarded as invalid points, and no back-propagation gradient values ​​are obtained during training.

[0035] 2) After the neural rendering network, the result of L*4*H*W can be obtained, where L refers to the number of layers, and 4 refers to the four channels of RGBA. Transparency fusion will be performed in L dimension. First, the blending weight w of the i-th layer is calculated by the Alpha value. i , where α i Represents the Alpha value of the i-th layer:

[0036]

[0037] The final Alpha value is calculated from the blending weight:

[0038]

[0039] The final RGB value is calculated by the mixing weight, where c1 represents the RGB value of the lth layer and c represents the final RGB value:

[0040]

[0041] Step 3: For the rendering results after layered projection in step 2, during the network training process, the constraint relationship between adjacent frames is considered to reduce the projection error caused by mutual occlusion of point clouds, and obtain a stable rendering result in the time domain.

[0042] Step three is also one of the core technologies of the present invention, and the specific problems and solutions solved are:

[0043] The surface depth value obtained by depth peeling and the camera internal and external parameters can restore the world coordinates of the surface point cloud. For two renderings generated by different perspectives, that is, different camera external parameters, denoted as I a ,Ib , then for I a Any pixel P on a,I (u a ,v a ), by its depth d a , camera intrinsic parameter K a , camera external parameters [R a |T a ]have:

[0044]

[0045] P a,w Then P a,I The corresponding world coordinates are generated by I b Parameters for reprojection:

[0046]

[0047] This established I a Each pixel position is related to I b The mapping relationship between the pixel positions on the image is called reprojection. Through reprojection, we can get I a to I b The color reference value of the reprojected pixel coordinates is a floating point type. Bilinear interpolation can be used for color calculation, but this method is equivalent to a blurring operation, which affects the details of the hair texture. Therefore, the nearest neighbor value is used, that is, the nearest pixel color is directly assigned. a ,I b are two adjacent frames, P a,I With P b,I There is a mapping relationship. If the colors of these two pixels are close, it means that the stability of the adjacent frames is good. However, since the result of reprojection is only a reference value, when the position change between the two frames is large, the reference value may not be available, so two more filterings are performed. First, filter out d a With d b Pixels with too large difference (corresponding to pixel position depth) are filtered out. a With c b Pixels with too large difference (corresponding to the color of the pixel position), the color of these filtered pixel positions is modified to match I b At the same time, since the reprojection depends on the point cloud, the color is also kept consistent in the position without point cloud coverage, so the reprojection reference image can be obtained, which is recorded as I a→b To reduce the bias caused by reprojecting only adjacent frames and suppress instabilities over longer periods, the final reference image is generated as follows:

[0048] The final result is generated in the following way: Let I a-1 with Ia is the result of two adjacent frames obtained by transparency fusion, is the final result of the time domain stabilization of the previous frame, which contains the information of the long time series, then:

[0049]

[0050]

[0051] where w 1～4 The mixed weights are w1+w2=1, w3+w4=1, I a-1→a Representative I a-1 To I a The reprojection error map of represent To I a The reprojection error map of isI a-1→a and The weighted fusion result of Represents the final temporal stabilization result of the current frame.

[0052] The loss value is calculated as:

[0053]

[0054] Where MSE stands for mean square error, Perceptual stands for perceptual loss function, and to avoid the back propagation chain being too long, Detach from the computational graph, providing only color reference without calculating gradients.

[0055] Temporal stability enhancement is performed after the neural rendering network training is completed, because the reference value obtained by the untrained neural rendering network is not meaningful. First, generate continuous surrounding camera external parameters, and then use the temporal stability enhancement network training for 1 to 2 cycles.

[0056] Step 4: Connect the rendering result obtained in step 3 to a small convolutional neural network to obtain the global illumination inference network.

[0057] Step 4 is also one of the core technologies of the present invention, and the specific problems and solutions solved are:

[0058] The global illumination inference algorithm first uses the same depth peeling, neural rendering, and transparency fusion process as steps 1 and 2.

[0059] Among them, the input of local illumination constitutes a 3*H*W tensor, where H and W are the height and width of the image respectively. The output result of the neural rendering network encodes the result of the multiple scattering part of the hair, and its output is a 4*H*W tensor. The two are concatenated to obtain a 5*H*W tensor I. In addition, a two-layer convolutional neural network is constructed, which is mainly used to extract the details of the multiple scattering results output by the neural rendering network and perform filtering operations on them. The first layer of the convolutional neural network uses 8 5*3*3 convolution kernels for filtering, and keeps the width and height dimensions of the output unchanged, and then uses 4 8*3*3 convolution kernels for filtering to obtain the final RGBA result. This network is equivalent to filtering and fusing with the local illumination result to obtain the final rendering result.

Claims

1. A method for neural rendering of hair point clouds with temporal stability, characterized in that: The following steps are involved: Step 1: Use the ray tracing algorithm to generate a hair rendering dataset, including local illumination results and global illumination results. Build a basic neural rendering network according to the point cloud-based neural rendering method framework, including point cloud projection to obtain feature maps, use the U-Net network to extract and render feature maps, and use the perceptual loss function as a constraint on the network rendering results. Step 2: Project the point cloud layer by layer into the network built in step 1 to obtain point cloud projection feature maps at different levels and rendering results of neural networks at different levels, and perform transparency fusion on the rendering results of each layer to obtain the final rendering result; Step 3: For the rendering results after layered projection in step 2, during the network training process, the constraint relationship between adjacent frames is considered to reduce the projection error caused by mutual occlusion of point clouds, and obtain a rendering result that is stable in the time domain; Step 4: Connect the rendering result obtained in step 3 to a small convolutional neural network to obtain the global illumination inference network.

2. The method for neural rendering of hair point cloud with temporal stability according to claim 1, characterized in that: The step 1 is implemented by the following sub-steps: (1.1) Use Blender to create a hair model consisting of hundreds of thousands of hairs, and generate 540 camera surround local illumination, global illumination rendering results and corresponding transparency masks for each model. The local illumination results are mainly used for the network to infer the global illumination results. The dataset generated by the global illumination results has rich hair details and lighting information; (1.2) Use CUDA to perform raster projection on the hair point cloud, and obtain a feature map of (32+3)*600*600, where 32 is the feature vector dimension of each point in the point cloud, and 3 is the unit direction vector corresponding to each projection point; input the feature map into the U-Net network, encoder and decoder, and obtain a (3+1)*600*600 RGBA network rendering result; the network loss function uses the perceptual loss function.

3. The method for neural rendering of hair point cloud with temporal stability according to claim 1, characterized in that: The step 2 is implemented by the following sub-steps: (2.1) For a given viewing angle and the corresponding camera parameters are known, the point cloud is first projected to the viewing angle by rasterization, and the feature vector of the point cloud is written into the projection result map as a pseudo-color value. After a single projection, the index, depth and feature vector corresponding to the current projection result are returned; the returned index will be used to delete the first projection point in the second projection, and then the above process is repeated to obtain each layer of projection; starting from the second layer of projection, the depth difference between the current layer and the previous layer will be used to determine the validity of each point. When the depth difference is too large, the depth value and transparency value of the corresponding invalid point will be set to zero to ensure that the points behind these invalid points are still regarded as invalid points, and no back-propagation gradient values ​​will be obtained during training; (2.2) The feature projection images of different layers are input into the rendering network, and the rendering results are transparently fused to obtain the final result.

4. The method for neural rendering of hair point cloud with temporal stability according to claim 1, characterized in that: The step three is specifically as follows: In the training process, continuous sequence frames are added. For adjacent frames I a and I b , since the corresponding camera parameters and depth map are known, I is obtained by reprojection a Each pixel and I b The mapping relationship between the two is compared, the color value and depth value information of the two are compared, and the pixels with too large difference between the color value and the depth value are filtered out to obtain the reprojected reference image I a→b , the final result is generated in the following way: Let I a-1 with I a is the result of two adjacent frames obtained by transparency fusion, is the final result of the time domain stabilization of the previous frame, which contains the information of the long time series, then: where w 1～4 The mixed weights are w1+w2=1, w3+w4=1, I a-1→a Representative I a-1 To I a The reprojection error map of represent To I a The reprojection error map of isI a-1→a and The weighted fusion result of Represents the final temporal stabilization result of the current frame.

5. The method for neural rendering of hair point cloud with temporal stability according to claim 1, characterized in that: The step four is specifically as follows: the local illumination result obtained by using ray tracing is used as the input of the network, and the result after transparency fusion of the network is used as the output, and then a small convolutional neural network is connected in series to obtain the result of inferring the global illumination from the local illumination of the hair.

Images