An editable free-viewpoint view synthesis method for street scenes and moving vehicles

Abstract

The application discloses an editable free-viewpoint view synthesis method for street scenes and moving vehicles, and specifically comprises the following steps: representing a street scene and a virtual vehicle model in the form of a neural radiance field; combining a physical rendering-based method to estimate the intrinsic properties and scene illumination of the street scene and the vehicle model; representing the scene illumination as a series of spherical Gaussian functions, transferring the real-world scene to the virtual vehicle model, and combining the intrinsic properties to render a virtual vehicle driving in a real street scene with strong reality; further improving the reality of the vehicle rendering effect through a deep mixed sampling-based generation network; and rendering the effect of the same scene under different illumination conditions based on the intrinsic properties of the scene and the vehicle. The application respectively establishes a neural radiance field for the vehicle and the scene, harmoniously inserts the vehicle into a real-world street scene, and realizes free-viewpoint rendering and scene editing with a large range, multiple angles and variable illumination.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a kind of editable free viewpoint view synthesis method for street view and moving vehicle based on neural radiance field. BACKGROUND

[0002] Humans can conceive unseen scenes in their minds, which greatly enhances the effectiveness of learning. Inspired by this ability, researchers in the field of autonomous driving focus on creating virtual environments containing synthetic vehicles to train perception models. Extensive research shows that this approach is crucial to solving challenges related to 3D vision decision problems.

[0003] Traditional 3D simulation involves the use of computer-aided design (CAD) models. However, generating a wide and diverse range of street views using only CAD models has proven to be very inefficient and costly in practical operations. Currently, some scholars have used 2D picture-based generation methods to synthesize editable street scene pictures (Wei Li, CW Pan, Rong Zhang, JP Ren, YX Ma, Jin Fang, FL Yan, QC Geng, XY Huang, HJ Gong, et al. Aads: Augmented autonomous driving simulation using data-driven algorithms. Science Robotics, 4(28): eaaw0863, 2019), using real scene pictures or depth maps as prior knowledge of the scene. These methods still have major problems in synthesizing and 3D information related to rotation, shadow occlusion relationships, etc. In recent times, the introduction of neural radiance field (Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020) has further improved the quality of 3D information synthesis, allowing for the rendering of high-quality street scene pictures from free viewpoints.

[0004] The above researches aim to synthesize street scene pictures with photo-realism. However, the training of the neural radiance field model requires the collection of multi-view pictures of the scene. Due to the complexity of the real-world environment, changes in lighting, changes in dynamic objects, changes in the appearance of the environment, etc. often occur in street scenes, which makes it difficult to ensure consistency when collecting multi-view pictures, and it is impossible to synthesize realistic street scenes under different time periods and lighting conditions. Therefore, how to decouple the intrinsic properties of the environment to match different lighting conditions is still a problem to be solved. SUMMARY

[0005] In order to synthesize a street scene with photo-realism, the present application provides an editable free-viewpoint view synthesis method for street scenes and moving vehicles based on a neural radiance field.

[0006] In order to achieve the above-mentioned purposes, the technical solutions adopted by the method of the present application are as follows:

[0007] An editable free-viewpoint view synthesis method for street scenes and moving vehicles, comprising the following steps:

[0008] S1, acquiring a street scene picture dataset, calculating the camera pose and camera parameters when the street scene picture is taken; using a normal and depth prediction network to obtain normal and depth prior conditions predicted based on the picture;

[0009] S2, based on the street scene picture dataset, using a neural radiance field algorithm to establish a street scene neural radiance field model for the street scene, using a multi-layer perceptron for the street scene to predict the intrinsic property parameters of the scene;

[0010] S3, constructing street scene neural radiance field model data according to the pictures in the street scene picture dataset in step S1 and the normal and depth prior conditions predicted based on the picture; using a rendering method based on physical principles to combine the scene lighting with the intrinsic property parameters to render a street scene picture, wherein a set of spherical Gaussian functions is used to represent the scene lighting; using a prior L1 loss function and an image L2 loss function to train the street scene neural radiance field model;

[0011] S4, collecting a large number of 3D models of vehicles to construct a three-dimensional virtual vehicle model dataset and render multi-view pictures of the vehicle model; using a neural radiance field algorithm to establish a vehicle neural radiance field model for the vehicle; using a multi-layer perceptron for the vehicle to predict the intrinsic property parameters of the vehicle;

[0012] S5, training the vehicle neural radiance field model according to the multi-view pictures of the vehicle model rendered in step S4 to realize three-dimensional reconstruction of the vehicle;

[0013] S6, combined with the street scene neural radiance field model and the vehicle neural radiance field model, the geometry and internal properties of the two are fused according to the depth relationship by using a hybrid ray tracing algorithm, and an editable free viewpoint view of the street scene and the moving vehicle is synthesized by using a renderer based on physical principles.

[0014] The present application constructs a three-dimensional virtual vehicle model dataset, which contains vehicle models of various types, colors and ages, and realizes the decoupling of the scene and the internal properties of the vehicle based on vehicle and scene data, inserts the vehicle harmoniously into the real-world street scene, and further realizes the free fusion of high realism, free viewpoint rendering and scene editing of a large range, multiple angles and variable lighting. The present application first realizes the editable free viewpoint view synthesis of the street scene and the moving vehicle based on the neural radiance field, and fills the research gap in this regard. The proposed method can be widely applied in the fields of autonomous driving, film production, game development, etc., and has high practical value and development prospect. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The flowchart of the method of the present application.

[0016] Figure 2 The schematic diagram of the three-dimensional vehicle model dataset in the embodiment of the present application.

[0017] Figure 3 The schematic diagram of hybrid sampling in the embodiment of the present application.

[0018] Figure 4 The flowchart of the running stage in the embodiment of the present application.

[0019] Figure 5 The result display diagram in the embodiment of the present application, (a) internal property decomposition result, (b) fusion result of different vehicle types and different scenes. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0021] In view of the defects in the prior art, the application proposes a hybrid solution which uses real captured sparse videos and synthetic 3D car composite materials to simulate street scenes to synthesize realistic free-viewpoint rendering in various scenes. First, the street scene and the virtual vehicle model are represented in the form of neural radiance fields; then the internal properties and scene lighting of the street scene and the vehicle model are estimated by combining a physically based rendering method; the scene lighting is represented as a series of spherical Gaussian functions, which is transferred from the real-world scene to the virtual vehicle model, combined with the internal property to render the virtual vehicle driving in the real street scene with strong realism; further improve the realism of the vehicle rendering effect through the deep hybrid sampling based generation network; finally, based on the internal properties of the scene and the vehicle, the effect of the same scene under different lighting conditions is rendered. As shown in Figure 1 The specific process of the application is as follows:

[0022] 1. There are a large number of real captured street view datasets at present, which contain a wide variety of street scenes under real-world lighting information. Based on the large street scene dataset (Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.) which has been published, the street scene pictures are obtained, and the camera pose and camera parameters when the street scene pictures are taken are calculated according to the GPS / IMU information in the dataset. The normal and depth prediction network (NeRF) is used to obtain the neural radiance field of the street scene, and the neural radiance field of the virtual vehicle model is obtained by training the neural radiance field of the street scene and the virtual vehicle model. Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3D common corruptions and data augmentation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 18963-18974, 2022.) get the normal and depth prior conditions based on the picture prediction. Among them, the street scene picture data set contains street scene pictures in 6 directions of front, left front, left rear, right front, right rear and rear

[0023] 2. Based on the above street scene data set, a street scene neural radiance field model is established for the street scene using the neural radiation field algorithm. First, according to the camera intrinsic parameters, the pixel coordinates of the picture are converted to the three-dimensional space coordinates of the camera, and then through the pose parameters of the camera, the three-dimensional space coordinates of the camera are converted to the three-dimensional space coordinates of the real world. The three-dimensional space coordinates of the real world are used as the input of the neural radiance field model, and the color of the corresponding pixel is the ground-truth color. The normal and depth predicted by the corresponding pixel are the ground-truth normal and depth.

[0024] 3. Realize the three-dimensional reconstruction of the scene geometry. First, select a scene from the data set, and use the multi-layer perception network for street scenes to predict the intrinsic attribute parameters of the scene such as basic color, roughness, metal degree, normal and depth. The input three-dimensional space coordinates are first passed through a hash encoder to obtain the hash features corresponding to each space coordinate. This embodiment designs a 3-layer geometric feature extraction multi-layer perception network, which takes the hash features output by the hash encoder as input, and outputs the voxel density σ and the intermediate vector b for each space coordinate through the multi-layer perception network. Then, this embodiment designs an intrinsic feature extraction network, which is composed of a 3-layer multi-layer perception network for normal extraction and a 3-layer multi-layer perception network for attribute extraction. The intermediate vector b is input into the normal extraction network and the attribute extraction network respectively to obtain the predicted normal n, roughness α, metal degree m and basic color c.

[0025] 4. A set of M spherical Gaussian functions is used to represent the scene lighting, which can be represented as:

[0026]

[0027] where k is the current spherical Gaussian function number, ω idenotes the direction of incident light, i denotes incident in, φ denotes the axial direction of the spherical Gaussian function, λ denotes the frequency of the spherical Gaussian function, and μ denotes the amplitude of the spherical Gaussian function.

[0028] 5. Physically-based rendering method based on rendering equation and simplified Disney principle (Brent Burley and Walt Disney Animation Studios. (2012). Physically-based shading at Disney. In SIGGRAPH, pages 1-7, vol. 2012.), which uses a rendering equation based on a spherical Gaussian function, combines a spherical Gaussian form of a light condition, and obtains a final rendering color. The BRDF function in the existing rendering equation is expressed as two parts of a diffuse reflection item and a specular reflection item: f r = f d + f s wherein the diffuse reflection item is mainly related to a base color and a metallicity: wherein k d is a diffuse reflection coefficient of an object, and m is a metallicity of the object. The specular reflection item is converted into a representation based on a spherical Gaussian function:

[0029]

[0030] wherein ω o is an observation direction, h is a half vector between the observation direction and the incident light direction ω i , n is a surface normal direction, and α is a surface roughness, is a product of Fresnel effect and shadow effect in the rendering equation. As an example, the embodiment uses a fast integral calculation method of the spherical Gaussian function (Wang, J., Ren, P., Gong, M., Snyder, J., & Guo, B. (2009). All-frequency rendering of dynamic, spatially-varying reflectance. In SIGGRAPH Asia, pages 1-10.), avoids complex calculation, and directly obtains a result of the rendering equation.

[0031] 6. Due to the complexity of the street view, the existence of various objects in the scene blocks the light, resulting in the formation of shadows in the entire scene. By using a volume rendering formula, a light ray tracing method is used to determine whether the light ray collides with the object in the scene; if a collision occurs, the visibility of the light ray is low, and a shadow is generated at the corresponding position in the image. The embodiment calculates a shadow map by uniformly sampling N l incident light rays in the upper hemisphere, and the incident light ray (direction lk The visibility V of a point p can be represented as the volume opacity of a ray r(t) = p + t k , where t is the distance from p to a point on the ray. Then the ray is traced in the scene to obtain the visibility using volume rendering. This method takes into account the complex interactions of light and objects in urban street scenes, accurately representing shadows, mathematically represented as follows:

[0032]

[0033] where k denotes the index of the incident ray, and finally, the shadow map is added to the scene by pixel-wise multiplication.

[0034] 7. When calculating the loss function of the rendering in step 5, the L2 loss function between the predicted color and the true color in the input image is used as supervision, which is expressed as:

[0035]

[0036] where, is the loss function of the rendered color; r is the camera ray, is the set of all camera rays. C render is the RGB value calculated by the rendering process of each camera ray , while C gt is the corresponding true color.

[0037] 8. When calculating the loss function of the intrinsic attribute in step 3, the L1 loss function of the predicted intrinsic attribute and the ground-truth intrinsic attribute is used as supervision, and additionally, for the normal, the normal extracted according to the voxel density is added as additional supervision, which is expressed as:

[0038]

[0039] L depth = Σ |d gt -d|

[0040] where, is the loss function of the normal, L depth is the loss function of the depth, is the loss function of the scene base color; n gt is the normal prior, is the reference normal direction extracted based on the voxel density σ, n is the normal output by the multi-layer perception network, d gt is the depth prior, d is the scene depth output by the multi-layer perception network.

[0041] An end-to-end training scheme is achieved by optimizing the multi-layer perceptron network that computes intrinsic properties in step 3 and the spherical Gaussian function that represents illumination in step 4.

[0042] As shown in Figure 2 The present application collects a large number of 3D vehicle models to build a virtual vehicle library. To seamlessly integrate these 3D models into a street scene, a vehicle neural radiance field is constructed to decompose the intrinsic properties of a vehicle. This neural radiance field takes the 3D position of a vehicle in canonical space as input and obtains the corresponding spatial features through a 512x512x512 tri-plane encoder. Subsequently, it obtains the density through a 1-layer perceptron with a hidden layer dimension of 64 and the intrinsic properties through a 2-layer perceptron with a hidden layer dimension of 128. These intrinsic properties include base color, material, and normal, which can be directly input into a street scene renderer. This neural radiance field is initially trained using vehicle images and then queried in the fusion stage.

[0043] As shown in Figure 3 The present application uses a hybrid sampling method to integrate vehicles and street scenes together. In addition to the original sampling points within the scene, a set of additional sampling points is introduced in the 3D bounding box of the vehicle when the sampling ray intersects the 3D bounding box of the vehicle in the scene. These additional points are specifically used to sample the vehicle and compute the intrinsic properties of the vehicle. After sampling both the vehicle and the scene, the sampling points are sorted and combined according to their depth in the real-world coordinates to obtain the combined sampling points {X}. Then, let r = o + td represent a camera ray with origin o and direction d, the color and intrinsic properties of the vehicle and scene fusion are obtained by performing standard volume rendering, presenting the correct appearance consistent with the depth occlusion relationship.

[0044] As shown in Figure 4As shown, the embodiment first divides the scene and the vehicle into two branches for processing. For the scene branch, the input three-dimensional coordinates are obtained according to the camera parameters, are subjected to hash coding, are input into a multi-layer perception network for calculating geometric shapes, the voxel density and the intermediate vector are obtained, the intermediate vector is input into a multi-layer perception network for calculating intrinsic attributes, the intrinsic attributes of the voxels are obtained, and the voxel density and the intrinsic attributes are used to obtain the final result by using a volume rendering method. For the vehicle branch, a similar strategy is adopted, the input three-dimensional coordinates are input into a multi-layer perception network for calculating geometric shapes by using a three-plane coding method, and finally the intrinsic attributes of the vehicle are obtained by combining volume rendering. In order to harmoniously insert the vehicle into the scene, according to the light information represented by the spherical Gaussian function extracted from the scene, the light is simultaneously applied to the scene and the vehicle, a physically-based rendering method is used to obtain the vehicle insertion effect conforming to the real light effect, a hybrid sampling method is combined to freely edit the insertion position and direction of the vehicle, the occlusion relationship is correctly handled, and finally the fusion result is obtained. Since the embodiment decouples all the intrinsic attributes of the scene and the vehicle, the light of the scene can also be flexibly modified and edited, and the diversity of the synthesized scene is greatly improved.

[0045] The final fusion result is shown in Figure 5 The vehicle is inserted into the scene, and the position of the vehicle can be freely moved, and the scene environment presents a harmonious effect.

[0046] The application first realizes an editable free-viewpoint view synthesis method of a street scene and a moving vehicle based on a neural radiance field, fills the research gap in this regard. The proposed method can be widely applied to the fields of automatic driving, film production, game development and the like, and has high practical value and development prospect.

Claims

1. An editable free-viewpoint view synthesis method for street scenes and moving vehicles, characterized in that, The method comprises the following steps: S1, obtaining a street scene picture data set, calculating the camera pose and camera parameters when the street scene picture is taken; using a normal and depth prediction network to obtain the normal and depth prior conditions predicted based on the picture; S2, based on the street scene picture data set, using a neural radiation field algorithm to establish a street scene neural radiation field model for the street scene, and using a multi-layer perception for the street scene to predict the intrinsic attribute parameters of the scene; S3, constructing street scene neural radiation field model data according to the pictures in the street scene picture data set in step S1 and the normal and depth prior conditions predicted based on the pictures, and using a rendering method based on physical principles to combine the scene illumination with the intrinsic attribute parameters to render a street picture, wherein a set of spherical Gaussian functions is used to represent the scene illumination; the street scene neural radiation field model is trained using a prior L1 loss function and an image L2 loss function; S4, collecting a large number of 3D models of vehicles to construct a three-dimensional virtual vehicle model data set and render multi-view pictures of the vehicle models; using a neural radiation field algorithm to establish a vehicle neural radiation field model for the vehicle; and using a multi-layer perception for the vehicle to predict the intrinsic attribute parameters of the vehicle; S5, training the vehicle neural radiation field model according to the multi-view pictures of the vehicle models rendered in step S4 to realize three-dimensional reconstruction of the vehicle; S6, combining the street scene neural radiation field model and the vehicle neural radiation field model, using a hybrid ray tracing algorithm to fuse the geometric shapes and intrinsic attributes of the two according to the depth relationship, and using a renderer based on physical principles to synthesize an editable free-viewpoint view of the street scene and the moving vehicle.

2. The method for editable free-viewpoint view synthesis for street scenes and moving vehicles of claim 1, wherein, In step S1, the street scene picture data set includes street pictures in six directions: front, left front, left rear, right front, right rear, and rear.

3. The method for editable free-viewpoint view synthesis for street scenes and moving vehicles of claim 1, wherein, In step S2, according to the camera pose and parameters, a ray tracing algorithm is used to sample the scene to obtain a series of three-dimensional space coordinates in the real world as input of the street scene neural radiation field model.

4. The method for editable free-viewpoint view synthesis for street scenes and moving vehicles of claim 1, wherein, In step S2, in the scene neural radiation field model, a hash coding algorithm is used to design an encoder to encode the input coordinates, a 3-layer multi-layer perception is used as a geometry prediction network to predict the scene voxel density, and another 3-layer multi-layer perception is used to predict the intrinsic attribute parameters of the scene, including base color, roughness, metallic degree, and normal.

5. The method for editable free-viewpoint view synthesis of street scenes and moving vehicles of claim 4, wherein, Based on the scene voxel density, the depth d and the normal n of the scene are calculated by integrating the volume rendering formula.

6. The method of claim 1, wherein, In step S3, the set of spherical Gaussian functions is: where ω i denotes the direction of incident light, φ denotes the axial direction of the spherical Gaussian function, λ denotes the frequency of the spherical Gaussian function, μ denotes the amplitude of the spherical Gaussian function, M is the total number of spherical Gaussian functions, and k is the serial number of the current spherical Gaussian function.

7. The method for synthesizing an editable free-viewpoint view of street scenes and moving vehicles according to claim 6, wherein, In step S3, the function value f of the bidirectional reflectance distribution function of each surface in the scene is calculated according to the inherent attribute parameters of the scene r : f r = f d + f s Among them, the diffuse reflection term k d is the diffuse reflection coefficient of the object, m is the metalness of the object; the specular reflection term f s is converted into a representation based on a spherical Gaussian function: where ω o is the observation direction, h is the half-way vector between the observation direction and the incident light direction ω i , n is the surface normal direction, and a is the surface roughness, is the product of the Fresnel effect and the shadowing effect in the rendering equation.

8. The method for synthesizing an editable free-viewpoint view of street scenes and moving vehicles according to claim 7, wherein, In step S3, the volume rendering formula is used to determine whether the light ray collides with the object in the scene by ray tracing; if a collision occurs, the visibility of the light ray is low, and a shadow is generated at the corresponding position in the image, which is mathematically expressed as follows: wherein N l uniformly sampled on the upper hemisphere, k denotes the index of the incident light ray, and the direction of the incident light ray at the surface point p is l k with visibility V.

9. The method for editable free-viewpoint view synthesis of street scenes and moving vehicles of claim 5, wherein, The L1 loss function between the network-predicted depth and normal and the pre-calculated depth and normal prior is used as supervision, and for the normal, the normal extracted according to the voxel density is added as additional supervision, which is expressed as follows: L depth =∑|d gt -d| wherein, is a loss function for normals, L depth is a loss function for depth; n gt is a normal prior, is a reference normal direction extracted based on voxel density σ, n is a normal output by a multi-layer perceptron network, d gt is a depth prior, d is a scene depth output by a multi-layer perceptron network.

10. The method for editable free-viewpoint view synthesis of street scenes and moving vehicles of claim 7, wherein, The L2 loss between the color rendered based on the physical principle and the real pixel color in the picture will be used, and the loss function is: where, is the loss function for rendering colors; r is the camera ray, is the set of all camera rays, C render is the RGB value computed by the rendering process for each camera ray r, while C gt is the corresponding real color.

Images