Method for synthesizing editable free viewpoint views for street views and moving vehicles

Through the neural radiation field-based method, the intrinsic properties of street scenes and vehicles are decoupled, combined with physical rendering and spherical Gaussian function, the synthesis of high-reality street scenes under different lighting conditions is achieved, and the lighting changes in the existing technology is solved The problem of scenario consistency is suitable for multi-field applications.

CN118982584AActive Publication Date: 2024-11-19NANJING UNIV

Patent Information

Application Number
CN202410905867.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-19
Estimated Expiration
2044-07-08

AI Technical Summary

Technical Problem

The prior art is difficult to synthesize street scenes with a sense of reality under different lighting conditions, especially in the real world with complex environments. The consistency of multi-view picture collection is difficult to ensure, resulting in the inability to generate real streets under different time periods and lighting conditions. Scene.

Method used

Using a neural radiation field-based method, by obtaining the picture data set of street scenes and vehicles, calculating camera poses and parameters, using normal and depth prediction networks to establish a neural radiation field model, combining physical rendering methods and spherical Gaussian functions to characterize illumination. The model is trained to decouple the intrinsic properties of the scene and the vehicle, and to synthesize an editable free viewpoint view through a hybrid ray tracing algorithm.

Benefits of technology

Realizes high-reality free viewpoint rendering and scene editing, and can generate photo-level street scenes under multiple angles and multiple lighting conditions, making up for the research gap in existing technology, and is suitable for autonomous driving, film and television production and game development and other fields.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982584A_ABST
    Figure CN118982584A_ABST
Patent Text Reader

Abstract

The invention discloses an editable free viewpoint view synthesis method for streetscape and moving vehicles. The editable free viewpoint view synthesis method specifically comprises the following steps: expressing a street scene and a virtual vehicle model in the form of a neural radiation field; estimating inherent attributes and scene illumination of the street scene and the vehicle model by combining a physical rendering-based method; scene illumination is expressed as a series of spherical Gaussian functions, the scene in the real world is transferred to a virtual vehicle model, and the scene of a virtual vehicle running in a real street with a strong sense of reality is rendered in combination with internal attributes; the reality sense of the vehicle rendering effect is further improved through the generation network based on deep mixed sampling; and rendering the effects of the same scene under different illumination conditions based on the internal attributes of the scene and the vehicle. According to the invention, neural radiation fields are respectively established for the vehicle and the scene, the vehicle is harmoniously inserted into the street scene in the real world, and large-range, multi-angle and variable-illumination free viewpoint rendering and scene editing are realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an editable free viewpoint view synthesis method for street scenes and moving vehicles based on neural radiation fields. Background Art

[0002] Humans can imagine unseen scenarios in their minds, which greatly enhances the effectiveness of learning. Drawing inspiration from this ability, researchers in the field of autonomous driving focus on creating virtual environments containing synthetic vehicles to train perception models. Extensive research has shown that this approach is essential to solving challenges associated with 3D visual decision-making problems.

[0003] Traditional 3D simulation involves the use of computer-aided design (CAD) models. However, using only CAD models to generate extensive and diverse street scenes has proven to be very inefficient and costly in practice. Currently, some scholars have used 2D image-based generation methods to synthesize editable street scene images (Wei Li, CW Pan, Rong Zhang, JPRen, YX Ma, Jin Fang, FL Yan, QC Geng, XY Huang, HJ Gong, et al. Aads: Augmented autonomous driving simulation using data-driven algorithms. Science Robotics, 4(28): eaaw0863, 2019), using real-life images or depth maps as prior knowledge of the scene. These methods still have major problems in synthesizing rotations related to 3D information, shadow occlusion relationships, and other aspects. Recently, the introduction of neural radiance fields (Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020) has further improved the synthesis quality of 3D information, so that street scene images with higher quality can be rendered from a free viewpoint.

[0004] The above research is dedicated to synthesizing street scene images with photo-realistic quality. However, the training of the neural radiance field model requires collecting multi-view images of the scene. Due to the complexity of the real-world environment, street scenes often experience changes in lighting, changes in dynamic objects, changes in the appearance of the environment, etc. This makes it difficult to ensure consistency when collecting multi-view images, and it is impossible to synthesize realistic street scenes under different time periods and lighting conditions. Therefore, how to decouple the intrinsic properties of the environment to match different lighting conditions remains an urgent problem to be solved. Summary of the invention

[0005] In order to synthesize street scenes with photorealistic feeling, the present invention provides an editable free viewpoint view synthesis method for street scenes and moving vehicles based on neural radiance fields.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the method of the present invention is as follows:

[0007] A method for synthesizing editable free viewpoint views of street scenes and moving vehicles, comprising the following steps:

[0008] S1, obtain a street scene image dataset, calculate the camera pose and camera parameters when taking street scene images; use the normal and depth prediction network to obtain the normal and depth prior conditions based on image prediction;

[0009] S2, based on the street scene image dataset, using a neural radiation field algorithm to establish a street scene neural radiation field model for the street scene, and using a multi-layer perceptron for the street scene to predict the intrinsic attribute parameters of the scene;

[0010] S3, constructing street view neural radiation field model data according to the pictures in the street scene picture dataset in step S1 and the normal and depth priors predicted based on the pictures; combining the scene illumination and the intrinsic attribute parameters by using a rendering method based on physical principles to render a street view picture, wherein a set of spherical Gaussian functions are used to characterize the scene illumination; and training the street view neural radiation field model by using a priori L1 loss function and image L2 loss function;

[0011] S4, collect a large number of 3D models of vehicles, build a 3D virtual vehicle model data set, and render multi-view images of vehicle models; use the neural radiation field algorithm to establish a vehicle neural radiation field model for the vehicle; use the multi-layer perceptron for the vehicle to predict the intrinsic attribute parameters of the vehicle;

[0012] S5, training the vehicle neural radiation field model according to the multi-view images of the vehicle model rendered in step S4 to achieve three-dimensional reconstruction of the vehicle;

[0013] S6, combining the street scene neural radiation field model and the vehicle neural radiation field model, using a hybrid ray tracing algorithm to fuse the geometric shapes and intrinsic properties of the two according to the depth relationship, using a physically based renderer to synthesize an editable free viewpoint view of the street scene and moving vehicles.

[0014] The present invention constructs a three-dimensional virtual vehicle model data set, which includes vehicle models of various types, colors, and ages. Based on vehicle and scene data, it realizes the decoupling of scene and vehicle intrinsic properties, harmoniously inserts vehicles into real-world street scenes, and further realizes high-realistic free fusion, which can realize large-scale, multi-angle, and variable-lighting free viewpoint rendering and scene editing. The present invention realizes the synthesis of editable free viewpoint views of street scenes and moving vehicles based on neural radiation fields for the first time, filling the research gap in this area. The proposed method can be widely used in fields such as autonomous driving, film and television production, and game development, and has high practical value and development prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Flow chart of the method of the present invention.

[0016] Figure 2 Schematic diagram of a vehicle three-dimensional model data set in an embodiment of the present invention.

[0017] Figure 3 Schematic diagram of mixed sampling in an embodiment of the present invention.

[0018] Figure 4 Flow chart of the operation phase in an embodiment of the present invention.

[0019] Figure 5 The following are the result display diagrams in the embodiments of the present invention: (a) the intrinsic attribute decomposition result, and (b) the fusion result of different vehicle models and different scenes. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] In response to the defects existing in the prior art, the present invention proposes a hybrid solution, which uses real captured sparse videos and synthetic 3D car composites to simulate street scenes, so as to synthesize realistic free-viewpoint rendering in a variety of scenarios. First, both the street scene and the virtual vehicle model are represented in the form of a neural radiation field; then, the intrinsic properties and scene lighting of the street scene and vehicle model are estimated in combination with the physically based rendering method; the scene lighting is represented as a series of spherical Gaussian functions, which are transferred from the real-world scene to the virtual vehicle model, and the intrinsic properties are combined to render a scene of a virtual vehicle driving on a real street with a strong sense of reality; then, a generative network based on deep hybrid sampling is used to further enhance the realism of the vehicle rendering effect; finally, the effect of the same scene under different lighting conditions is rendered based on the intrinsic properties of the scene and the vehicle. Figure 1 As shown, the present invention is an editable free viewpoint view synthesis method for street scenes and moving vehicles, and the specific process is as follows:

[0022] 1. There are currently a large number of real-world captured street view datasets, which contain extremely diverse street scenes under real-world lighting information. Based on the publicly available large-scale street scene dataset (Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.), obtain street scene images and calculate the camera pose and camera parameters when taking street scene images based on the GPS / IMU information in the dataset. Using the normal and depth prediction network ( Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3D common corruptions and data augmentation. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pages 18963–18974, 2022.) obtains normal and depth prior conditions based on image prediction. Among them, the street scene image dataset contains street view images in six directions: front, left front, left rear, right front, right rear, and rear.

[0023] 2. Based on the above street scene dataset, a street scene neural radiation field model is established for street scenes using the neural radiation field algorithm. First, the pixel coordinates of the image are converted to the camera's three-dimensional space coordinate system according to the camera's intrinsic parameters, and then the camera's three-dimensional space coordinates are converted to the three-dimensional space coordinates of the real world through the camera's posture parameters. The three-dimensional space coordinates of the real world are used as the input of the neural radiation field model. The color of the corresponding pixel is the color of the true value ground-truth, and the normal and depth predicted by the corresponding pixel are the normal and depth of the true value ground-truth.

[0024] 3. Realize three-dimensional reconstruction of scene geometry. First, select a scene from the data set, and use the multi-layer perceptron network for street scenes to predict the scene's basic color, roughness, metalness, normal, depth and other intrinsic attribute parameters. The input three-dimensional space coordinates are first passed through a hash encoder to obtain the hash features corresponding to each space coordinate. This embodiment designs a 3-layer geometric feature extraction multi-layer perceptron network, and uses the hash features output by the hash encoder as the input of the network. After passing through the multi-layer perceptron, the output is the voxel density σ of each space coordinate, and the intermediate vector b. Then, this embodiment designs an intrinsic feature extraction network, which is composed of a normal extraction network composed of a 3-layer multi-layer perceptron and an attribute extraction network composed of a 3-layer multi-layer perceptron. The intermediate vector b is used as input, and is input into the normal extraction network and the attribute extraction network respectively to obtain the predicted normal n, roughness α, metalness m, and basic color c.

[0025] 4. Use a set of M spherical Gaussian functions to characterize the scene illumination. This set of functions can be expressed as:

[0026]

[0027] Among them, k is the number of the current spherical Gaussian function, ω irepresents the direction of the incident light, i represents the incident in, φ represents the axial direction of the spherical Gaussian function, λ represents the frequency of the spherical Gaussian function, and μ represents the amplitude of the spherical Gaussian function.

[0028] 5. Physically-based shading method based on rendering equation and simplified Disney principle (Brent Burley and Walt Disney Animation Studios. (2012). Physically-based shading at Disney. InSIGGRAPH, pages 1-7, vol. 2012.), using the rendering equation based on spherical Gaussian function, combined with the lighting conditions in the form of spherical Gaussian, the final rendering color is obtained. The BRDF function in the existing rendering equation is expressed as two parts: diffuse reflection term and specular reflection term: f r =f d +f s , where the diffuse term is mainly related to the base color and metalness: Among them, k d is the diffuse reflection coefficient of the object, and m is the metalness of the object. The specular reflection term is converted into a representation based on a spherical Gaussian function:

[0029]

[0030] Among them, ω o is the observation direction, h is the observation direction and the incident light direction ω i The half-distance vector between n and α is the surface normal direction, is the product of the Fresnel effect and the shadow effect in the rendering equation. As an example, this embodiment uses a fast integral calculation method of a spherical Gaussian function (Wang, J., Ren, P., Gong, M., Snyder, J., & Guo, B. (2009). All-frequency rendering of dynamic, spatially-varying reflectance. In SIGGRAPH Asia, pages 1-10.), avoiding complex calculations and directly obtaining the result of the rendering equation.

[0031] 6. Due to the complexity of the street scene, the existence of various objects in the scene blocks the light, resulting in shadows in the entire scene. Through the ray tracing method, the volume rendering formula is used to determine whether the light collides with the objects in the scene; if a collision occurs, the visibility of the light is low, and a shadow is generated at the corresponding position in the image. This embodiment uniformly samples N in the upper hemisphere. l The shadow map is calculated by the incident light at the surface point p (direction lk ) can be expressed as the volume opacity of the light: r(t) = p + tl k , t is the distance from a point on the ray to p. This ray is then traced in the scene, and volume rendering is used to obtain visibility. This method takes into account the complex interaction between light and objects in urban street scenes and accurately represents shadows. The mathematical representation is as follows:

[0032]

[0033] Among them, k represents the sequence number of the incident light. Finally, the shadow map is added to the scene by pixel-by-pixel multiplication.

[0034] 7. When calculating the rendering loss function in step 5, the L2 loss function between the predicted color and the true color in the input image is used as supervision, and its formula is expressed as:

[0035]

[0036] in, is the loss function of rendering color; r is the camera light, is the set of all camera rays. C render is the light passing through each camera The RGB value calculated by the rendering process, and C gt is the corresponding true color.

[0037] 8. When calculating the loss function of the intrinsic attribute in step 3, the L1 loss function of the predicted intrinsic attribute and the ground-truth intrinsic attribute is used as supervision. In addition, for the normal, the normal extracted according to the voxel density is added as additional supervision, and the formula is expressed as:

[0038]

[0039] L depth =Σ|d gt -d|

[0040] in, is the loss function of the normal, L depth is the depth loss function, is the loss function of the scene's basic color; n gt is the normal prior, is the reference normal direction extracted based on the voxel density σ, n is the normal output by the multilayer perceptron network, d gt is the depth prior, and d is the scene depth output by the multi-layer perceptron network.

[0041] An end-to-end training scheme is implemented by optimizing the multilayer perceptron network for computing intrinsic properties in step 3 and the spherical Gaussian function for representing illumination in step 4.

[0042] like Figure 2 As shown, the present invention collects a large number of 3D vehicle models to establish a virtual vehicle library. In order to seamlessly integrate these 3D models into the street view, a vehicle neural radiation field is constructed to decompose the intrinsic properties of the vehicle. This neural radiation field takes the 3D position of the vehicle in the canonical space as input, and obtains the corresponding spatial features through a three-plane encoder with a resolution of 512×512×512. Subsequently, it obtains density through a 1-layer perceptron with a hidden layer dimension of 64, and obtains intrinsic properties through a 2-layer perceptron with a hidden layer dimension of 128. These intrinsic properties include basic color, material, and normal, which can be directly input into the street view renderer. This neural radiation field is initially trained using vehicle images and then queried in the fusion stage.

[0043] like Figure 3 As shown, the present invention uses a hybrid sampling method to integrate vehicles and street scenes. In addition to the original sampling points in the scene, when the sampling light intersects with the three-dimensional bounding box of the vehicle in the scene, a set of additional sampling points are introduced in the three-dimensional bounding box of the vehicle. These additional points are specifically used to sample the vehicle and calculate the intrinsic properties of the vehicle. After sampling both the vehicle and the scene, the sampling points are sorted and combined according to their depths in real-world coordinates to obtain combined sampling points {X}. Then, let r=o+td represent the camera ray with starting point o and direction d, and perform standard volume rendering to obtain the color and intrinsic properties of the fusion of the vehicle and the scene, presenting a correct appearance that conforms to the depth occlusion relationship.

[0044] like Figure 4As shown, this embodiment first divides the scene and the vehicle into two branches for processing. For the scene branch, the input three-dimensional coordinates are obtained according to the camera parameters, and after hash coding, they are input into the multi-layer perceptron network of the computational geometry to obtain the voxel density and the intermediate vector, and then the intermediate vector is input into the multi-layer perceptron network of the computational intrinsic attribute to obtain the intrinsic attribute of the voxel, and the voxel density and the intrinsic attribute are used to obtain the final result by the volume rendering method. For the vehicle branch, a similar strategy is adopted, and the input three-dimensional coordinates are input into the multi-layer perceptron network of the computational geometry by three-plane coding, and finally the intrinsic attribute of the vehicle is obtained by combining the volume rendering. In order to insert the vehicle harmoniously into the scene, according to the illumination information expressed in the form of a spherical Gaussian function extracted from the scene, the illumination is applied to the scene and the vehicle at the same time, and a physically based rendering method is used to obtain a vehicle insertion effect that conforms to the real illumination effect. Combined with the hybrid sampling method, the insertion position and direction of the vehicle can be freely edited, and the occlusion relationship is correctly handled to obtain the final fusion result. Since this embodiment decouples all the intrinsic attributes of the scene and the vehicle, the illumination of the scene can also be flexibly modified and edited, which greatly improves the diversity of the synthetic scene.

[0045] The final fusion result is Figure 5 As shown in the figure, the vehicle is inserted into the scene and its position can be moved freely, while presenting a harmonious effect with the scene environment.

[0046] This invention realizes for the first time a method for synthesizing street scenes and moving vehicles with editable free viewpoints based on neural radiation fields, filling the research gap in this area. The proposed method can be widely used in the fields of autonomous driving, film and television production, game development, etc., and has high practical value and development prospects.

Claims

1. A method for synthesizing editable free viewpoint views of street scenes and moving vehicles, characterized in that: The method comprises the following steps: S1, obtain a street scene image dataset, calculate the camera pose and camera parameters when taking street scene images; use the normal and depth prediction network to obtain the normal and depth prior conditions based on image prediction; S2, based on the street scene image dataset, using a neural radiation field algorithm to establish a street scene neural radiation field model for the street scene, and using a multi-layer perceptron for the street scene to predict the intrinsic attribute parameters of the scene; S3, constructing street view neural radiation field model data according to the pictures in the street scene picture dataset in step S1 and the normal and depth priors predicted based on the pictures; combining the scene illumination and the intrinsic attribute parameters by using a rendering method based on physical principles to render a street view picture, wherein a set of spherical Gaussian functions are used to characterize the scene illumination; and training the street view neural radiation field model by using a priori L1 loss function and image L2 loss function; S4, collect a large number of 3D models of vehicles, build a 3D virtual vehicle model data set, and render multi-view images of vehicle models; use the neural radiation field algorithm to establish a vehicle neural radiation field model for the vehicle; use the multi-layer perceptron for the vehicle to predict the intrinsic attribute parameters of the vehicle; S5, training the vehicle neural radiation field model according to the multi-view images of the vehicle model rendered in step S4 to achieve three-dimensional reconstruction of the vehicle; S6, combining the street scene neural radiation field model and the vehicle neural radiation field model, using a hybrid ray tracing algorithm to fuse the geometric shapes and intrinsic properties of the two according to the depth relationship, using a physically based renderer to synthesize an editable free viewpoint view of the street scene and moving vehicles.

2. The method for synthesizing editable free viewpoint views of street scenes and moving vehicles according to claim 1, characterized in that: In step S1, the street scene image data set includes street scene images in six directions: front, left front, left rear, right front, right rear, and rear.

3. The method for synthesizing editable free viewpoint views for street scenes and moving vehicles according to claim 1, characterized in that: In step S2, the scene is sampled using a ray tracing algorithm according to the camera pose and parameters to obtain a series of three-dimensional spatial coordinates in the real world as input to the street view neural radiation field model.

4. The method for synthesizing editable free viewpoint views for street scenes and moving vehicles according to claim 1, characterized in that: In step S2, in the scene neural radiation field model, an encoder is designed using a hash coding algorithm to encode the input coordinates, a 3-layer multilayer perceptron is used as a geometric prediction network to predict the scene voxel density, and then another 3-layer multilayer perceptron is used to predict the intrinsic attribute parameters of the scene, and the intrinsic attribute parameters include: basic color, roughness, metalness and normal.

5. The method for synthesizing editable free viewpoint views for street scenes and moving vehicles according to claim 4, characterized in that: Based on the voxel density of the scene, integration is performed according to the volume rendering formula to calculate the depth d and normal n of the scene.

6. The method for synthesizing editable free viewpoint views for street scenes and moving vehicles according to claim 1, characterized in that: In step S3, the set of spherical Gaussian functions is: where ω i represents the direction of the incident light, φ represents the axial direction of the spherical Gaussian function, λ represents the frequency of the spherical Gaussian function, μ represents the amplitude of the spherical Gaussian function, M is the total number of spherical Gaussian functions, and k is the serial number of the current spherical Gaussian function.

7. The method for synthesizing editable free viewpoint views for street scenes and moving vehicles according to claim 6, characterized in that: In step S3, the function value f of the bidirectional reflectance distribution function of each surface in the scene is calculated according to the intrinsic attribute parameters of the scene. r : f r =f d +f s Among them, the diffuse reflection term k d is the diffuse reflection coefficient of the object, m is the metalness of the object; the specular reflection term f s is converted into a representation based on a spherical Gaussian function: Among them, ω o is the observation direction, h is the observation direction and the incident light direction ω i The half-distance vector between n and α is the surface normal direction, It is the product of the Fresnel effect and the shadow effect in the rendering equation.

8. The method for synthesizing editable free viewpoint views for street scenes and moving vehicles according to claim 7, characterized in that: In step S3, a volume rendering formula is used to determine whether the light ray collides with an object in the scene by using a ray tracing method; if a collision occurs, the visibility of the light ray is low, and a shadow is generated at the corresponding position in the image, which is mathematically expressed as follows: Among them, N is sampled uniformly in the upper hemisphere l There are incident rays, k represents the number of the incident rays, and the direction of the incident ray at the surface point p is l k , whose visibility is V.

9. The method for synthesizing editable free viewpoint views for street scenes and moving vehicles according to claim 5, characterized in that: The L1 loss function is calculated between the depth and normal predicted by the network and the pre-calculated depth and normal priors as supervision. In addition, for the normal, the normal extracted according to the voxel density is added as additional supervision, and the formula is expressed as: L depth =∑|d gt -d| in, is the loss function of the normal, L depth is the depth loss function; n gt is the normal prior, is the reference normal direction extracted based on the voxel density σ, n is the normal output by the multilayer perceptron network, d gt is the depth prior, and d is the scene depth output by the multi-layer perceptron network.

10. The method for synthesizing editable free viewpoint views for street scenes and moving vehicles according to claim 7, characterized in that: The L2 loss is calculated between the physically rendered color and the real pixel color in the image. The loss function is: in, is the loss function of rendering color; r is the camera light, is the set of all camera rays, C render is the light passing through each camera The RGB value calculated by the rendering process, and C gt is the corresponding true color.

Citation Information

Patent Citations

  • Explicit scene combination method based on neural radiation field

    CN116363299A

  • Vehicle digital twinning method suitable for automatic driving scene

    CN116682082A

  • Geometric splicing-based wide baseline image new viewpoint synthesis method

    CN116993945A

  • New view synthesis method and system based on real-time rendering generalizable neural radiation field

    CN117635801A

  • Human body weight illumination method based on dynamic surface reflection field

    CN117953137A

Cited By

  • Dynamic streetscape reconstruction method and device under two-dimensional semantic priori, equipment and medium

    CN122199739A