Real-time 3D scene modeling system based on physical rendering

Abstract

The invention discloses a real-time 3D scene modeling system based on physical rendering. The real-time 3D scene modeling system comprises a scene data acquisition module, a geometric processing and reconstruction module, a material attribute extraction and processing module, a dynamic illumination acquisition and analysis module, a real-time physical rendering engine module and a rendering output and post-processing module. Real-time scene geometry and material data capture, physical law-based material and illumination model construction and an efficient real-time rendering engine are integrated, an image with a high physical reality sense is output on the premise that the interaction frame rate is kept, the system can automatically or semi-automatically process real world scene information, and the real world scene information processing efficiency is improved. According to the method, the three-dimensional content is converted into a digital model with rich physical attribute description, and an optimized real-time physical rendering algorithm is used for drawing, so that the visualization quality and immersion of real-time application are greatly improved, the dependence on manual adjustment of professional art designers is reduced, and the production efficiency and the automation degree of the three-dimensional content are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional visualization, in particular to a real-time 3D scene modeling system based on physical rendering. BACKGROUND

[0002] In the field of real-time three-dimensional graphics applications such as games, virtual reality, digital twins, etc., the rendering requirements for the realism of the scene are increasingly high. Traditional real-time rendering techniques usually rely on simplified lighting and material models (such as the Phong model), and simulate visual effects through a large amount of pre-computation or artistic adjustment, which is difficult to accurately reflect the complex light interaction physical laws in the real world, and the realism is bottlenecked. On the other hand, physical-based rendering techniques can generate highly realistic images by simulating the physical principles of light propagation, but their computational complexity is extremely high, usually limited to offline rendering, and cannot meet the frame rate requirements of real-time interactive applications. Therefore, how to combine high-fidelity physical rendering effects with real-time interaction performance has become the main challenge currently faced by the technology. SUMMARY

[0003] To this end, the present application provides a real-time 3D scene modeling system based on physical rendering to solve the problem that the real-time rendering techniques in the prior art usually rely on simplified lighting and material models (such as the Phong model), and simulate visual effects through a large amount of pre-computation or artistic adjustment, which is difficult to accurately reflect the complex light interaction physical laws in the real world, and the realism is bottlenecked.

[0004] In order to achieve the above purpose, the present application provides the following technical scheme:

[0005] The real-time 3D scene modeling system based on physical rendering comprises the following modules:

[0006] A scene data acquisition module for real-time acquisition of multi-source data of a target real scene, the multi-source data at least including a color image sequence, a depth image sequence and optional multispectral information;

[0007] A geometry processing and reconstruction module for receiving and processing the multi-source data, generating a dynamic or static triangular mesh model of the target real scene through a real-time three-dimensional reconstruction algorithm, and outputting a geometry data stream containing vertex position and normal information;

[0008] A material attribute extraction and processing module for analyzing and decoupling the visual performance of the surface material of the target real scene based on the color image sequence, the depth image sequence and the geometry data stream, estimating and generating a material parameter set conforming to physical laws, the material parameter set at least including diffuse reflectance, roughness, metallicity and normal map information, and associating and mapping the material parameter set with the triangular mesh in the geometry data stream;

[0009] a dynamic light capture and analysis module for analyzing the ambient light information of the target real scene in real time from the multi-source data or by independent sensors, constructing or updating a high dynamic range ambient map or a resolved light model for physical rendering;

[0010] a real-time physical rendering engine module for receiving the geometry data stream, the associated material parameter set and the ambient light information;

[0011] the real-time physical rendering engine module is integrated with a physically-based shader program, which uses a micro-surface model for light calculation and adopts an image-based lighting technique or an approximate model of real-time ray tracing to simulate global lighting effects, to perform rendering calculation on the triangular mesh model at a real-time frame rate, generating an intermediate rendering image;

[0012] a rendering output and post-processing module for performing tone mapping, anti-aliasing and color correction operations on the intermediate rendering image, generating a final display signal and outputting to a display device.

[0013] Preferably, the geometry processing and reconstruction module comprises a real-time simultaneous localization and mapping unit and a surface meshing optimization unit;

[0014] the real-time simultaneous localization and mapping unit is configured to estimate the pose of the data acquisition device and generate a sparse or dense scene point cloud in real time according to the color image sequence and the depth image sequence;

[0015] the surface meshing optimization unit is configured to perform real-time triangulation processing on the point cloud, generate a preliminary triangular mesh, and perform topological optimization and simplification on the preliminary triangular mesh to reduce rendering load while maintaining geometric features, and output the optimized geometry data stream.

[0016] Preferably, the material attribute extraction and processing module comprises a texture image generation unit and a physical material parameter inversion unit;

[0017] the texture image generation unit is configured to project the color image sequence onto the surface of the triangular mesh model according to the geometry data stream, perform fusion and denoising, and generate an initial diffuse reflection texture map;

[0018] the physical material parameter inversion unit is configured to separate the diffuse reflection and highlight reflection components of the surface based on the initial diffuse reflection texture map, the change information of the color image sequence under different lighting conditions, and a pre-set material physical attribute library, and then solve the material parameter set.

[0019] Preferably, the dynamic light acquisition and analysis module is specifically configured to: periodically or continuously capture panoramic light distribution of the scene by the on-board omnidirectional camera or high dynamic range photometric sensor, generate and update high dynamic range environment map; or inversely deduce the position, intensity and color of the main light source in the scene by analyzing the light and shadow areas in the color image sequence, combined with the geometry data stream, and construct a parameterized analytical lighting model.

[0020] Preferably, the micro-surface model adopted by the physically-based shader program is the Cook-Torrance model, and the specular reflection term is calculated based on the GGX normal distribution function, Smith geometric shading function and Fresnel-Schlick approximation formula.

[0021] Preferably, the real-time physical rendering engine module further comprises a rendering parameter dynamic configuration submodule, which is configured to monitor the rendering performance indicators of the system in real time, and dynamically adjust the rendering quality level, shadow resolution, global light sampling number and post-processing effect switch of the shader program according to the preset performance target and current scene complexity, so as to achieve adaptive balance between visual quality and rendering efficiency.

[0022] Preferably, the scene model database and interactive editing module is further configured to store and manage the reconstructed triangular mesh model and its associated material parameter set, and provide a graphical user interface for users to retrieve, view, edit and modify the material parameters or geometry structure of the stored model, and re-import the edited model data into the real-time physical rendering engine module for rendering preview.

[0023] Preferably, the editing function in the scene model database and interactive editing module includes: providing a visual tool to manually correct inaccurate areas in the material parameter set; supporting dragging and replacing the material of a specific surface from a preset physical material library; allowing users to add, delete or transform model instances in the scene; and supporting aligning and splicing multiple independently reconstructed local scene models to construct a larger unified three-dimensional scene.

[0024] Preferably, the scene data acquisition module is specifically implemented as a movable handheld device or an automatic moving platform integrated with an RGB-D camera, an inertial measurement unit and an optional multi-spectral camera; and the output interface of the rendering output and post-processing module supports a standard video output protocol and can output the final display signal to a head-mounted display, a stereoscopic projection system or a flat panel display.

[0025] The application has the advantages that the application integrates real-time scene geometry and material data capture, material and lighting model construction based on physical laws, and an efficient real-time rendering engine, realizes output of images with high physical realism on the premise of maintaining an interactive frame rate, and the system can automatically or semi-automatically process real-world scene information, convert it into a digital model with rich physical property description, and draw it by using an optimized real-time physical rendering algorithm, which not only greatly improves the visualization quality and immersion of real-time applications, but also reduces the dependence on manual adjustment of professional artists, improves the production efficiency and automation degree of three-dimensional content. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more intuitively illustrate the prior art and the present application, exemplary drawings are given below. It should be understood that the specific shapes, structures shown in the drawings should not be regarded as limiting conditions in the implementation of the present application; for example, based on the technical concepts and exemplary drawings disclosed in the present application, those skilled in the art can easily make routine adjustments or further optimizations to some units (components) in terms of increase / decrease / assignment, specific shape, positional relationship, connection mode, size ratio relationship, etc.

[0027] Figure 1 A module diagram of the real-time 3D scene modeling system based on physical rendering provided for the embodiments of the present application. DETAILED DESCRIPTION

[0028] The embodiments of the present application are described below by specific specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the present specification. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. It should be understood that these embodiments are only for further illustration of the present application, and cannot be understood as a limitation on the protection scope of the present application. The technical engineers in the field can make some non-essential improvements and adjustments to the present application according to the content of the above-mentioned application; based on the embodiments in the present application, all other embodiments obtained by the ordinary skilled in the art without making creative labor are within the protection scope of the present application.

[0029] Please refer to Figure 1 , the real-time 3D scene modeling system based on physical rendering includes the following modules:

[0030] The scene data acquisition module is used for real-time acquisition of multi-source data of a target real scene, and the multi-source data at least includes a color image sequence, a depth image sequence and optional multispectral information;

[0031] a geometry processing and reconstruction module configured to receive and process the multi-source data, generate a dynamic or static triangle mesh model of the target real scene by a real-time 3D reconstruction algorithm, and output a geometry data stream containing vertex position and normal information;

[0032] a material property extraction and processing module configured to analyze and decouple visual representation of surface material of the target real scene based on the color image sequence, the depth image sequence and the geometry data stream, estimate and generate a material parameter set conforming to physical laws, the material parameter set at least including diffuse reflectance, roughness, metalness and normal map information, and associate and map the material parameter set with the triangle mesh in the geometry data stream;

[0033] a dynamic lighting acquisition and analysis module configured to analyze environmental lighting information of the target real scene from the multi-source data in real time or acquire the environmental lighting information by an independent sensor, and construct or update a high dynamic range environment map or an analytical lighting model for physical rendering;

[0034] a real-time physical rendering engine module configured to receive the geometry data stream, the associated material parameter set and the environmental lighting information;

[0035] the real-time physical rendering engine module is integrated with a physical-based shader program, the shader program uses a micro-surface model for lighting calculation, and adopts an image-based lighting technique or an approximate model of real-time ray tracing to simulate global lighting effects, so as to perform rendering calculation on the triangle mesh model at a real-time frame rate, and generate an intermediate rendering image;

[0036] a rendering output and post-processing module configured to perform tone mapping, anti-aliasing and color correction operations on the intermediate rendering image, generate a final display signal and output the final display signal to a display device.

[0037] The scene data acquisition module synchronously collects color, depth and multispectral data streams of the scene. This multi-source and synchronous data acquisition strategy ensures that the subsequent processing has complete and consistent information sources, lays a reliable data foundation for building a high-fidelity digital model, and avoids model distortion caused by data loss or asynchronization. The collected data is immediately sent to the geometric processing and reconstruction module, which converts discrete point cloud data into a triangular mesh model with accurate spatial relationships in real time through real-time SLAM and other technologies. This step generates an accurate geometric framework of the scene, enabling the virtual scene to strictly correspond to the scale and structure of the real space, which is a physical prerequisite for achieving high-immersion visualization. At the same time, the material property extraction and processing module cooperates with the geometric data and color images to calculate the physical parameters (diffuse reflectivity, roughness, etc.) of each material (such as brick wall, glass, metal) on the grid surface through physical inversion algorithm. This process automatically gives the model realistic surface visual properties, transforming the traditional manual mapping work of art personnel into a computable process, greatly improving the production efficiency and physical accuracy of realistic materials.

[0038] After building a model with accurate geometry and physical materials, the dynamic lighting acquisition and analysis module captures or estimates the environmental lighting conditions of the scene in real time and converts them into high-dynamic-range environment maps or light source parameters that can be used by the rendering engine. This enables the lighting of virtual objects to dynamically match the real environmental lighting, ensuring that virtual objects blend into real scenes in terms of light and shadow and color reflection. All the above elements (geometry, material, lighting) are gathered into the core real-time physical rendering engine module, which uses physically-based shaders (such as the Cook-Torrance model) and global illumination techniques to perform real-time ray interaction simulation calculations on the input model, calculating images close to offline rendering quality at interactive frame rates, successfully balancing rendering quality and real-time performance. Finally, the rendering output and post-processing module performs tone mapping and other processing on the rendered image to adapt it to the dynamic range of the display device and optimize visual details, ultimately outputting a picture with outstanding visual realism, directly meeting the immediate display needs of high-end visualization, VR / AR and other applications.

[0039] The geometric processing and reconstruction module includes a real-time simultaneous localization and mapping unit and a surface meshing optimization unit;

[0040] The real-time simultaneous localization and mapping unit is configured to estimate the pose of the data acquisition device and generate a sparse or dense scene point cloud in real time according to the color image sequence and the depth image sequence;

[0041] The surface meshing optimization unit is used for real-time triangulation processing on the point cloud, generating a preliminary triangular mesh, and topological optimization and simplification on the preliminary triangular mesh to reduce rendering load while maintaining geometric features, and outputting the optimized geometric data stream.

[0042] The real-time simultaneous localization and mapping unit can estimate the motion trajectory of the camera itself (localization) and gradually build a three-dimensional point cloud map of the scene (mapping) by processing continuous images and depth frames. For example, when a handheld scanning device is roaming in a room, this unit can track the device's position in real time and continuously expand the point cloud model of the room. The surface meshing optimization unit is responsible for converting discrete and possibly noisy point clouds into clear and renderable triangular mesh surfaces. It performs smoothing, denoising, and simplification operations on the generated preliminary mesh, while preserving key geometric features such as corners and window edges, and minimizing the number of triangles as much as possible, thereby directly reducing the rendering burden of the GPU and ensuring the smoothness of subsequent real-time rendering.

[0043] The material property extraction and processing module includes a texture image generation unit and a physical material parameter inversion unit.

[0044] The texture image generation unit is used to project the color image sequence onto the surface of the triangular mesh model based on the geometric data stream, perform fusion and denoising, and generate an initial diffuse reflection texture map.

[0045] The physical material parameter inversion unit is used to separate the diffuse reflection and highlight reflection components of the surface based on the initial diffuse reflection texture map, the color image sequence under different lighting conditions, and the pre-set material physical property library, and then solve for the material parameter set.

[0046] The texture image generation unit "wraps" the color images taken at different angles onto the three-dimensional mesh surface based on the known camera pose and geometric model, and fuses to generate a seamless, consistent resolution initial color texture map. The physical material parameter inversion unit then performs deeper physical analysis, for example, it analyzes the changes in brightness and highlights of the same surface under different lighting angles (such as moving the light source during scanning or observing from different angles), combines with the pre-set physical reflection model library, and calculates the roughness, metal degree, and micro concave-convex details (normal map) of the surface in addition to the basic color.

[0047] The dynamic light acquisition and analysis module is specifically configured to: periodically or continuously capture panoramic light distribution of a scene by using an omnidirectional camera or a high dynamic range light sensor, generate and update a high dynamic range environment map; or inversely deduce positions, intensities and colors of main light sources in the scene by analyzing light and shadow areas in the color image sequence in combination with the geometric data stream, and construct a parameterized analytical light model.

[0048] The technical solution provides two implementation manners of the dynamic light acquisition and analysis module: the first is a direct capture method, which directly obtains a panoramic environment map recording colors and intensities of light in all directions by using a professional omnidirectional HDR camera or a photometer to quickly capture a scene, the light information obtained by this method is very accurate, and the method is suitable for static or semi-static scenes with extremely high requirements for light reality; the second is an analysis and deduction method, which is suitable for scenes without special acquisition devices or scenes with changing light, the module analyzes shapes of shadows, positions of highlights and brightness in ordinary RGB images in combination with a known geometric model, and inversely calculates directions, colors and intensities of main light sources by using a computer vision algorithm, so as to construct a simplified parameterized light model.

[0049] The micro-surface model adopted by the physical-based shader program is a Cook-Torrance model, a specular reflection term of the Cook-Torrance model is calculated based on a GGX normal distribution function, a Smith geometric shading function and a Fresnel-Schlick approximation formula; and the global light effect is realized by pre-computed radiosity transmission, screen space ambient occlusion and reflection technology, or by using a hybrid pipeline of real-time ray tracing with simplified ray stepping and noise reduction.

[0050] The real-time physical rendering engine module further includes a rendering parameter dynamic configuration submodule, the rendering parameter dynamic configuration submodule is configured to monitor rendering performance indexes of a system in real time, and dynamically adjust a rendering quality level of the shader program, a shadow resolution, a global light sampling number and a post-processing effect switch according to a preset performance target and a current scene complexity, so as to realize adaptive balance between visual quality and rendering efficiency.

[0051] In practical applications, the scene complexity (such as the number of objects, material details) and the performance of the running hardware (such as high-end PC and mobile VR all-in-one machine) differ greatly. The rendering parameter dynamic configuration submodule monitors performance indicators such as the frame rate per second in real time. For example, when the frame rate is detected to drop below a set threshold, it will automatically reduce the sampling number of global illumination, lower the resolution of shadow maps, or temporarily turn off some expensive post-processing effects; on the contrary, when the performance is sufficient, it will automatically improve these settings to obtain the best image quality. This dynamic adaptability ensures that the system can provide the best visual experience on different hardware platforms while maintaining smooth operation, greatly enhancing the practicality and universality of the system.

[0052] The scene model database and interactive editing module is further included for storing and managing the reconstructed triangular mesh model and its associated material parameter set, and providing a graphical user interface for users to retrieve, view, edit and modify the material parameters or geometry of the stored model, and re-import the edited model data into the real-time physical rendering engine module for rendering preview.

[0053] The scene model database and interactive editing module stores and manages the reconstructed three-dimensional model and its complete set of material maps in a structured manner, and users can conveniently retrieve historical scan models through a graphical interface. Moreover, users can also correct imperfections that may be generated by the automated process, such as when the automatically extracted metal material reflectivity is insufficient, users can directly select the area on the 3D view, adjust the "metallicity" and "roughness" parameters, and immediately see the rendering preview update.

[0054] The editing functions in the scene model database and interactive editing module include: providing visual tools to manually correct inaccurate areas in the material parameter set; supporting dragging and replacing the material of a specific surface from a pre-set physical material library; allowing users to add, delete or transform model instances in the scene; and supporting aligning and splicing multiple independently reconstructed local scene models to construct a larger unified three-dimensional scene.

[0055] The scene data acquisition module is specifically implemented as a movable handheld device or an automatic moving platform integrating an RGB-D camera, an inertial measurement unit, and an optional multispectral camera; the output interface of the rendering output and post-processing module supports standard video output protocols and can output the final display signal to a head-mounted display, a stereoscopic projection system, or a flat panel display.

[0056] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A real-time 3D scene modeling system based on physically based rendering, characterized in that, Includes the following modules: The scene data acquisition module is used to collect multi-source data of the target real-world scene in real time. The multi-source data includes at least color image sequences, depth image sequences, and optional multispectral information. The geometry processing and reconstruction module is used to receive and process the multi-source data, generate a dynamic or static triangular mesh model of the target real scene through a real-time 3D reconstruction algorithm, and output a geometric data stream containing vertex position and normal information. The material attribute extraction and processing module is used to analyze and decouple the visual performance of the surface material of the target real scene based on the color image sequence, depth image sequence and geometric data stream, estimate and generate a material parameter set that conforms to physical laws, the material parameter set includes at least diffuse reflectance, roughness, metallicity and normal map information, and associate and map the material parameter set with the triangular mesh in the geometric data stream; The dynamic lighting acquisition and analysis module is used to analyze the ambient lighting information of the target real scene in real time from the multi-source data or to acquire it through independent sensors, and to construct or update a high dynamic range environment map or analytical lighting model for physical rendering. A real-time physical rendering engine module is used to receive the geometric data stream, the associated material parameter set, and the ambient lighting information; The real-time physical rendering engine module integrates a physically based shader program. The shader program uses a micro-surface model to perform lighting calculations and employs image-based lighting techniques or an approximate model of real-time ray tracing to simulate global lighting effects. It performs rendering calculations on the triangular mesh model at a real-time frame rate to generate intermediate rendering images. The rendering output and post-processing module is used to perform tone mapping, anti-aliasing and color correction operations on the intermediate rendered image, generate the final display signal and output it to the display device.

2. The real-time 3D scene modeling system based on physically based rendering according to claim 1, characterized in that, The geometry processing and reconstruction module includes a real-time synchronous positioning and map building unit and a surface meshing optimization unit; The real-time synchronous positioning and map building unit is used to estimate the pose of the data acquisition device in real time and generate sparse or dense scene point clouds based on the color image sequence and depth image sequence. The surface meshing optimization unit is used to perform real-time triangulation processing on the point cloud, generate a preliminary triangular mesh, and perform topology optimization and simplification on the preliminary triangular mesh to reduce the rendering load while maintaining geometric features, and output the optimized geometric data stream.

3. The real-time 3D scene modeling system based on physically based rendering according to claim 2, characterized in that, The material property extraction and processing module includes a texture image generation unit and a physical material parameter inversion unit; The texture image generation unit is used to backproject the color image sequence onto the surface of the triangular mesh model according to the geometric data stream, perform fusion and denoising, and generate an initial diffuse texture map; The physical material parameter inversion unit is used to separate the diffuse and specular reflection components of the surface based on the initial diffuse texture map, the variation information of the color image sequence under different lighting conditions, and the preset material physical property library, and then solve for the material parameter set.

4. The real-time 3D scene modeling system based on physically based rendering according to claim 1, characterized in that, The dynamic lighting acquisition and analysis module is specifically used to: periodically or continuously capture the panoramic lighting distribution of the scene using an omnidirectional camera or a high dynamic range photometric sensor, and generate and update a high dynamic range environment map; or analyze the bright and shadow areas in the color image sequence, and in combination with the geometric data stream, reversely deduce the position, intensity and color of the main light source in the scene, and construct a parameterized analytical lighting model.

5. The real-time 3D scene modeling system based on physically based rendering according to claim 4, characterized in that, The physical shader program uses the Cook-Torrance model as its microsurface model, and its specular reflection term is calculated based on the GGX normal distribution function, the Smith geometric occlusion function, and the Fresnel-Schlick approximation formula.

6. The real-time 3D scene modeling system based on physically based rendering according to claim 5, characterized in that, The real-time physical rendering engine module also includes a dynamic rendering parameter configuration submodule. This submodule is used to monitor the rendering performance indicators of the system in real time, and dynamically adjust the rendering quality level, shadow resolution, global illumination sampling number, and post-processing effect on / off of the shader program according to the preset performance target and the current scene complexity, so as to achieve an adaptive balance between visual quality and rendering efficiency.

7. The real-time 3D scene modeling system based on physically based rendering according to claim 1, characterized in that, It also includes a scene model database and an interactive editing module for storing and managing the reconstructed triangular mesh model and its associated material parameter set, and provides a graphical user interface for users to retrieve, view, edit and modify the material parameters or geometric structure of the stored model, as well as re-import the edited model data into the real-time physical rendering engine module for rendering preview.

8. The real-time 3D scene modeling system based on physically based rendering according to claim 7, characterized in that, The editing functions in the scene model database and interactive editing module include: providing visual tools to manually correct inaccurate areas in the material parameter set; supporting drag-and-drop replacement of materials for specific surfaces from a preset physical material library; allowing users to add, delete, or transform model instances in the scene; and supporting the alignment and stitching of multiple independently reconstructed local scene models to construct a larger-scale unified 3D scene.

9. The real-time 3D scene modeling system based on physically based rendering according to claim 1, characterized in that, The scene data acquisition module is specifically implemented as a mobile handheld device or automatic mobile platform that integrates an RGB-D camera, an inertial measurement unit, and an optional multispectral camera; the output interface of the rendering output and post-processing module supports standard video output protocols and can output the final display signal to a head-mounted display, a stereoscopic projection system, or a flat panel display.

Images