Virtual reality building walkthrough method and system based on scene rendering

Abstract

The application provides a virtual reality building roaming method and system based on scene rendering, relates to the technical field of virtual reality, and comprises the following steps: dividing building three-dimensional model data by means of an octree space segmentation algorithm, and determining a detail level threshold according to a viewpoint distance to construct a multi-level detail model. A basic light map is baked by means of a pre-computed irradiance transfer technology, and is combined with a multi-resolution shadow map to perform real-time synthesis. A physical-based rendering method is adopted to calculate light, and dynamic light rendering data is generated by fusing the light and shadow synthesis data of a scene. A current viewpoint is determined based on spatial position and posture data of a virtual reality head-mounted display, view frustum culling and occlusion culling are performed, and GPU instance rendering is performed on visible building data. A rendering image is optimized by means of high dynamic range processing and temporal anti-aliasing technology, and a final scene image is generated by combining an atmospheric scattering algorithm. The application can effectively improve the rendering efficiency and scene realism of virtual reality building roaming.

Description

Technical Field

[0001] The present invention relates to virtual reality technology, and in particular to a virtual reality building roaming method and system based on scene rendering. Background Art

[0002] Virtual reality technology is increasingly being used in the architectural field, providing an immersive walkthrough experience that allows users to feel as if they are immersed in a real-life architectural scene. Traditional VR architectural walkthrough methods typically directly load complete architectural models for rendering. This places enormous computational pressure on complex, large architectural models, resulting in a drop in rendering frame rate and a poor user experience. To address this issue, researchers have proposed various optimization strategies, such as model simplification, frustum culling, and occlusion culling. While these techniques can improve rendering efficiency to a certain extent, they still have some limitations.

[0003] Insufficient model level of detail processing: Although traditional LOD (Level of Detail) technology can simplify models based on distance, the switching process is relatively abrupt, prone to noticeable visual jumps, affecting the sense of immersion. Moreover, it is difficult to achieve fine level division for the complex details inside buildings, resulting in the loss of nearby details or over-rendering of distant details. Poor lighting rendering effects: Traditional real-time rendering methods cannot strike a balance between lighting quality and rendering efficiency. Although pre-baked light maps can increase rendering speed, they cannot express dynamic lighting and shadow effects. Real-time lighting calculations are computationally intensive and difficult to apply to complex architectural scenes. Lack of optimization for virtual reality devices: Traditional rendering pipelines are mainly designed for desktop monitors and do not fully consider the characteristics of virtual reality devices, such as high refresh rates, low latency, and head tracking. This can cause problems such as screen freezes and delays in virtual reality scenes, affecting the user experience. Summary of the Invention

[0004] The embodiments of the present invention provide a virtual reality building roaming method and system based on scene rendering, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention,

[0006] Provides a virtual reality architectural roaming method based on scene rendering, including:

[0007] Collecting three-dimensional building model data and dividing the three-dimensional building model data into multiple hierarchical blocks using an octree space segmentation algorithm; calculating viewpoint relative distances for the hierarchical blocks and determining building detail level thresholds based on the viewpoint relative distances; constructing a multi-level detail model based on the building detail level thresholds to obtain optimized building model data containing near and far level information; calculating illumination distribution parameters of the optimized building model data using a pre-calculated irradiance transfer technique, and baking the illumination distribution parameters to generate a basic light map;

[0008] Divide the optimized building model data into multiple rendering depth intervals, and set a shadow map of corresponding resolution for each rendering depth interval; synthesize the basic light map and the shadow map in real time to obtain scene light and shadow synthesis data; calculate direct lighting and indirect lighting using a physically based rendering method to obtain a lighting calculation result; fuse the lighting calculation result with the scene light and shadow synthesis data to generate dynamic lighting rendering data; construct a deferred rendering pipeline based on the dynamic lighting rendering data to perform lighting processing on pixels visible from the current viewpoint;

[0009] The method collects spatial position and posture data of a virtual reality headset in real time, and determines current viewpoint parameters based on the spatial position and posture data; performs view frustum culling and occlusion culling based on the current viewpoint parameters to screen out visible building data; performs GPU instanced rendering on the visible building data to generate an initial rendered image; inputs the initial rendered image into a high dynamic range processing unit, and uses an adaptive exposure adjustment algorithm to optimize the scene light and dark contrast to obtain an optimized rendered image; applies temporal anti-aliasing technology to the optimized rendered image to eliminate geometric edges and temporal domain flicker, and combines it with an image-space-based atmospheric scattering algorithm to generate a final scene image.

[0010] Collecting building three-dimensional model data, dividing the building three-dimensional model data into multiple hierarchical blocks using an octree space segmentation algorithm; calculating viewpoint relative distances for the hierarchical blocks, and determining building detail level thresholds based on the viewpoint relative distances include:

[0011] A terrestrial laser scanner is used to obtain first point cloud data ranging from the base of the building to a height of 20 meters, with a scanning resolution of 1.6 millimeters per 10 meters. A drone is used for oblique photogrammetry to obtain second point cloud data from a height of 20 meters to the roof structure, with a flight altitude of 30 meters from the top of the building. Feature points are extracted from the first and second point cloud data, a registration matrix is ​​calculated based on the feature points, and the first and second point cloud data are registered and fused to obtain three-dimensional building model data.

[0012] construct a scene bounding box based on the architectural three-dimensional model data, take the scene bounding box as a root node, calculate a diagonal line length of the scene bounding box as a reference scale, calculate a patch density of the root node based on the architectural three-dimensional model data, when the patch density is greater than a preset density threshold and a node size is greater than a sixteenth of the reference scale, determine an optimal segmentation plane by using principal component analysis, perform spatial segmentation on the architectural three-dimensional model data based on the optimal segmentation plane to obtain a plurality of sub-nodes, calculate a barycentric position of a triangular patch crossing the optimal segmentation plane, and distribute the triangular patch to a corresponding sub-node to form a plurality of hierarchical blocks; and establish a link table between adjacent nodes of the hierarchical blocks to record shared patch information;

[0013] discretize the hierarchical blocks into a voxel grid, calculate a shortest distance from each voxel in the voxel grid to a surface of the architectural three-dimensional model data to form a distance field, compress and store the distance field by using a third-order spherical harmonic function to obtain spherical harmonic coefficients, query the spherical harmonic coefficients of a corresponding voxel based on a viewpoint position, and reconstruct the viewpoint relative distance by using the spherical harmonic coefficients;

[0014] set an architectural detail level threshold value for the hierarchical blocks based on the viewpoint relative distance, when the viewpoint relative distance is less than or equal to a first distance threshold value, set the architectural detail level threshold value as a first threshold value and maintain an original precision of the hierarchical blocks, when the viewpoint relative distance is greater than the first distance threshold value and less than or equal to a second distance threshold value, set the architectural detail level threshold value as a second threshold value and apply an edge collapse algorithm to the hierarchical blocks for simplification, when the viewpoint relative distance is greater than the second distance threshold value and less than or equal to a third distance threshold value, set the architectural detail level threshold value as a third threshold value and perform coplanar triangle merging on the hierarchical blocks, and when the viewpoint relative distance is greater than the third distance threshold value, set the architectural detail level threshold value as a fourth threshold value and perform contour preserving simplification on the hierarchical blocks.

[0015] construct a multi-level detail model based on the architectural detail level threshold value to obtain optimized architectural model data containing near-far level information, and calculate illumination distribution parameters of the optimized architectural model data by using a precomputed irradiance transfer technique, and bake the illumination distribution parameters to generate a base light map including:

[0016] calculate a vertex curvature value and a boundary weight based on the architectural detail level threshold value, construct a comprehensive error metric function based on the vertex curvature value and the boundary weight, and perform edge collapse operations, coplanar merging operations and contour preserving operations on the architectural three-dimensional model data based on the comprehensive error metric function to generate simplified mesh models with different precisions.

[0017] Establishing an octree spatial structure, storing normal information of the three-dimensional building model data in the octree spatial structure; mapping the normal information to the simplified mesh model to generate a multi-level detail model with normal information;

[0018] Constructing a texture space parameterized grid, calculating triangle areas and stretch coefficients based on the multi-level detail model with normal information; constructing a texture distortion energy function by multiplying the triangle areas and the stretch coefficients; and generating optimized building model data by iteratively optimizing UV coordinates to converge the texture distortion energy function.

[0019] Expanding the incident illumination and bidirectional reflectance distribution function into spherical harmonic basis functions; sampling multiple incident directions for the vertices of the optimized building model data; calculating the integral of the visibility function and the spherical harmonic basis functions to obtain a transmission matrix; and calculating illumination distribution parameters based on the transmission matrix;

[0020] A uniformly distributed sampling point grid is constructed, and irradiance values ​​are calculated for the sampling points of the sampling point grid; the irradiance values ​​are spatially interpolated using radial basis functions; a UV space is divided based on the illumination distribution parameters and texture optimization is performed; the UV space is divided into pixel blocks of fixed size; the primary and secondary illumination directions of the pixel blocks are extracted, compressed using principal component analysis, and anisotropic filtering is performed to generate a basic light map.

[0021] Dividing the optimized building model data into multiple rendering depth intervals, setting a shadow map of corresponding resolution for each rendering depth interval; synthesizing the basic light map and the shadow map in real time to obtain scene light and shadow synthesis data includes:

[0022] Calculating a rendering depth range of the optimized building model data, dividing the rendering depth range into a near-view rendering depth interval, a mid-view rendering depth interval, and a far-view rendering depth interval, wherein a depth value of the near-view rendering depth interval is from 0 meters to 50 meters, a depth value of the mid-view rendering depth interval is from 50 meters to 200 meters, and a depth value of the far-view rendering depth interval is greater than 200 meters;

[0023] Generate a first-resolution shadow map for the near-view rendering depth interval, generate a second-resolution shadow map for the mid-view rendering depth interval, and generate a third-resolution shadow map for the far-view rendering depth interval, wherein the resolution of the first-resolution shadow map is 2048 by 2048, the resolution of the second-resolution shadow map is 1024 by 1024, and the resolution of the third-resolution shadow map is 512 by 512;

[0024] Extracting a diffuse lighting component from the base lightmap; performing hardware filtering on the diffuse lighting component by applying a four-by-four pixel percentage proximity filtering sampling method; calculating a depth offset value based on a normal vector of the optimized building model data, and correcting the diffuse lighting component using the depth offset value;

[0025] Constructing a first transition zone at a boundary between the near rendering depth interval and the mid rendering depth interval, and constructing a second transition zone at a boundary between the mid rendering depth interval and the far rendering depth interval; performing a first bilinear interpolation on the first-resolution shadow map and the second-resolution shadow map within the first transition zone, and performing a second bilinear interpolation on the second-resolution shadow map and the third-resolution shadow map within the second transition zone;

[0026] Constructing a deferred rendering buffer, storing position data, normal data, and depth data of the optimized building model data in the deferred rendering buffer; extracting the position data from the deferred rendering buffer, and determining rendering depth intervals to which respective parts of the scene belong based on the position data; calculating a scene change value for the rendering depth interval, and setting a shadow map update frequency based on the scene change value; asynchronously updating the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map according to the shadow map update frequency;

[0027] When the viewpoint position moves, the dividing boundary of the rendering depth interval is recalculated, and the dividing boundary is filtered by applying a time domain smoothing coefficient; when the rendering depth interval is switched, cross-gradient synthesis is performed on the shadow maps of adjacent rendering depth intervals; the diffuse reflection lighting component of the basic light map is synthesized with the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map in real time to generate scene light and shadow synthesis data.

[0028] Calculating direct and indirect lighting using a physically based rendering method to obtain a lighting calculation result, fusing the lighting calculation result with the scene light and shadow synthesis data to generate dynamic lighting rendering data; constructing a deferred rendering pipeline based on the dynamic lighting rendering data to perform lighting processing on pixels visible at the current viewpoint includes:

[0029] Build a physics-based lighting calculation model to decompose lighting information into direct lighting and indirect lighting; construct a ray tracing path from the camera viewpoint, record the intersection position of the light and the scene objects, surface normals, and material properties; calculate the light source visibility based on the intersection position, calculate the light incidence angle using the surface normal, and calculate the direct lighting contribution value based on the material properties;

[0030] Emitting a secondary ray at the intersection position, calculating an indirect light contribution value by using Monte Carlo integration; decomposing the indirect light contribution value into a diffuse reflection component and a specular reflection component according to the material attribute; determining a secondary ray direction by using importance sampling, and collecting ambient light information along the secondary ray direction;

[0031] Weighting and superimposing the direct light contribution value and the indirect light contribution value to obtain a light calculation result; performing time domain accumulation on the light calculation result, eliminating time domain jitter by using an exponential weighted average method; calculating a mixing weight coefficient based on the material attribute, fusing the light calculation result and scene light and shadow synthesis data according to the mixing weight coefficient, and generating dynamic light rendering data;

[0032] Writing position information, normal information, depth information and material information of a scene geometry into the deferred rendering buffer; constructing a dynamic light map containing the dynamic light rendering data, and binding the dynamic light map to the deferred rendering buffer;

[0033] Analyzing the depth information in the deferred rendering buffer to determine visible pixels of a current viewpoint; calculating a spatial relationship between the visible pixels and a light source, and performing light processing on the visible pixels based on the spatial relationship.

[0034] Real-time collecting spatial position and attitude data of a virtual reality head-mounted display, determining a current viewpoint parameter according to the spatial position and attitude data; performing view frustum culling and occlusion culling based on the current viewpoint parameter, and screening out visible building data; performing GPU instanced rendering on the visible building data to generate an initial rendering image, including:

[0035] Obtaining attitude data and spatial position data of a virtual reality head-mounted display; performing attitude fusion on the attitude data by using an extended Kalman filter to obtain head-mounted attitude quaternions; processing the spatial position data based on a simultaneous localization and mapping algorithm to obtain head-mounted spatial coordinates; combining the head-mounted attitude quaternions and the head-mounted spatial coordinates to generate a head-mounted spatial pose matrix;

[0036] Extracting head-mounted position data as a viewpoint position according to the head-mounted spatial pose matrix; converting the head-mounted attitude quaternions into a line-of-sight direction vector; constructing a view frustum equation based on the viewpoint position and the line-of-sight direction vector; generating a projection matrix in combination with a head-mounted field of view angle; combining the view frustum equation and the projection matrix to form a viewpoint culling parameter;

[0037] Constructing a hierarchical bounding box for the buildings in the scene; performing an intersection check on the hierarchical bounding box and the viewpoint culling parameter to obtain a set of candidate buildings that intersect with the view frustum equation; calculating the depth values ​​of the buildings in the candidate building set relative to the viewpoint position; constructing an occlusion map based on the depth values, culling completely occluded buildings, and obtaining a set of visible buildings;

[0038] Classifying the visible building set according to building type to obtain building type groups; extracting shared geometric data and material data for each building type group; uploading the geometric data and material data to a GPU storage buffer; calculating an instanced transformation matrix for each building based on the head-mounted display spatial pose matrix; and organizing the instanced transformation matrix into an instanced data buffer;

[0039] The method comprises the steps of: reading the geometric data and the material data from the GPU storage buffer; applying the instantiated transformation matrix in the instantiated data buffer to the geometric data; rendering the set of visible buildings in parallel using a GPU instantiated rendering pipeline; and synthesizing the rendering result with a background image to generate an initial rendered image of the virtual reality scene.

[0040] According to a second aspect of the embodiments of the present invention,

[0041] Provides a virtual reality architectural roaming system based on scene rendering, including:

[0042] The first unit is configured to collect three-dimensional building model data, divide the three-dimensional building model data into a plurality of hierarchical blocks using an octree space segmentation algorithm, calculate relative viewpoint distances for the hierarchical blocks, and determine a building detail level threshold based on the relative viewpoint distances; construct a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing near and far level information; calculate illumination distribution parameters of the optimized building model data using a pre-calculated irradiance transfer technique, and bake the illumination distribution parameters to generate a basic light map;

[0043] The second unit is configured to divide the optimized building model data into a plurality of rendering depth intervals, set a shadow map of corresponding resolution for each rendering depth interval, synthesize the basic light map with the shadow map in real time to obtain scene light and shadow synthesis data, calculate direct lighting and indirect lighting using a physically based rendering method, fuse the lighting calculation results with the scene light and shadow synthesis data to generate dynamic lighting rendering data, and construct a deferred rendering pipeline based on the dynamic lighting rendering data to perform lighting processing on pixels visible from the current viewpoint.

[0044] The third unit is used to collect the spatial position and posture data of the virtual reality headset in real time, determine the current viewpoint parameters based on the spatial position and posture data; perform frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; perform GPU instantiation rendering on the visible building data to generate an initial rendered image; input the initial rendered image into the high dynamic range processing unit, use an adaptive exposure adjustment algorithm to optimize the scene light and dark contrast, and obtain an optimized rendered image; apply temporal anti-aliasing technology to the optimized rendered image to eliminate geometric edges and temporal domain flicker, and generate a final scene image in combination with an image-space-based atmospheric scattering algorithm.

[0045] According to a third aspect of the embodiments of the present invention,

[0046] An electronic device is provided, comprising:

[0047] processor;

[0048] a memory for storing processor-executable instructions;

[0049] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0050] According to a fourth aspect of the embodiments of the present invention,

[0051] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0052] The beneficial effects of this application are as follows:

[0053] 1. Improve rendering efficiency and performance: Through technologies such as octree space segmentation, multi-level detail model, and frustum / occlusion culling, the rendering calculation workload is effectively reduced, achieving fast loading and smooth roaming of architectural scenes, which is particularly suitable for real-time rendering needs in virtual reality environments.

[0054] 2. Enhanced scene realism and immersion: Using technologies such as pre-calculated irradiance transmission, physically based rendering, dynamic lighting rendering, high dynamic range processing and atmospheric scattering, the lighting and shadow effects, material texture and spatial atmosphere of buildings are realistically simulated, enhancing the realism and immersion of virtual reality scenes.

[0055] 3. Optimizing visual experience and comfort: Utilizing technologies such as multi-resolution settings for shadow maps, adaptive exposure adjustment, and temporal anti-aliasing, this effectively resolves common issues in VR, such as aliasing, flickering, and light-dark contrast imbalance, thereby optimizing the user's visual experience and comfort. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1Schematic diagram of the flow of a virtual reality building roaming method based on scene rendering according to an embodiment of the present invention;

[0057] Figure 2 This is a structural diagram of a virtual reality building roaming system based on scene rendering according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0059] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0060] Figure 1 FIG. 1 is a flow chart of a virtual reality building roaming method based on scene rendering according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0061] S11. Collecting 3D building model data, dividing the 3D building model data into multiple hierarchical blocks using an octree spatial segmentation algorithm; calculating viewpoint relative distances for the hierarchical blocks, and determining building detail level thresholds based on the viewpoint relative distances; constructing a multi-level detail model based on the building detail level thresholds to obtain optimized building model data containing near and far level information; calculating illumination distribution parameters for the optimized building model data using a pre-calculated irradiance transfer technique, and baking the illumination distribution parameters to generate a basic light map;

[0062] S12. Divide the optimized building model data into multiple rendering depth intervals, and set a shadow map of corresponding resolution for each rendering depth interval; synthesize the basic light map with the shadow map in real time to obtain scene light and shadow synthesis data; calculate direct lighting and indirect lighting using a physically based rendering method to obtain a lighting calculation result; fuse the lighting calculation result with the scene light and shadow synthesis data to generate dynamic lighting rendering data; construct a deferred rendering pipeline based on the dynamic lighting rendering data to perform lighting processing on pixels visible from the current viewpoint;

[0063] S13. Collect spatial position and posture data of the virtual reality headset in real time, and determine current viewpoint parameters based on the spatial position and posture data; perform frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; perform GPU instantiation rendering on the visible building data to generate an initial rendered image; input the initial rendered image into a high dynamic range processing unit, and use an adaptive exposure adjustment algorithm to optimize the scene light and dark contrast to obtain an optimized rendered image; apply temporal anti-aliasing technology to the optimized rendered image to eliminate geometric edges and temporal domain flicker, and generate a final scene image in combination with an image-space-based atmospheric scattering algorithm.

[0064] In an optional embodiment, collecting building 3D model data, dividing the building 3D model data into a plurality of hierarchical blocks using an octree space segmentation algorithm; calculating viewpoint relative distances for the hierarchical blocks, and determining building detail level thresholds based on the viewpoint relative distances includes:

[0065] A terrestrial laser scanner is used to obtain first point cloud data ranging from the base of the building to a height of 20 meters, with a scanning resolution of 1.6 millimeters per 10 meters. A drone is used for oblique photogrammetry to obtain second point cloud data from a height of 20 meters to the roof structure, with a flight altitude of 30 meters from the top of the building. Feature points are extracted from the first and second point cloud data, a registration matrix is ​​calculated based on the feature points, and the first and second point cloud data are registered and fused to obtain three-dimensional building model data.

[0066] A scene bounding box is constructed for the three-dimensional building model data, the scene bounding box is used as a root node, and the diagonal length of the scene bounding box is calculated as a reference scale; a facet density of the root node is calculated based on the three-dimensional building model data, and when the facet density is greater than a preset density threshold and the node size is greater than one-sixteenth of the reference scale, an optimal segmentation plane is determined using principal component analysis; the three-dimensional building model data is spatially segmented based on the optimal segmentation plane to obtain a plurality of child nodes; the centroid positions of triangular facets crossing the optimal segmentation plane are calculated, and the triangular facets are assigned to corresponding child nodes to form a plurality of hierarchical blocks; and a link table is established between adjacent nodes of the hierarchical blocks to record shared facet information;

[0067] Discretizing the hierarchical blocks into a voxel grid, calculating the shortest distance from each voxel in the voxel grid to the surface of the building three-dimensional model data to form a distance field; compressing and storing the distance field using a third-order spherical harmonic function to obtain spherical harmonic coefficients; querying the spherical harmonic coefficients of the corresponding voxel according to the viewpoint position, and reconstructing the viewpoint relative distance using the spherical harmonic coefficients;

[0068] An architectural detail level threshold is set for the hierarchical block based on the viewpoint relative distance. When the viewpoint relative distance is less than or equal to a first distance threshold, the architectural detail level threshold is set to the first threshold and the original precision of the hierarchical block is maintained. When the viewpoint relative distance is greater than the first distance threshold and less than or equal to a second distance threshold, the architectural detail level threshold is set to the second threshold, and the hierarchical block is simplified by applying an edge collapse algorithm. When the viewpoint relative distance is greater than the second distance threshold and less than or equal to a third distance threshold, the architectural detail level threshold is set to the third threshold, and coplanar triangle merging is performed on the hierarchical block. When the viewpoint relative distance is greater than the third distance threshold, the architectural detail level threshold is set to the fourth threshold, and the hierarchical block is simplified with contour preservation.

[0069] Collect building 3D model data and divide it into hierarchical blocks using the octree space segmentation algorithm. Dynamically adjust the building detail level according to the relative distance of the viewpoint to achieve efficient 3D model rendering.

[0070] First, a terrestrial laser scanner was used to acquire point cloud data from the base of the building to a height of 20 meters. The scanning resolution was set to 1.6 mm per 10 meters to ensure high-precision data. Simultaneously, drone-assisted oblique photogrammetry was used to acquire point cloud data from a height of 20 meters above the building to the roof structure. The flight altitude was set to 30 meters from the building's top to ensure data integrity. For example, the terrestrial laser scanner captured detailed data on columns and walls at the base of the building, while the drone-assisted oblique photogrammetry captured data on elevated structures such as the roof and chimney.

[0071] Next, feature points are extracted from the point cloud data obtained from terrestrial laser scanning and drone-derived oblique photogrammetry. For example, distinct features such as building corners and edges are extracted. A registration matrix is ​​calculated based on these feature points, and the two point cloud data sets are fused and registered. The registration process can employ methods such as the iterative closest point algorithm. Ultimately, a complete 3D building model is obtained. For example, column data obtained from terrestrial laser scanning can be precisely aligned with roof data obtained from drone-derived oblique photogrammetry to form a complete building model.

[0072] Next, a scene bounding box is constructed for the building 3D model data. The scene bounding box is used as the root node of the octree. The diagonal length of the scene bounding box is calculated as the reference scale. For example, for a building that is 100 meters long, 50 meters wide, and 30 meters high, the diagonal length of its scene bounding box is approximately 114 meters, which serves as the reference scale for subsequent segmentation. Then, the face density of the root node is calculated, that is, the number of triangular facets per unit volume. If the face density is greater than the preset density threshold (for example, 100 facets per cubic meter) and the node size is greater than one sixteenth of the reference scale (for example, 114 / 16 = 7.125 meters), principal component analysis is used to determine the optimal segmentation plane. The principal component analysis method can find the direction with the largest data variance as the normal vector of the segmentation plane. Based on the optimal segmentation plane, the space of the current node is divided into eight child nodes. The center of gravity positions of the triangular facets that cross the optimal segmentation plane are calculated, and these triangular facets are assigned to the corresponding child nodes to form multiple hierarchical blocks. A link table is established between adjacent nodes to record shared facet information for subsequent processing.

[0073] The hierarchical blocks are then discretized into a voxel grid. For example, each block is divided into voxels of 1 cubic meter. For each voxel in the voxel grid, the shortest distance to the surface of the 3D building model data is calculated to form a distance field. The distance field is compressed and stored using third-order spherical harmonics, resulting in spherical harmonic coefficients. Spherical harmonics represent the distance field as a combination of coefficients, thus reducing storage space. Based on the viewpoint position, the spherical harmonic coefficients of the corresponding voxel are queried and reconstructed to obtain the viewpoint's relative distance.

[0074] Finally, a building detail level threshold is set for the hierarchical block based on the relative distance of the viewpoint. For example, three distance thresholds are set: the first distance threshold is 10 meters, the second distance threshold is 50 meters, and the third distance threshold is 100 meters. When the relative distance of the viewpoint is less than or equal to 10 meters, the building detail level threshold is set to the first threshold (for example, level 0), and the original accuracy of the hierarchical block is maintained. When the relative distance of the viewpoint is greater than 10 meters and less than or equal to 50 meters, the building detail level threshold is set to the second threshold (for example, level 1), and the edge collapse algorithm is applied to the hierarchical block for simplification. When the relative distance of the viewpoint is greater than 50 meters and less than or equal to 100 meters, the building detail level threshold is set to the third threshold (for example, level 2), and coplanar triangle merging is performed on the hierarchical block. When the relative distance of the viewpoint is greater than 100 meters, the building detail level threshold is set to the fourth threshold (for example, level 3), and the outline-preserving simplification is performed on the hierarchical block.

[0075] The solution of this application can:

[0076] Improved rendering efficiency: By dynamically adjusting the level of building detail based on viewpoint distance, the amount of rendered data is reduced, thereby improving rendering efficiency. Reduced storage space: Using spherical harmonics to compress and store distance fields reduces storage space requirements. Improved user experience: While preserving nearby building details, the model of distant buildings is simplified, achieving efficient scene rendering and improving the user experience.

[0077] In an optional embodiment, constructing a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing near and far level information; calculating illumination distribution parameters of the optimized building model data using a pre-calculated irradiance transfer technique, and baking the illumination distribution parameters to generate a basic light map includes:

[0078] Calculating vertex curvature values ​​and boundary weights according to the building detail level threshold, and constructing a comprehensive error metric function with the vertex curvature values ​​and the boundary weights; performing edge collapse operations, coplanar merging operations, and contour preservation operations on the building three-dimensional model data based on the comprehensive error metric function to generate simplified mesh models of different accuracies;

[0079] Establishing an octree spatial structure, storing normal information of the three-dimensional building model data in the octree spatial structure; mapping the normal information to the simplified mesh model to generate a multi-level detail model with normal information;

[0080] Constructing a texture space parameterized grid, calculating triangle areas and stretch coefficients based on the multi-level detail model with normal information; constructing a texture distortion energy function by multiplying the triangle areas and the stretch coefficients; and generating optimized building model data by iteratively optimizing UV coordinates to converge the texture distortion energy function.

[0081] Expanding the incident illumination and bidirectional reflectance distribution function into spherical harmonic basis functions; sampling multiple incident directions for the vertices of the optimized building model data; calculating the integral of the visibility function and the spherical harmonic basis functions to obtain a transmission matrix; and calculating illumination distribution parameters based on the transmission matrix;

[0082] A uniformly distributed sampling point grid is constructed, and irradiance values ​​are calculated for the sampling points of the sampling point grid; the irradiance values ​​are spatially interpolated using radial basis functions; a UV space is divided based on the illumination distribution parameters and texture optimization is performed; the UV space is divided into pixel blocks of fixed size; the primary and secondary illumination directions of the pixel blocks are extracted, compressed using principal component analysis, and anisotropic filtering is performed to generate a basic light map.

[0083] A method for generating basic lightmaps for building models can effectively improve rendering efficiency and visual quality. The core idea of ​​this method is to construct a multi-level detail model based on the building detail level threshold, calculate the lighting distribution parameters using pre-computed irradiance transfer technology, and finally bake the generated basic lightmap.

[0084] First, set the Level of Detail threshold. For example, set the threshold to 0.1. This means that details in the model that are smaller than this threshold will be simplified.

[0085] Next, the curvature value and boundary weight are calculated for each vertex in the model. The vertex curvature value measures the degree of geometric variation around the vertex. For example, vertices with large curvature values ​​are often located at the edges or corners of the model. The boundary weight measures the likelihood that a vertex is located on the model boundary. For example, vertices located on the outer contour of the model have a higher boundary weight. These two values ​​are combined into a comprehensive error metric function. An example function is a weighted sum of the curvature value and boundary weight, where the weights can be adjusted according to actual needs, such as setting the curvature weight to 0.7 and the boundary weight to 0.3.

[0086] Based on a comprehensive error metric function, edge collapse, coplanar merging, and contour-preserving operations are performed on the 3D building model data to generate simplified mesh models of varying precision. For example, three models with varying precision can be generated, retaining 90%, 50%, and 10% of the original model's triangular faces, respectively. The edge collapse operation merges an edge and its two vertices into a new vertex, the coplanar merging operation merges multiple faces lying on the same plane into a single face, and the contour-preserving operation strives to preserve the model's original outline during the simplification process.

[0087] An octree spatial structure is established, and the normal information of the original high-precision model is stored within it. An octree is a spatial partitioning data structure that recursively divides a three-dimensional space into eight subspaces. This normal information is then mapped to the previously generated simplified mesh model. Specifically, for each vertex in the simplified model, the smallest cell containing that vertex is found in the octree, and the normal information stored in that cell is used as the normal for that vertex.

[0088] A parameterized mesh in texture space is constructed, and triangle areas and stretch coefficients are calculated based on a multi-level detail model with normal information. The stretch coefficient measures the degree to which the texture is stretched or compressed on the model surface. For example, a larger stretch coefficient indicates that the texture is stretched more severely in that area. The product of the triangle area and the stretch coefficient is used to construct a texture distortion energy function. The texture distortion energy function is converged by iteratively optimizing the UV coordinates, generating optimized building model data. For example, gradient descent is used to iteratively optimize the UV coordinates until the change in the energy function value is less than a preset threshold, such as 0.001.

[0089] Expand the incident illumination and bidirectional reflectance distribution functions into spherical harmonic basis functions. Sample multiple incident directions for the vertices of the optimized building model data, for example, sampling six directions evenly distributed across a hemisphere. Integrate the visibility function and the spherical harmonic basis functions to obtain a transfer matrix. Calculate the illumination distribution parameters based on the transfer matrix.

[0090] Construct a uniformly distributed grid of sampling points and calculate irradiance values ​​for each of these points. Use radial basis functions to spatially interpolate these irradiance values. Partition the UV space based on the lighting distribution parameters and perform texture optimization. Divide the UV space into fixed-size pixel blocks, for example, 8x8 pixels. Extract the primary and secondary lighting directions for each block, compress them using principal component analysis, and perform anisotropic filtering to generate a base lightmap.

[0091] The solution of this application can:

[0092] Improved rendering efficiency: Through multi-level detail models and pre-calculated irradiance transfer technology, the computational effort required for real-time rendering is reduced, thereby improving rendering efficiency. Enhanced visual effects: The important details and lighting information of the model are retained, making the generated scene more realistic and visually better. Optimized resource management: The generated basic lightmaps occupy less storage space, reducing the demand on hardware resources.

[0093] In an optional embodiment, dividing the optimized building model data into a plurality of rendering depth intervals, setting a shadow map of corresponding resolution for each rendering depth interval; and synthesizing the basic light map and the shadow map in real time to obtain scene light and shadow synthesis data includes:

[0094] Calculating a rendering depth range of the optimized building model data, dividing the rendering depth range into a near-view rendering depth interval, a mid-view rendering depth interval, and a far-view rendering depth interval, wherein a depth value of the near-view rendering depth interval is from 0 meters to 50 meters, a depth value of the mid-view rendering depth interval is from 50 meters to 200 meters, and a depth value of the far-view rendering depth interval is greater than 200 meters;

[0095] Generate a first-resolution shadow map for the near-view rendering depth interval, generate a second-resolution shadow map for the mid-view rendering depth interval, and generate a third-resolution shadow map for the far-view rendering depth interval, wherein the resolution of the first-resolution shadow map is 2048 by 2048, the resolution of the second-resolution shadow map is 1024 by 1024, and the resolution of the third-resolution shadow map is 512 by 512;

[0096] Extracting a diffuse lighting component from the base lightmap; performing hardware filtering on the diffuse lighting component by applying a four-by-four pixel percentage proximity filtering sampling method; calculating a depth offset value based on a normal vector of the optimized building model data, and correcting the diffuse lighting component using the depth offset value;

[0097] Constructing a first transition zone at a boundary between the near rendering depth interval and the mid rendering depth interval, and constructing a second transition zone at a boundary between the mid rendering depth interval and the far rendering depth interval; performing a first bilinear interpolation on the first-resolution shadow map and the second-resolution shadow map within the first transition zone, and performing a second bilinear interpolation on the second-resolution shadow map and the third-resolution shadow map within the second transition zone;

[0098] Constructing a deferred rendering buffer, storing position data, normal data, and depth data of the optimized building model data in the deferred rendering buffer; extracting the position data from the deferred rendering buffer, and determining rendering depth intervals to which respective parts of the scene belong based on the position data; calculating a scene change value for the rendering depth interval, and setting a shadow map update frequency based on the scene change value; asynchronously updating the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map according to the shadow map update frequency;

[0099] When the viewpoint position moves, the dividing boundary of the rendering depth interval is recalculated, and the dividing boundary is filtered by applying a time domain smoothing coefficient; when the rendering depth interval is switched, cross-gradient synthesis is performed on the shadow maps of adjacent rendering depth intervals; the diffuse reflection lighting component of the basic light map is synthesized with the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map in real time to generate scene light and shadow synthesis data.

[0100] The shadow map generation and synthesis method for optimizing architectural model rendering is implemented as follows:

[0101] First, preprocess and optimize the building model data. Vertex position information for the building model is obtained, and the distance from each vertex to the viewpoint is calculated to determine the model's rendering depth range. For example, the model's rendering depth range is 0 to 300 meters. Based on this range, the rendering depth range is divided into multiple intervals, such as the near-sighted zone (0 to 50 meters), the mid-sighted zone (50 to 200 meters), and the distant zone (over 200 meters).

[0102] Then, shadow maps of varying resolutions are generated for different rendering depth ranges. For the near-field range, a first-resolution shadow map with a resolution of 2048x2048 is generated. For the mid-field range, a second-resolution shadow map with a resolution of 1024x1024 is generated. For the distant range, a third-resolution shadow map with a resolution of 512x512 is generated. Shadow maps can be generated using shadow mapping, which renders the scene using the light source as the viewpoint and stores depth information in the shadow map.

[0103] Next, the diffuse lighting component is extracted from the pre-generated base lightmap. Hardware filtering is applied to this extracted diffuse lighting component using a 4x4 pixel percentage-neighbor filtering sampling scheme to smooth the lighting effect. For example, if the diffuse lighting component values ​​of a pixel for its 16 neighboring pixels are 0.1, 0.2, ..., 0.16, respectively, the diffuse lighting component value of that pixel after filtering is the weighted average of these values.

[0104] A depth offset is then calculated based on the normal vector of the optimized building model data. This offset is used to correct the filtered diffuse lighting component to avoid artifacts such as shadow acne. For example, if the normal vector is at a large angle to the light direction, a larger depth offset is used to prevent shadows from shifting incorrectly across the surface.

[0105] A first transition zone is constructed at the boundary of the near and mid-range rendering depth intervals (at 50 meters), for example, a transition zone with a width of 10 meters. A second transition zone is constructed at the boundary of the mid and far-range rendering depth intervals (at 200 meters), for example, a transition zone with a width of 20 meters. Bilinear interpolation is performed on the first and second resolution shadow maps within the first transition zone to smooth shadow transitions. Bilinear interpolation is performed on the second and third resolution shadow maps within the second transition zone to smooth shadow transitions.

[0106] Construct a deferred rendering buffer and store the position, normal, and depth data of the optimized building model data in the deferred rendering buffer. Extract the position data from the deferred rendering buffer. Based on the position data, determine the rendering depth range for each part of the scene. Calculate the scene change value for the rendering depth range. The scene change value can be calculated based on changes in factors such as the position and lighting of objects in the scene. Set the shadow map update frequency based on the scene change value. For example, if the scene change value is large, increase the shadow map update frequency. Asynchronously update the first, second, and third resolution shadow maps based on the shadow map update frequency.

[0107] When the viewpoint moves, the boundaries between the rendering depth intervals are recalculated. Temporal smoothing coefficients are applied to the boundaries to avoid drastic changes. For example, a low-pass filter is used to smooth the boundary values. When switching between rendering depth intervals, cross-fading is performed on the shadow maps of adjacent rendering depth intervals to avoid sudden shadow changes. Finally, the diffuse lighting component of the base light map is composited with the shadow map of the corresponding resolution in real time to generate the scene lighting and shadow synthesis data.

[0108] The solution of this application can:

[0109] Improve rendering efficiency. By dividing the rendering depth range according to distance and using shadow maps of different resolutions, unnecessary calculations can be reduced and rendering efficiency can be improved, which is especially significant when dealing with large-scale scenes. Improve shadow quality. Through depth offset correction and bilinear interpolation of transition bands, artifacts such as shadow acne and aliasing at shadow edges can be effectively reduced, improving shadow quality and making the scene more realistic. Enhance the dynamic performance of the scene. By dynamically updating the shadow map according to the scene change value and smoothly transitioning the shadow when the viewpoint moves, the dynamic performance of the scene can be made more natural and smooth.

[0110] In an optional embodiment, direct lighting and indirect lighting are calculated using a physically based rendering method to obtain lighting calculation results, the lighting calculation results are merged with the scene light and shadow synthesis data to generate dynamic lighting rendering data; and a deferred rendering pipeline is constructed based on the dynamic lighting rendering data to perform lighting processing on pixels visible at the current viewpoint, including:

[0111] Build a physics-based lighting calculation model to decompose lighting information into direct lighting and indirect lighting; construct a ray tracing path from the camera viewpoint, record the intersection position of the light and the scene objects, surface normals, and material properties; calculate the light source visibility based on the intersection position, calculate the light incidence angle using the surface normal, and calculate the direct lighting contribution value based on the material properties;

[0112] A secondary light is emitted at the intersection position, and an indirect lighting contribution value is calculated using Monte Carlo integration; the indirect lighting contribution value is decomposed into a diffuse reflection component and a specular reflection component according to the material properties; the direction of the secondary light is determined using importance sampling, and ambient lighting information is collected along the direction of the secondary light;

[0113] The direct illumination contribution value and the indirect illumination contribution value are weightedly superimposed to obtain an illumination calculation result; the illumination calculation result is temporally accumulated, and temporal jitter is eliminated using an exponentially weighted average method; a mixing weight coefficient is calculated based on the material properties, and the illumination calculation result is fused with the scene light and shadow synthesis data according to the mixing weight coefficient to generate dynamic illumination rendering data;

[0114] Writing the position information, normal information, depth information, and material information of the scene geometry into the deferred rendering buffer; constructing a dynamic light map containing the dynamic lighting rendering data, and binding the dynamic light map to the deferred rendering buffer;

[0115] The depth information in the deferred rendering buffer is analyzed to determine visible pixels of the current viewpoint; a spatial relationship between the visible pixels and a light source is calculated, and lighting processing is performed on the visible pixels based on the spatial relationship.

[0116] The physically based dynamic lighting rendering method achieves high-quality real-time rendering effects by combining ray tracing and scene light and shadow synthesis data.

[0117] First, perform scene preprocessing. Model all objects in the scene and assign them corresponding material properties, including diffuse reflectance, specular reflectance, and roughness. For example, a red plastic ball might have a diffuse reflectance of 0.8, a specular reflectance of 0.2, and a roughness of 0.1. Also, record the light source information in the scene, including its type, position, color, and intensity. For example, a point light source might have a position of (0,5,0), a color of white, and an intensity of 1000.

[0118] Next, a physically based lighting calculation model is constructed. For each pixel in the scene, a ray tracing path is constructed starting from the camera viewpoint. This path records the intersection information of the light and the scene objects, including the intersection position, surface normal, material properties, etc. For example, the light intersects with the red plastic ball at point (1,1,1), and the surface normal is (0,1,0). Then, the visibility of the light source is calculated based on the intersection position. If the light source is blocked by other objects, the direct lighting contribution of the pixel is zero. Otherwise, the surface normal and the light source direction are used to calculate the light incident angle, and the direct lighting contribution is calculated by combining the material properties and the light source intensity. For example, the light source direction is (0,-1,0), the incident angle is 0 degrees, and the direct lighting contribution is 800.

[0119] At the same time, multiple secondary rays are emitted at the intersection, and the indirect lighting contribution is calculated using Monte Carlo integration. The direction of the secondary rays is determined by importance sampling. For example, for rough surfaces, the secondary light directions are more dispersed; for smooth surfaces, the secondary light directions are more concentrated. Ambient lighting information is collected along the secondary ray directions, and the indirect lighting contribution is decomposed into diffuse and specular components based on the material properties. For example, the indirect lighting contribution of a red plastic ball is 200, of which the diffuse component is 160 and the specular component is 40.

[0120] The direct and indirect lighting contributions are weighted and superimposed to obtain the illumination calculation result for the pixel. To eliminate temporal jitter, the illumination calculation results are accumulated over time using an exponentially weighted average. For example, if the illumination calculation result for the current frame is 1000 and the illumination calculation result for the previous frame is 900, and the weighted average coefficient is 0.9, the final illumination result is 990.

[0121] The lighting calculation results are then combined with the pre-calculated scene lighting and shadow data. A blending weight coefficient is calculated based on the material properties. For example, for opaque objects, the blending weight coefficient is 1, completely using the lighting calculation results. For translucent objects, the blending weight coefficient is less than 1, and the lighting calculation results are blended with the scene lighting and shadow data. This ultimately generates dynamic lighting rendering data.

[0122] Finally, the deferred rendering pipeline is constructed. The position, normal, depth, and material information of the scene geometry are written to the deferred rendering buffer. The dynamic lighting rendering data is constructed into a dynamic lightmap and bound to the deferred rendering buffer. The depth information in the deferred rendering buffer is analyzed to determine the visible pixels from the current viewpoint. The spatial relationship between the visible pixels and the light source is calculated, and lighting is performed on the visible pixels based on this spatial relationship, ultimately rendering the image.

[0123] The solution of this application can:

[0124] Higher rendering efficiency: Deferred rendering technology separates lighting calculations from geometry processing, improving rendering efficiency. More realistic lighting effects: Physically based rendering methods more accurately simulate the propagation and interaction of light, resulting in more realistic lighting effects. Stronger artistic expression: By integrating scene light and shadow synthesis data, richer artistic effects can be achieved, such as global illumination and ambient occlusion.

[0125] In an optional embodiment, real-time acquisition of spatial position and posture data of a virtual reality headset, determining current viewpoint parameters based on the spatial position and posture data; performing frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; and performing GPU instanced rendering on the visible building data to generate an initial rendered image includes:

[0126] Acquire posture data and spatial position data of a virtual reality head-mounted display (HMD); perform posture fusion on the posture data using an extended Kalman filter to obtain a HMD posture quaternion; process the spatial position data based on a real-time positioning and mapping algorithm to obtain HMD spatial coordinates; and combine the HMD posture quaternion with the HMD spatial coordinates to generate a HMD spatial pose matrix;

[0127] Extracting head-mounted display position data as a viewpoint position according to the head-mounted display spatial pose matrix; converting the head-mounted display pose quaternion into a line of sight direction vector; constructing a view cone equation based on the viewpoint position and the line of sight direction vector; generating a projection matrix based on the head-mounted display field of view angle; and combining the view cone equation with the projection matrix to form a viewpoint culling parameter;

[0128] Constructing a hierarchical bounding box for the buildings in the scene; performing an intersection check on the hierarchical bounding box and the viewpoint culling parameter to obtain a set of candidate buildings that intersect with the view frustum equation; calculating the depth values ​​of the buildings in the candidate building set relative to the viewpoint position; constructing an occlusion map based on the depth values, culling completely occluded buildings, and obtaining a set of visible buildings;

[0129] Classifying the visible building set according to building type to obtain building type groups; extracting shared geometric data and material data for each building type group; uploading the geometric data and material data to a GPU storage buffer; calculating an instanced transformation matrix for each building based on the head-mounted display spatial pose matrix; and organizing the instanced transformation matrix into an instanced data buffer;

[0130] The method comprises the steps of: reading the geometric data and the material data from the GPU storage buffer; applying the instantiated transformation matrix in the instantiated data buffer to the geometric data; rendering the set of visible buildings in parallel using a GPU instantiated rendering pipeline; and synthesizing the rendering result with a background image to generate an initial rendered image of the virtual reality scene.

[0131] A virtual reality scene rendering method is used to efficiently render large-scale urban building models, providing an immersive virtual reality experience. The core of this method is to leverage the spatial positioning and posture information of the virtual reality headset, combined with frustum culling and occlusion culling techniques, to perform GPU instanced rendering of visible buildings, thereby achieving a smooth virtual reality scene display.

[0132] First, obtain the VR headset's pose data and spatial position data. The headset's pose data includes the headset's rotation angle, while the spatial position data describes the headset's three-dimensional coordinates in the real world. For example, pose data can be expressed as rotation angles around the X, Y, and Z axes, while spatial position data can be expressed as X, Y, and Z coordinate values, such as (1.2, 2.5, 0.8).

[0133] Then, the acquired posture data is fused. Using an extended Kalman filter, the HMD posture data is processed to obtain more accurate and smooth posture information, represented by quaternions. For example, the fused posture quaternion might be (0.7, 0.2, 0.1, 0.6).

[0134] At the same time, the acquired spatial position data is processed. The spatial position data of the headset is processed using a Simultaneous Localization and Mapping (SLAM) algorithm to obtain the precise spatial coordinates of the headset in the virtual scene. For example, the processed spatial coordinates can be (10.5, 25.2, 8.7).

[0135] Next, the fused pose quaternion and the processed spatial coordinates are combined to generate a headset spatial pose matrix. This matrix describes the position and orientation of the headset in the virtual scene.

[0136] According to the headset spatial pose matrix, the headset position data is extracted as the viewpoint position, and the headset pose quaternion is converted into a line-of-sight direction vector. For example, the viewpoint position can be (10.5, 25.2, 8.7), and the line-of-sight direction vector can be (0.3, 0.6, 0.7).

[0137] Based on the viewpoint position and the line-of-sight direction vector, combined with the field-of-view angle of the headset, a frustum equation is constructed and a projection matrix is generated. The frustum equation defines the visible area in the virtual scene, and the projection matrix is used to project the three-dimensional scene onto a two-dimensional screen. For example, the field-of-view angle can be 90 degrees.

[0138] A hierarchical bounding volume is constructed for the buildings in the scene. A hierarchical bounding volume is a data structure used for fast collision detection, which wraps the building model with multiple bounding boxes at different levels.

[0139] The constructed hierarchical bounding volume is intersected with the viewpoint rejection parameters. A candidate building set that intersects with the frustum equation is obtained. This means that these buildings may be within the user's field of view.

[0140] The depth values of the buildings in the candidate building set relative to the viewpoint position are calculated. The depth value represents the distance between the building and the viewpoint.

[0141] Based on the depth values, a occlusion map is constructed, and the completely occluded buildings are removed to obtain the final visible building set. This means that these buildings are within the user's field of view and are not occluded by other buildings.

[0142] The visible building set is classified according to building type to obtain building type groups. For example, buildings can be classified into residential, commercial, industrial, etc.

[0143] For each building type group, shared geometric data and material data are extracted. For example, all residential buildings may share the same roof geometry and material.

[0144] Upload the extracted geometry and material data to the GPU storage buffer.

[0145] Calculate the instanced transformation matrix of each building based on the headset space pose matrix. This matrix describes the position, rotation, and scale of each building in the virtual scene.

[0146] Organizes the computed instanced transformation matrices into instanced data buffers.

[0147] Reads geometry and material data from GPU storage buffers.

[0148] Applies the instanced transformation matrix in the instanced data buffer to the geometry data.

[0149] The GPU instanced rendering pipeline is used to render the visible building collection in parallel.

[0150] The rendering result is synthesized with the background image to generate the initial rendering image of the virtual reality scene, which is finally presented in the user's virtual reality headset.

[0151] The solution of this application can:

[0152] Improve rendering efficiency: Through frustum culling and occlusion culling techniques, the number of buildings that need to be rendered is effectively reduced, thereby significantly improving rendering efficiency and making real-time rendering of large-scale urban building models possible. Enhance immersion: By collecting and processing the spatial position and posture data of the headset in real time, the user's perspective is updated in real time, enhancing the user's immersion and providing a more realistic virtual reality experience. Reduce hardware pressure: Through GPU instanced rendering technology, the GPU's computational burden is reduced, lowering the requirements for hardware devices, allowing this method to run on a wider range of hardware platforms.

[0153] Figure 2 FIG. 1 is a structural diagram of a virtual reality building roaming system based on scene rendering according to an embodiment of the present invention. Figure 2 As shown, the system includes:

[0154] The first unit is configured to collect three-dimensional building model data, divide the three-dimensional building model data into a plurality of hierarchical blocks using an octree space segmentation algorithm, calculate relative viewpoint distances for the hierarchical blocks, and determine a building detail level threshold based on the relative viewpoint distances; construct a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing near and far level information; calculate illumination distribution parameters of the optimized building model data using a pre-calculated irradiance transfer technique, and bake the illumination distribution parameters to generate a basic light map;

[0155] The second unit is used for dividing a plurality of rendering depth intervals based on the optimized building model data, setting a corresponding resolution shadow map for each rendering depth interval, performing real-time synthesis on the basic light map and the shadow map to obtain scene light synthesis data, calculating direct light and indirect light by using a physical-based rendering method, fusing the light calculation result and the scene light synthesis data to generate dynamic light rendering data, and constructing a deferred rendering pipeline according to the dynamic light rendering data to perform light processing on a current view point visible pixel.

[0156] The third unit is used for collecting spatial position and posture data of a virtual reality head-mounted display in real time, determining a current view point parameter according to the spatial position and posture data, performing view cone culling and occlusion culling based on the current view point parameter to screen out visible building data, performing GPU instantiation rendering on the visible building data to generate an initial rendering image, inputting the initial rendering image into a high dynamic range processing unit, and adopting an adaptive exposure adjustment algorithm to optimize scene light contrast to obtain an optimized rendering image, applying time domain anti-aliasing technology to the optimized rendering image to eliminate geometric edges and time domain flicker, and combining an image space-based atmospheric scattering algorithm to generate a final scene image.

[0157] The third aspect of the embodiment of the present application,

[0158] An electronic device is provided.

[0159] A processor;

[0160] A memory for storing processor-executable instructions;

[0161] The processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0162] The fourth aspect of the embodiment of the present application,

[0163] A computer readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0164] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having stored thereon computer readable program instructions that, when executed by a processor, implement various aspects of the present application.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual reality architectural roaming method based on scene rendering, characterized in that: include: Collecting building three-dimensional model data, dividing the building three-dimensional model data into multiple hierarchical blocks using an octree space segmentation algorithm; calculating viewpoint relative distances for the hierarchical blocks, and determining building detail level thresholds based on the viewpoint relative distances; Constructing a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing near and far level information; calculating illumination distribution parameters of the optimized building model data using pre-calculated irradiance transfer technology, and baking the illumination distribution parameters to generate a basic light map; Dividing the optimized building model data into a plurality of rendering depth intervals, and setting a shadow map of corresponding resolution for each rendering depth interval; Synthesize the basic light map and the shadow map in real time to obtain scene light and shadow synthesis data; Calculating direct lighting and indirect lighting using a physically based rendering method, fusing the lighting calculation results with the scene light and shadow synthesis data to generate dynamic lighting rendering data; Constructing a deferred rendering pipeline based on the dynamic lighting rendering data to perform lighting processing on pixels visible from the current viewpoint; Collecting spatial position and posture data of a virtual reality head display in real time, and determining current viewpoint parameters based on the spatial position and posture data; Performing frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; Performing GPU instanced rendering on the visible building data to generate an initial rendered image; inputting the initial rendered image into a high dynamic range processing unit, and optimizing the scene light and dark contrast using an adaptive exposure adjustment algorithm to obtain an optimized rendered image; applying a temporal anti-aliasing technique to the optimized rendered image to eliminate geometric edges and temporal domain flicker, and combining it with an image-space-based atmospheric scattering algorithm to generate a final scene image; The step of calculating the relative viewpoint distance for the hierarchical blocks and determining the architectural detail level threshold according to the relative viewpoint distance specifically includes: Discretizing the hierarchical blocks into a voxel grid, calculating the shortest distance from each voxel in the voxel grid to the surface of the building three-dimensional model data to form a distance field; compressing and storing the distance field using a third-order spherical harmonic function to obtain spherical harmonic coefficients; querying the spherical harmonic coefficients of the corresponding voxel according to the viewpoint position, and reconstructing the viewpoint relative distance using the spherical harmonic coefficients; When the viewpoint relative distance is less than or equal to a first distance threshold, the architectural detail level threshold is set to a first threshold and the original precision of the hierarchical block is maintained; when the viewpoint relative distance is greater than the first distance threshold and less than or equal to a second distance threshold, the architectural detail level threshold is set to a second threshold, and the hierarchical block is simplified by applying an edge collapse algorithm; when the viewpoint relative distance is greater than the second distance threshold and less than or equal to a third distance threshold, the architectural detail level threshold is set to a third threshold, and coplanar triangle merging is performed on the hierarchical block; when the viewpoint relative distance is greater than the third distance threshold, the architectural detail level threshold is set to a fourth threshold, and the hierarchical block is simplified with contour preservation.

2. The method according to claim 1, characterized in that Collecting building 3D model data and dividing the building 3D model data into multiple hierarchical blocks using an octree space segmentation algorithm includes: A terrestrial laser scanner is used to obtain first point cloud data ranging from the base of the building to a height of 20 meters, with a scanning resolution of 1.6 millimeters per 10 meters. Second point cloud data of the upper part of the building and the roof structure is obtained using drone oblique photogrammetry, with a flight altitude of 30 meters from the top of the building. Feature points are extracted from the first and second point cloud data, a registration matrix is ​​calculated based on the feature points, and the first and second point cloud data are registered and fused to obtain three-dimensional building model data. A scene bounding box is constructed for the building three-dimensional model data, the scene bounding box is used as a root node, and the diagonal length of the scene bounding box is calculated as a reference scale; the facet density of the root node is calculated based on the building three-dimensional model data, and when the facet density is greater than a preset density threshold and the node size is greater than one sixteenth of the reference scale, the optimal segmentation plane is determined using principal component analysis; the building three-dimensional model data is spatially segmented based on the optimal segmentation plane to obtain multiple child nodes; the center of gravity position of the triangular facets crossing the optimal segmentation plane is calculated, and the triangular facets are assigned to corresponding child nodes to form multiple hierarchical blocks; and a link table is established between adjacent nodes of the hierarchical block to record shared facet information.

3. The method according to claim 1, characterized in that Dividing the optimized building model data into a plurality of rendering depth intervals, and setting a shadow map of corresponding resolution for each rendering depth interval; The basic light map and the shadow map are synthesized in real time to obtain scene light and shadow synthesis data, including: Calculating a rendering depth range of the optimized building model data, dividing the rendering depth range into a near-view rendering depth interval, a mid-view rendering depth interval, and a far-view rendering depth interval, wherein a depth value of the near-view rendering depth interval is from 0 meters to 50 meters, a depth value of the mid-view rendering depth interval is from 50 meters to 200 meters, and a depth value of the far-view rendering depth interval is greater than 200 meters; Generate a first-resolution shadow map for the near-view rendering depth interval, generate a second-resolution shadow map for the mid-view rendering depth interval, and generate a third-resolution shadow map for the far-view rendering depth interval, wherein the resolution of the first-resolution shadow map is 2048 by 2048, the resolution of the second-resolution shadow map is 1024 by 1024, and the resolution of the third-resolution shadow map is 512 by 512; Extracting a diffuse lighting component from the base lightmap; performing hardware filtering on the diffuse lighting component by applying a four-by-four pixel percentage proximity filtering sampling method; calculating a depth offset value based on a normal vector of the optimized building model data, and correcting the diffuse lighting component using the depth offset value; Constructing a first transition zone at a boundary between the near rendering depth interval and the mid rendering depth interval, and constructing a second transition zone at a boundary between the mid rendering depth interval and the far rendering depth interval; performing a first bilinear interpolation on the first-resolution shadow map and the second-resolution shadow map within the first transition zone, and performing a second bilinear interpolation on the second-resolution shadow map and the third-resolution shadow map within the second transition zone; Constructing a deferred rendering buffer, storing position data, normal data, and depth data of the optimized building model data in the deferred rendering buffer; extracting the position data from the deferred rendering buffer, and determining rendering depth intervals to which respective parts of the scene belong based on the position data; calculating a scene change value for the rendering depth interval, and setting a shadow map update frequency based on the scene change value; asynchronously updating the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map according to the shadow map update frequency; When the viewpoint position moves, the dividing boundary of the rendering depth interval is recalculated, and the dividing boundary is filtered by applying a time domain smoothing coefficient; when the rendering depth interval is switched, cross-gradient synthesis is performed on the shadow maps of adjacent rendering depth intervals; the diffuse reflection lighting component of the basic light map is synthesized with the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map in real time to generate scene light and shadow synthesis data.

4. The method according to claim 1, wherein Calculating direct lighting and indirect lighting using a physically based rendering method, fusing the lighting calculation results with the scene light and shadow synthesis data to generate dynamic lighting rendering data; Constructing a deferred rendering pipeline based on the dynamic lighting rendering data, and performing lighting processing on the visible pixels of the current viewpoint includes: Build a physics-based lighting calculation model to decompose lighting information into direct lighting and indirect lighting; construct a ray tracing path from the camera viewpoint, record the intersection position of the light and the scene objects, surface normals, and material properties; calculate the light source visibility based on the intersection position, calculate the light incidence angle using the surface normal, and calculate the direct lighting contribution value based on the material properties; A secondary light is emitted at the intersection position, and an indirect lighting contribution value is calculated using Monte Carlo integration; the indirect lighting contribution value is decomposed into a diffuse reflection component and a specular reflection component according to the material properties; the direction of the secondary light is determined using importance sampling, and ambient lighting information is collected along the direction of the secondary light; The direct illumination contribution value and the indirect illumination contribution value are weightedly superimposed to obtain an illumination calculation result; the illumination calculation result is temporally accumulated, and temporal jitter is eliminated using an exponentially weighted average method; a mixing weight coefficient is calculated based on the material properties, and the illumination calculation result is fused with the scene light and shadow synthesis data according to the mixing weight coefficient to generate dynamic illumination rendering data; Writing the position information, normal information, depth information, and material information of the scene geometry into a deferred rendering buffer; constructing a dynamic light map containing the dynamic lighting rendering data, and binding the dynamic light map to the deferred rendering buffer; The depth information in the deferred rendering buffer is analyzed to determine visible pixels of the current viewpoint; a spatial relationship between the visible pixels and a light source is calculated, and lighting processing is performed on the visible pixels based on the spatial relationship.

5. The method according to claim 1, wherein Collecting spatial position and posture data of a virtual reality head display in real time, and determining current viewpoint parameters based on the spatial position and posture data; Performing frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; Performing GPU instanced rendering on the visible building data to generate an initial rendered image includes: Acquire posture data and spatial position data of a virtual reality head display; perform posture fusion on the posture data using an extended Kalman filter to obtain a head display posture quaternion; process the spatial position data based on a SLAM algorithm to obtain head display spatial coordinates; and combine the head display posture quaternion with the head display spatial coordinates to generate a head display spatial pose matrix; Extracting head-mounted display position data as a viewpoint position according to the head-mounted display spatial pose matrix; converting the head-mounted display pose quaternion into a line of sight direction vector; constructing a view cone equation based on the viewpoint position and the line of sight direction vector; generating a projection matrix based on the head-mounted display field of view angle; and combining the view cone equation with the projection matrix to form a viewpoint culling parameter; Constructing a hierarchical bounding box for the buildings in the scene; performing an intersection check on the hierarchical bounding box and the viewpoint culling parameter to obtain a set of candidate buildings that intersect with the view frustum equation; calculating the depth values ​​of the buildings in the candidate building set relative to the viewpoint position; constructing an occlusion map based on the depth values, culling completely occluded buildings, and obtaining a set of visible buildings; Classifying the visible building set according to building type to obtain building type groups; extracting shared geometric data and material data for each building type group; uploading the geometric data and material data to a GPU storage buffer; calculating an instanced transformation matrix for each building based on the head-mounted display spatial pose matrix; and organizing the instanced transformation matrix into an instanced data buffer; The method comprises the steps of: reading the geometric data and the material data from the GPU storage buffer; applying the instantiated transformation matrix in the instantiated data buffer to the geometric data; rendering the set of visible buildings in parallel using a GPU instantiated rendering pipeline; and synthesizing the rendering result with a background image to generate an initial rendered image of the virtual reality scene.

6. A virtual reality architectural roaming system based on scene rendering, used to implement the method according to any one of claims 1 to 5, characterized in that: include: The first unit is configured to collect three-dimensional building model data, divide the three-dimensional building model data into a plurality of hierarchical blocks using an octree space segmentation algorithm, calculate relative viewpoint distances for the hierarchical blocks, and determine a building detail level threshold based on the relative viewpoint distances; construct a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing near and far level information; calculate illumination distribution parameters of the optimized building model data using a pre-calculated irradiance transfer technique, and bake the illumination distribution parameters to generate a basic light map; A second unit is configured to divide the optimized building model data into a plurality of rendering depth intervals, and set a shadow map of corresponding resolution for each rendering depth interval; Synthesize the basic light map and the shadow map in real time to obtain scene light and shadow synthesis data; Calculating direct lighting and indirect lighting using a physically based rendering method, fusing the lighting calculation results with the scene light and shadow synthesis data to generate dynamic lighting rendering data; Constructing a deferred rendering pipeline based on the dynamic lighting rendering data to perform lighting processing on pixels visible from the current viewpoint; The third unit is used to collect spatial position and posture data of the virtual reality head display in real time, and determine the current viewpoint parameters according to the spatial position and posture data; Performing frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; GPU instanced rendering is performed on the visible building data to generate an initial rendered image; the initial rendered image is input into a high dynamic range processing unit, and an adaptive exposure adjustment algorithm is used to optimize the light and dark contrast of the scene to obtain an optimized rendered image; a temporal anti-aliasing technique is applied to the optimized rendered image to eliminate geometric edges and temporal domain flicker, and a final scene image is generated in combination with an image space-based atmospheric scattering algorithm.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images