Virtual reality building roaming method and system based on scene rendering

Abstract

The invention provides a virtual reality building roaming method and system based on scene rendering, and relates to the technical field of virtual reality, and the method comprises the steps: dividing building three-dimensional model data through an octree space segmentation algorithm, and determining a detail level threshold according to a viewpoint distance, and constructing a multi-level detail model. And baking the basic illumination chartlet by using a pre-calculation irradiance transmission technology, and performing real-time synthesis in combination with the multi-resolution shadow chartlet. And calculating illumination by adopting a physics-based rendering method, and fusing the illumination with the scene light and shadow synthesis data to generate dynamic illumination rendering data. And determining a current viewpoint based on the spatial position and attitude data of the virtual reality head-mounted display, performing view cone elimination and shielding elimination, and performing GPU instantiation rendering on visible building data. And the rendered image is optimized through high dynamic range processing and a time domain anti-aliasing technology, and a final scene image is generated in combination with an atmospheric scattering algorithm. According to the invention, the rendering efficiency and scene reality sense of virtual reality building roaming can be effectively improved.

Description

Technical Field

[0001] The present invention relates to virtual reality technology, and in particular to a virtual reality building roaming method and system based on scene rendering. Background Art

[0002] Virtual reality technology is increasingly used in the field of architecture, providing an immersive roaming experience that allows users to feel as if they are in a real architectural scene. Traditional virtual reality architectural roaming methods usually render by directly loading complete architectural models, which brings huge computing pressure to complex and large architectural models, resulting in a drop in rendering frame rate and affecting user experience. To solve this problem, researchers have proposed various optimization strategies, such as model simplification, frustum culling, occlusion culling and other technologies. These technologies can improve rendering efficiency to a certain extent, but there are still some limitations.

[0003] Insufficient model detail level processing: Although the traditional LOD (Level of Detail) technology can simplify the model according to the distance, the switching process is relatively abrupt, and it is easy to have obvious visual jumps, which affects the sense of immersion. Moreover, it is difficult to achieve fine level division for the complex details inside the building, resulting in the loss of nearby details or over-rendering of distant details. Poor lighting rendering effect: Traditional real-time rendering methods are difficult to balance lighting quality and rendering efficiency. Although pre-baked light maps can increase the rendering speed, they cannot show dynamic light and shadow effects. Real-time lighting calculations are computationally intensive and difficult to apply to complex architectural scenes. Lack of optimization for virtual reality devices: Traditional rendering pipelines are mainly designed for desktop monitors, and do not fully consider the characteristics of virtual reality devices, such as high refresh rate, low latency, head tracking, etc. This will cause problems such as screen freezes and delays in virtual reality scenes, affecting user experience. Summary of the invention

[0004] The embodiments of the present invention provide a virtual reality building roaming method and system based on scene rendering, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention,

[0006] Provides a virtual reality building roaming method based on scene rendering, including:

[0007] Collecting building three-dimensional model data, dividing the building three-dimensional model data into multiple hierarchical blocks by using an octree space segmentation algorithm; calculating the viewpoint relative distance for the hierarchical blocks, and determining the building detail level threshold according to the viewpoint relative distance; constructing a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing near and far level information; calculating the illumination distribution parameters of the optimized building model data by using a pre-calculated irradiance transmission technology, and baking the illumination distribution parameters to generate a basic illumination map;

[0008] Divide a plurality of rendering depth intervals based on the optimized building model data, and set a shadow map of corresponding resolution for each rendering depth interval; synthesize the basic light map and the shadow map in real time to obtain scene light and shadow synthesis data; calculate direct lighting and indirect lighting using a physically based rendering method to obtain a lighting calculation result; merge the lighting calculation result with the scene light and shadow synthesis data to generate dynamic lighting rendering data; construct a deferred rendering pipeline based on the dynamic lighting rendering data to perform lighting processing on pixels visible from the current viewpoint;

[0009] The spatial position and posture data of the virtual reality head display are collected in real time, and the current viewpoint parameters are determined according to the spatial position and posture data; the view frustum culling and occlusion culling are performed based on the current viewpoint parameters to screen out the visible building data; the visible building data is subjected to GPU instantiation rendering to generate an initial rendered image; the initial rendered image is input into a high dynamic range processing unit, and an adaptive exposure adjustment algorithm is used to optimize the scene light and dark contrast to obtain an optimized rendered image; the temporal anti-aliasing technology is applied to the optimized rendered image to eliminate geometric edges and temporal domain flicker, and a final scene image is generated in combination with an atmospheric scattering algorithm based on image space.

[0010] Collecting building three-dimensional model data, dividing the building three-dimensional model data into multiple hierarchical blocks by using an octree space segmentation algorithm; calculating viewpoint relative distances for the hierarchical blocks, and determining building detail level thresholds according to the viewpoint relative distances, including:

[0011] A ground laser scanner is used to obtain first point cloud data ranging from the bottom of the building to a height of 20 meters, with a scanning resolution of 1.6 millimeters per 10 meters; a drone is used for oblique photogrammetry to obtain second point cloud data from a height of 20 meters to the roof structure, with a flight altitude of 30 meters from the top of the building; feature points are extracted from the first point cloud data and the second point cloud data, a registration matrix is ​​calculated based on the feature points, and the first point cloud data and the second point cloud data are registered and fused to obtain the three-dimensional model data of the building;

[0012] A scene bounding box is constructed for the building three-dimensional model data, the scene bounding box is used as a root node, and the diagonal length of the scene bounding box is calculated as a reference scale; the face density of the root node is calculated according to the building three-dimensional model data, and when the face density is greater than a preset density threshold and the node size is greater than one sixteenth of the reference scale, the optimal segmentation plane is determined by principal component analysis; the building three-dimensional model data is spatially segmented based on the optimal segmentation plane to obtain a plurality of child nodes; the centroid position of the triangular facets crossing the optimal segmentation plane is calculated, and the triangular facets are assigned to corresponding child nodes to form a plurality of hierarchical blocks; a link table is established between adjacent nodes of the hierarchical block to record shared facet information;

[0013] The hierarchical blocks are discretized into voxel grids, and the shortest distance from each voxel in the voxel grid to the data surface of the building three-dimensional model is calculated to form a distance field; the distance field is compressed and stored using a third-order spherical harmonic function to obtain spherical harmonic coefficients; the spherical harmonic coefficients of the corresponding voxels are queried according to the viewpoint position, and the viewpoint relative distance is reconstructed by the spherical harmonic coefficients;

[0014] An architectural detail level threshold is set for the hierarchical block based on the viewpoint relative distance. When the viewpoint relative distance is less than or equal to a first distance threshold, the architectural detail level threshold is set to the first threshold and the original precision of the hierarchical block is maintained; when the viewpoint relative distance is greater than the first distance threshold and less than or equal to a second distance threshold, the architectural detail level threshold is set to the second threshold, and the hierarchical block is simplified by applying an edge collapse algorithm; when the viewpoint relative distance is greater than the second distance threshold and less than or equal to a third distance threshold, the architectural detail level threshold is set to the third threshold, and coplanar triangles are merged for the hierarchical block; when the viewpoint relative distance is greater than the third distance threshold, the architectural detail level threshold is set to the fourth threshold, and the hierarchical block is simplified with contour preservation.

[0015] Based on the building detail level threshold, a multi-level detail model is constructed to obtain optimized building model data containing near and far level information; using the pre-calculated irradiance transmission technology to calculate the illumination distribution parameters of the optimized building model data, and baking the illumination distribution parameters to generate a basic illumination map, including:

[0016] According to the building detail level threshold, vertex curvature values ​​and boundary weights are respectively calculated, and a comprehensive error metric function is constructed with the vertex curvature values ​​and the boundary weights; based on the comprehensive error metric function, edge collapse operations, coplanar merging operations, and contour preservation operations are performed on the building three-dimensional model data to generate simplified mesh models with different precisions;

[0017] Establishing an octree spatial structure, storing the normal information of the building three-dimensional model data in the octree spatial structure; mapping the normal information to the simplified mesh model to generate a multi-level detail model with normal information;

[0018] Constructing a texture space parameterized grid, calculating the triangle area and the stretching coefficient based on the multi-level detail model with normal information; constructing a texture distortion energy function by multiplying the triangle area and the stretching coefficient; converging the texture distortion energy function by iteratively optimizing UV coordinates, and generating optimized building model data;

[0019] Expanding incident illumination and bidirectional reflectance distribution functions into spherical harmonic basis functions; sampling multiple incident directions for the vertices of the optimized building model data; calculating the integral of the visibility function and the spherical harmonic basis functions to obtain a transmission matrix; and calculating illumination distribution parameters based on the transmission matrix;

[0020] Construct a uniformly distributed sampling point grid, calculate irradiance values ​​for the sampling points of the sampling point grid; use radial basis function to spatially interpolate the irradiance values; divide the UV space based on the illumination distribution parameters and perform texture optimization; divide the UV space into pixel blocks of fixed size; extract the primary and secondary illumination directions of the pixel blocks, use principal component analysis to compress them, and perform anisotropic filtering to generate a basic light map.

[0021] Dividing a plurality of rendering depth intervals based on the optimized building model data, setting a shadow map of corresponding resolution for each rendering depth interval; synthesizing the basic light map and the shadow map in real time to obtain scene light and shadow synthesis data includes:

[0022] Calculating a rendering depth range of the optimized building model data, dividing the rendering depth range into a near-view rendering depth interval, a mid-view rendering depth interval, and a far-view rendering depth interval, wherein a depth value of the near-view rendering depth interval is from zero to fifty meters, a depth value of the mid-view rendering depth interval is from fifty to two hundred meters, and a depth value of the far-view rendering depth interval is greater than two hundred meters;

[0023] Generate a first-resolution shadow map for the near-view rendering depth interval, generate a second-resolution shadow map for the mid-view rendering depth interval, and generate a third-resolution shadow map for the far-view rendering depth interval, wherein the resolution of the first-resolution shadow map is 2048 by 2048, the resolution of the second-resolution shadow map is 1024 by 1024, and the resolution of the third-resolution shadow map is 512 by 512;

[0024] Extracting a diffuse lighting component from the basic light map; applying a four-by-four pixel percentage proximity filter sampling to the diffuse lighting component for hardware filtering; calculating a depth offset value based on a normal vector of the optimized building model data, and correcting the diffuse lighting component using the depth offset value;

[0025] A first transition zone is constructed at a boundary between the near-view rendering depth interval and the mid-view rendering depth interval, and a second transition zone is constructed at a boundary between the mid-view rendering depth interval and the far-view rendering depth interval; a first bilinear interpolation is performed on the first-resolution shadow map and the second-resolution shadow map within the first transition zone, and a second bilinear interpolation is performed on the second-resolution shadow map and the third-resolution shadow map within the second transition zone;

[0026] Constructing a deferred rendering buffer, storing the position data, normal data, and depth data of the optimized building model data in the deferred rendering buffer; extracting the position data from the deferred rendering buffer, and determining the rendering depth intervals to which each part of the scene belongs according to the position data; calculating a scene change value of the rendering depth interval, and setting a shadow map update frequency based on the scene change value; asynchronously updating the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map according to the shadow map update frequency;

[0027] When the viewpoint position moves, the dividing boundary of the rendering depth interval is recalculated, and the dividing boundary is filtered by applying a time domain smoothing coefficient; when the rendering depth interval is switched, a cross-gradient synthesis is performed on the shadow maps of adjacent rendering depth intervals; the diffuse reflection lighting component of the basic light map is synthesized in real time with the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map to generate scene light and shadow synthesis data.

[0028] Calculating direct illumination and indirect illumination using a physically based rendering method to obtain illumination calculation results, fusing the illumination calculation results with the scene light and shadow synthesis data to generate dynamic illumination rendering data; constructing a deferred rendering pipeline based on the dynamic illumination rendering data, and performing illumination processing on pixels visible at the current viewpoint includes:

[0029] Construct a physically based lighting calculation model to decompose lighting information into direct lighting and indirect lighting; construct a ray tracing path from the camera viewpoint, record the intersection position of the light and the scene object, the surface normal, and the material properties; calculate the light source visibility based on the intersection position, calculate the lighting incident angle using the surface normal, and calculate the direct lighting contribution value in combination with the material properties;

[0030] A secondary light is emitted to the intersection position, and an indirect lighting contribution value is calculated by Monte Carlo integration; the indirect lighting contribution value is decomposed into a diffuse reflection component and a specular reflection component according to the material properties; the direction of the secondary light is determined by importance sampling, and the ambient lighting information is collected along the direction of the secondary light;

[0031] The direct illumination contribution value and the indirect illumination contribution value are weightedly superimposed to obtain an illumination calculation result; the illumination calculation result is accumulated in the time domain, and the time domain jitter is eliminated by using an exponential weighted average method; a mixing weight coefficient is calculated based on the material properties, and the illumination calculation result is merged with the scene light and shadow synthesis data according to the mixing weight coefficient to generate dynamic illumination rendering data;

[0032] Writing the position information, normal information, depth information, and material information of the scene geometry into the deferred rendering buffer; constructing a dynamic light map containing the dynamic lighting rendering data, and binding the dynamic light map to the deferred rendering buffer;

[0033] The depth information in the deferred rendering buffer is analyzed to determine visible pixels of the current viewpoint; a spatial relationship between the visible pixels and a light source is calculated, and illumination processing is performed on the visible pixels based on the spatial relationship.

[0034] Collecting the spatial position and posture data of the virtual reality head display in real time, determining the current viewpoint parameters according to the spatial position and posture data; performing frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; performing GPU instantiation rendering on the visible building data to generate an initial rendered image includes:

[0035] Acquire posture data and spatial position data of a virtual reality head display; use an extended Kalman filter to perform posture fusion on the posture data to obtain a head display posture quaternion; process the spatial position data based on a real-time positioning and map building algorithm to obtain a head display spatial coordinate; combine the head display posture quaternion with the head display spatial coordinate to generate a head display spatial position matrix;

[0036] Extracting the head display position data as the viewpoint position according to the head display spatial posture matrix; converting the head display posture quaternion into a sight direction vector; constructing a view cone equation based on the viewpoint position and the sight direction vector; generating a projection matrix in combination with the head display field of view angle; combining the view cone equation with the projection matrix to form a viewpoint culling parameter;

[0037] Constructing a hierarchical bounding box for the buildings in the scene; performing intersection detection on the hierarchical bounding box and the viewpoint culling parameter to obtain a set of candidate buildings that intersect with the view frustum equation; calculating the depth values ​​of the buildings in the candidate building set relative to the viewpoint position; constructing an occlusion map based on the depth values, culling completely occluded buildings, and obtaining a set of visible buildings;

[0038] Classifying the visible building set according to building type to obtain building type groups; extracting shared geometry data and material data for each building type group; uploading the geometry data and the material data to a GPU storage buffer; calculating an instantiation transformation matrix for each building according to the head display spatial pose matrix; and organizing the instantiation transformation matrix into an instantiation data buffer;

[0039] The geometric data and the material data are read from the GPU storage buffer; the instantiated transformation matrix in the instantiated data buffer is applied to the geometric data; the visible building set is rendered in parallel using the GPU instantiated rendering pipeline; and the rendering result is synthesized with the background image to generate an initial rendered image of the virtual reality scene.

[0040] According to a second aspect of the embodiments of the present invention,

[0041] Provides a virtual reality architectural roaming system based on scene rendering, including:

[0042] The first unit is used to collect building three-dimensional model data, divide the building three-dimensional model data into multiple hierarchical blocks by using an octree space segmentation algorithm; calculate the viewpoint relative distance for the hierarchical blocks, and determine the building detail level threshold according to the viewpoint relative distance; construct a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing far and near level information; calculate the illumination distribution parameters of the optimized building model data by using a pre-calculated irradiance transmission technology, and bake the illumination distribution parameters to generate a basic illumination map;

[0043] The second unit is used to divide a plurality of rendering depth intervals based on the optimized building model data, and set a shadow map of corresponding resolution for each rendering depth interval; synthesize the basic light map and the shadow map in real time to obtain scene light and shadow synthesis data; calculate direct lighting and indirect lighting using a physically based rendering method, merge the lighting calculation results with the scene light and shadow synthesis data, and generate dynamic lighting rendering data; construct a deferred rendering pipeline according to the dynamic lighting rendering data, and perform lighting processing on visible pixels of the current viewpoint;

[0044] The third unit is used to collect the spatial position and posture data of the virtual reality head display in real time, determine the current viewpoint parameters according to the spatial position and posture data; perform frustum culling and occlusion culling based on the current viewpoint parameters to screen out visible building data; perform GPU instantiation rendering on the visible building data to generate an initial rendered image; input the initial rendered image into a high dynamic range processing unit, use an adaptive exposure adjustment algorithm to optimize the scene light and dark contrast, and obtain an optimized rendered image; apply the temporal anti-aliasing technology to the optimized rendered image to eliminate geometric edges and temporal domain flicker, and generate a final scene image in combination with an atmospheric scattering algorithm based on image space.

[0045] According to a third aspect of the embodiments of the present invention,

[0046] An electronic device is provided, comprising:

[0047] processor;

[0048] a memory for storing processor-executable instructions;

[0049] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0050] A fourth aspect of the embodiments of the present invention is:

[0051] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0052] The beneficial effects of this application are as follows:

[0053] 1. Improve rendering efficiency and performance: Through technologies such as octree space segmentation, multi-level detail model and frustum / occlusion culling, the amount of rendering calculations is effectively reduced, and fast loading and smooth roaming of architectural scenes are achieved, which is especially suitable for real-time rendering needs in virtual reality environments.

[0054] 2. Enhance the realism and immersion of the scene: Using technologies such as pre-calculated irradiance transmission, physically based rendering, dynamic lighting rendering, high dynamic range processing and atmospheric scattering, the light and shadow effects, material texture and spatial atmosphere of the building are realistically simulated, enhancing the realism and immersion of the virtual reality scene.

[0055] 3. Optimize visual experience and comfort: Utilizing technologies such as multi-resolution settings of shadow maps, adaptive exposure adjustment, and temporal anti-aliasing, it effectively solves common problems such as aliasing, flickering, and imbalance of light and dark contrast in virtual reality scenes, thereby optimizing the user's visual experience and comfort. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1A flowchart of a virtual reality building roaming method based on scene rendering according to an embodiment of the present invention;

[0057] Figure 2 It is a structural schematic diagram of a virtual reality building roaming system based on scene rendering according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0060] Figure 1 FIG. 1 is a flow chart of a virtual reality building roaming method based on scene rendering according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0061] S11. Collecting building three-dimensional model data, dividing the building three-dimensional model data into multiple hierarchical blocks by using an octree space segmentation algorithm; calculating the viewpoint relative distance for the hierarchical blocks, and determining the building detail level threshold according to the viewpoint relative distance; constructing a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing near and far level information; calculating the illumination distribution parameters of the optimized building model data by using a pre-calculated irradiance transmission technology, and baking the illumination distribution parameters to generate a basic illumination map;

[0062] S12. Divide the optimized building model data into multiple rendering depth intervals, set a shadow map of corresponding resolution for each rendering depth interval; synthesize the basic light map and the shadow map in real time to obtain scene light and shadow synthesis data; calculate direct lighting and indirect lighting using a physically based rendering method to obtain a lighting calculation result; merge the lighting calculation result with the scene light and shadow synthesis data to generate dynamic lighting rendering data; construct a deferred rendering pipeline according to the dynamic lighting rendering data, and perform lighting processing on visible pixels of the current viewpoint;

[0063] S13. Collect the spatial position and posture data of the virtual reality headset in real time, and determine the current viewpoint parameters according to the spatial position and posture data; perform frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; perform GPU instantiation rendering on the visible building data to generate an initial rendered image; input the initial rendered image into a high dynamic range processing unit, and use an adaptive exposure adjustment algorithm to optimize the scene light and dark contrast to obtain an optimized rendered image; apply temporal anti-aliasing technology to the optimized rendered image to eliminate geometric edges and temporal domain flicker, and generate a final scene image in combination with an atmospheric scattering algorithm based on image space.

[0064] In an optional implementation, collecting building 3D model data, dividing the building 3D model data into a plurality of hierarchical blocks by using an octree space segmentation algorithm; calculating viewpoint relative distances for the hierarchical blocks, and determining building detail level thresholds according to the viewpoint relative distances comprises:

[0065] A ground laser scanner is used to obtain first point cloud data ranging from the bottom of the building to a height of 20 meters, with a scanning resolution of 1.6 millimeters per 10 meters; a drone is used for oblique photogrammetry to obtain second point cloud data from a height of 20 meters to the roof structure, with a flight altitude of 30 meters from the top of the building; feature points are extracted from the first point cloud data and the second point cloud data, a registration matrix is ​​calculated based on the feature points, and the first point cloud data and the second point cloud data are registered and fused to obtain the three-dimensional model data of the building;

[0066] A scene bounding box is constructed for the building three-dimensional model data, the scene bounding box is used as a root node, and the diagonal length of the scene bounding box is calculated as a reference scale; the face density of the root node is calculated according to the building three-dimensional model data, and when the face density is greater than a preset density threshold and the node size is greater than one sixteenth of the reference scale, the optimal segmentation plane is determined by principal component analysis; the building three-dimensional model data is spatially segmented based on the optimal segmentation plane to obtain a plurality of child nodes; the centroid position of the triangular facets crossing the optimal segmentation plane is calculated, and the triangular facets are assigned to corresponding child nodes to form a plurality of hierarchical blocks; a link table is established between adjacent nodes of the hierarchical block to record shared facet information;

[0067] The hierarchical blocks are discretized into voxel grids, and the shortest distance from each voxel in the voxel grid to the data surface of the building three-dimensional model is calculated to form a distance field; the distance field is compressed and stored using a third-order spherical harmonic function to obtain spherical harmonic coefficients; the spherical harmonic coefficients of the corresponding voxels are queried according to the viewpoint position, and the viewpoint relative distance is reconstructed by the spherical harmonic coefficients;

[0068] An architectural detail level threshold is set for the hierarchical block based on the viewpoint relative distance. When the viewpoint relative distance is less than or equal to a first distance threshold, the architectural detail level threshold is set to the first threshold and the original precision of the hierarchical block is maintained; when the viewpoint relative distance is greater than the first distance threshold and less than or equal to a second distance threshold, the architectural detail level threshold is set to the second threshold, and the hierarchical block is simplified by applying an edge collapse algorithm; when the viewpoint relative distance is greater than the second distance threshold and less than or equal to a third distance threshold, the architectural detail level threshold is set to the third threshold, and coplanar triangles are merged for the hierarchical block; when the viewpoint relative distance is greater than the third distance threshold, the architectural detail level threshold is set to the fourth threshold, and the hierarchical block is simplified with contour preservation.

[0069] Collect building 3D model data and divide it into hierarchical blocks using the octree space segmentation algorithm. Dynamically adjust the building detail level according to the relative distance of the viewpoint to achieve efficient 3D model rendering.

[0070] First, a terrestrial laser scanner was used to obtain point cloud data from the bottom of the building to a height of 20 meters. The scanning resolution was set to 1.6 mm per 10 meters to ensure high-precision data. At the same time, drone oblique photogrammetry was used to obtain point cloud data from a height of more than 20 meters to the roof structure. The flight altitude was set to 30 meters from the top of the building to ensure data integrity. For example, the terrestrial laser scanning obtained detailed data such as the pillars and walls at the bottom of the building, and the drone oblique photogrammetry obtained data on high-rise structures such as the roof and chimney.

[0071] Then, feature points are extracted from the point cloud data obtained by ground laser scanning and drone oblique photogrammetry. For example, obvious features such as building corners and edge points are extracted. The registration matrix is ​​calculated based on these feature points, and the two sets of point cloud data are registered and fused. The registration process can use methods such as iterative closest point algorithm. Finally, the complete building 3D model data is obtained. For example, the column data obtained by ground laser scanning is accurately aligned with the roof data obtained by drone oblique photogrammetry to form a complete building model.

[0072] Next, a scene bounding box is constructed for the building 3D model data. The scene bounding box is used as the root node of the octree. The diagonal length of the scene bounding box is calculated as the reference scale. For example, for a building that is 100 meters long, 50 meters wide, and 30 meters high, the diagonal length of its scene bounding box is about 114 meters, which is used as the reference scale for subsequent segmentation. Then, the face density of the root node is calculated, that is, the number of triangular facets per unit volume. If the face density is greater than the preset density threshold (for example, 100 facets per cubic meter) and the node size is greater than one sixteenth of the reference scale (for example, 114 / 16 = 7.125 meters), principal component analysis is used to determine the optimal segmentation plane. The principal component analysis method can find the direction with the largest data variance as the normal vector of the segmentation plane. The space of the current node is divided into eight child nodes based on the optimal segmentation plane. The centroid position of the triangular facets that cross the optimal segmentation plane is calculated, and these triangular facets are assigned to the corresponding child nodes to form multiple hierarchical blocks. A link table is established between adjacent nodes to record shared facet information for subsequent processing.

[0073] After that, the hierarchical blocks are discretized into voxel grids. For example, each block is divided into voxels of 1 cubic meter. The shortest distance to the data surface of the building 3D model is calculated for each voxel in the voxel grid to form a distance field. The distance field is compressed and stored using the third-order spherical harmonic function to obtain the spherical harmonic coefficients. The spherical harmonic function can represent the distance field as a combination of a series of coefficients, thereby reducing the storage space. The spherical harmonic coefficients of the corresponding voxel are queried according to the viewpoint position, and the relative distance of the viewpoint is reconstructed through the spherical harmonic coefficients.

[0074] Finally, the architectural detail level threshold is set for the hierarchical block based on the relative distance of the viewpoint. For example, three distance thresholds are set: the first distance threshold is 10 meters, the second distance threshold is 50 meters, and the third distance threshold is 100 meters. When the relative distance of the viewpoint is less than or equal to 10 meters, the architectural detail level threshold is set to the first threshold (for example, level 0), and the original accuracy of the hierarchical block is maintained. When the relative distance of the viewpoint is greater than 10 meters and less than or equal to 50 meters, the architectural detail level threshold is set to the second threshold (for example, level 1), and the edge collapse algorithm is applied to the hierarchical block for simplification. When the relative distance of the viewpoint is greater than 50 meters and less than or equal to 100 meters, the architectural detail level threshold is set to the third threshold (for example, level 2), and the coplanar triangles of the hierarchical block are merged. When the relative distance of the viewpoint is greater than 100 meters, the architectural detail level threshold is set to the fourth threshold (for example, level 3), and the hierarchical block is simplified with contour preservation.

[0075] The solution of this application can:

[0076] Improve rendering efficiency: By dynamically adjusting the level of building details according to the viewpoint distance, the amount of data to be rendered is reduced, thereby improving rendering efficiency. Reduce storage space: Use spherical harmonics to compress and store distance fields, reducing storage space requirements. Improve user experience: While ensuring the details of nearby buildings, the distant building models are simplified, achieving efficient scene rendering and improving user experience.

[0077] In an optional implementation, constructing a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing near and far level information; calculating the illumination distribution parameters of the optimized building model data using a pre-calculated irradiance transmission technology, and baking the illumination distribution parameters to generate a basic illumination map includes:

[0078] According to the building detail level threshold, vertex curvature values ​​and boundary weights are respectively calculated, and a comprehensive error metric function is constructed with the vertex curvature values ​​and the boundary weights; based on the comprehensive error metric function, edge collapse operations, coplanar merging operations, and contour preservation operations are performed on the building three-dimensional model data to generate simplified mesh models with different precisions;

[0079] Establishing an octree spatial structure, storing the normal information of the building three-dimensional model data in the octree spatial structure; mapping the normal information to the simplified mesh model to generate a multi-level detail model with normal information;

[0080] Constructing a texture space parameterized grid, calculating the triangle area and the stretching coefficient based on the multi-level detail model with normal information; constructing a texture distortion energy function by multiplying the triangle area and the stretching coefficient; converging the texture distortion energy function by iteratively optimizing UV coordinates, and generating optimized building model data;

[0081] Expanding incident illumination and bidirectional reflectance distribution functions into spherical harmonic basis functions; sampling multiple incident directions for the vertices of the optimized building model data; calculating the integral of the visibility function and the spherical harmonic basis functions to obtain a transmission matrix; and calculating illumination distribution parameters based on the transmission matrix;

[0082] Construct a uniformly distributed sampling point grid, calculate irradiance values ​​for the sampling points of the sampling point grid; use radial basis function to spatially interpolate the irradiance values; divide the UV space based on the illumination distribution parameters and perform texture optimization; divide the UV space into pixel blocks of fixed size; extract the primary and secondary illumination directions of the pixel blocks, use principal component analysis to compress them, and perform anisotropic filtering to generate a basic light map.

[0083] A method for generating basic lightmaps for building models can effectively improve rendering efficiency and visual effects. The core idea of ​​this method is to build a multi-level detail model based on the building detail level threshold, calculate the illumination distribution parameters using pre-calculated irradiance transfer technology, and finally bake and generate basic lightmaps.

[0084] First, set the building detail level threshold. For example, set the threshold to 0.1. This means that details in the model that are smaller than this threshold will be simplified.

[0085] Next, the curvature value and boundary weight of each vertex in the model are calculated. The vertex curvature value is used to measure the degree of geometric shape change around the vertex. For example, vertices with larger curvature values ​​are usually located at the edge or corner of the model. The boundary weight is used to measure the probability that the vertex is on the boundary of the model. For example, vertices located on the outer contour of the model have higher boundary weights. These two values ​​are combined into a comprehensive error metric function. An example function is to perform a weighted sum of the curvature value and the boundary weight, where the weight can be adjusted according to actual needs, such as setting the curvature weight to 0.7 and the boundary weight to 0.3.

[0086] Based on the comprehensive error metric function, edge collapse, coplanar merging and contour preservation operations are performed on the building 3D model data to generate simplified mesh models with different precisions. For example, three models with different precisions can be generated, retaining 90%, 50% and 10% of the triangle faces of the original model respectively. The edge collapse operation merges an edge and its two vertices into a new vertex, the coplanar merging operation merges multiple faces on the same plane into one face, and the contour preservation operation tries to retain the original contour of the model during the simplification process.

[0087] An octree space structure is established, and the normal information of the original high-precision model is stored in it. An octree is a space partitioning data structure that can recursively divide a three-dimensional space into eight subspaces. The normal information is mapped to the simplified mesh model generated previously. Specifically, for each vertex on the simplified model, the smallest unit containing the vertex is found in the octree, and the normal information stored in the unit is used as the normal of the vertex.

[0088] Construct a texture space parameterized mesh, and calculate the triangle area and stretch coefficient based on a multi-level detail model with normal information. The stretch coefficient is used to measure the degree of stretching or compression of the texture on the model surface. For example, the larger the stretch coefficient, the more severe the texture is stretched in this area. The product of the triangle area and the stretch coefficient is used to construct a texture distortion energy function. The texture distortion energy function is converged by iteratively optimizing the UV coordinates to generate optimized building model data. For example, the UV coordinates are iteratively optimized using the gradient descent method until the change in the energy function value is less than a preset threshold, such as 0.001.

[0089] Expand the incident illumination and bidirectional reflectance distribution functions into spherical harmonic basis functions. Sample multiple incident directions for the vertices of the optimized building model data, for example, sample 6 directions evenly distributed on the hemisphere. Calculate the integral of the visibility function and the spherical harmonic basis function to obtain the transmission matrix. Calculate the illumination distribution parameters based on the transmission matrix.

[0090] Construct a uniformly distributed grid of sampling points and calculate the irradiance values ​​for the sampling points on the grid. Use radial basis functions to spatially interpolate the irradiance values. Divide the UV space based on the illumination distribution parameters and perform texture optimization. Divide the UV space into fixed-size pixel blocks, for example, each pixel block is 8x8 pixels. Extract the primary and secondary illumination directions of each pixel block, compress them using principal component analysis, and perform anisotropic filtering to generate a basic light map.

[0091] The solution of this application can:

[0092] Improve rendering efficiency: Through multi-level detail models and pre-calculated irradiance transmission technology, the amount of calculation for real-time rendering is reduced, thereby improving rendering efficiency. Enhance visual effects: The important details and lighting information of the model are retained, making the generated scene more realistic and the visual effect better. Optimize resource management: The generated basic light map occupies less storage space, reducing the demand for hardware resources.

[0093] In an optional implementation, dividing a plurality of rendering depth intervals based on the optimized building model data, setting a shadow map of corresponding resolution for each rendering depth interval; synthesizing the basic light map and the shadow map in real time to obtain scene light and shadow synthesis data includes:

[0094] Calculating a rendering depth range of the optimized building model data, dividing the rendering depth range into a near-view rendering depth interval, a mid-view rendering depth interval, and a far-view rendering depth interval, wherein a depth value of the near-view rendering depth interval is from zero to fifty meters, a depth value of the mid-view rendering depth interval is from fifty to two hundred meters, and a depth value of the far-view rendering depth interval is greater than two hundred meters;

[0095] Generate a first-resolution shadow map for the near-view rendering depth interval, generate a second-resolution shadow map for the mid-view rendering depth interval, and generate a third-resolution shadow map for the far-view rendering depth interval, wherein the resolution of the first-resolution shadow map is 2048 by 2048, the resolution of the second-resolution shadow map is 1024 by 1024, and the resolution of the third-resolution shadow map is 512 by 512;

[0096] Extracting a diffuse lighting component from the basic light map; applying a four-by-four pixel percentage proximity filter sampling to the diffuse lighting component for hardware filtering; calculating a depth offset value based on a normal vector of the optimized building model data, and correcting the diffuse lighting component using the depth offset value;

[0097] A first transition zone is constructed at a boundary between the near-view rendering depth interval and the mid-view rendering depth interval, and a second transition zone is constructed at a boundary between the mid-view rendering depth interval and the far-view rendering depth interval; a first bilinear interpolation is performed on the first-resolution shadow map and the second-resolution shadow map within the first transition zone, and a second bilinear interpolation is performed on the second-resolution shadow map and the third-resolution shadow map within the second transition zone;

[0098] Constructing a deferred rendering buffer, storing the position data, normal data, and depth data of the optimized building model data in the deferred rendering buffer; extracting the position data from the deferred rendering buffer, and determining the rendering depth intervals to which each part of the scene belongs according to the position data; calculating a scene change value of the rendering depth interval, and setting a shadow map update frequency based on the scene change value; asynchronously updating the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map according to the shadow map update frequency;

[0099] When the viewpoint position moves, the dividing boundary of the rendering depth interval is recalculated, and the dividing boundary is filtered by applying a time domain smoothing coefficient; when the rendering depth interval is switched, a cross-gradient synthesis is performed on the shadow maps of adjacent rendering depth intervals; the diffuse reflection lighting component of the basic light map is synthesized in real time with the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map to generate scene light and shadow synthesis data.

[0100] The shadow map generation and synthesis method for optimizing architectural model rendering is implemented as follows:

[0101] First, preprocess and optimize the building model data. Get the vertex position information of the building model, calculate the distance from each vertex to the viewpoint, and get the rendering depth range of the model. For example, the rendering depth range of the model is 0 to 300 meters. Based on this range, the rendering depth range is divided into multiple intervals, for example, the near-view interval of 0 to 50 meters, the mid-view interval of 50 to 200 meters, and the far-view interval of more than 200 meters.

[0102] Then, different resolution shadow maps are generated for different rendering depth intervals. For the near-field interval, a first resolution shadow map with a resolution of 2048x2048 is generated. For the mid-field interval, a second resolution shadow map with a resolution of 1024x1024 is generated. For the distant-field interval, a third resolution shadow map with a resolution of 512x512 is generated. Shadow maps can be generated using shadow mapping technology, rendering the scene with the light source as the viewpoint and storing depth information in the shadow map.

[0103] Next, the diffuse lighting component is extracted from the pre-generated base lightmap. The extracted diffuse lighting component is hardware filtered using a 4x4 pixel percentage neighbor filtering sampling to smooth the lighting effect. For example, for a pixel, the diffuse lighting component values ​​of its 16 neighboring pixels are 0.1, 0.2, ..., 0.16, respectively. After filtering, the diffuse lighting component value of the pixel is the weighted average of these values.

[0104] The depth offset value is then calculated based on the normal vector of the optimized building model data. The depth offset value is used to correct the filtered diffuse lighting component to avoid artifacts such as shadow acne. For example, if the angle between the normal vector and the light direction is large, the depth offset value is large to avoid the shadow from causing an erroneous offset on the surface of the object.

[0105] A first transition zone is constructed at the boundary of the near and mid-view rendering depth intervals (at 50 meters), for example, a transition zone with a width of 10 meters. A second transition zone is constructed at the boundary of the mid-view and far-view rendering depth intervals (at 200 meters), for example, a transition zone with a width of 20 meters. Bilinear interpolation is performed on the first and second resolution shadow maps within the first transition zone to smooth shadow transitions. Bilinear interpolation is performed on the second and third resolution shadow maps within the second transition zone to smooth shadow transitions.

[0106] Construct a deferred rendering buffer and store the position data, normal data, and depth data of the optimized building model data in the deferred rendering buffer. Extract the position data from the deferred rendering buffer. Determine the rendering depth interval to which each part of the scene belongs based on the position data. Calculate the scene change value of the rendering depth interval. The scene change value can be calculated based on the changes in factors such as the position and lighting of objects in the scene. Set the shadow map update frequency based on the scene change value. For example, if the scene change value is large, increase the shadow map update frequency. Asynchronously update the first resolution, second resolution, and third resolution shadow maps based on the shadow map update frequency.

[0107] When the viewpoint position moves, the boundary of the rendering depth interval is recalculated. The boundary is filtered by applying a temporal smoothing coefficient to avoid drastic changes in the boundary. For example, a low-pass filter is used to smooth the boundary value. When the rendering depth interval is switched, the shadow maps of adjacent rendering depth intervals are cross-faded to avoid sudden shadow changes. Finally, the diffuse lighting component of the basic light map is synthesized in real time with the shadow map of the corresponding resolution to generate scene light and shadow synthesis data.

[0108] The solution of this application can:

[0109] Improve rendering efficiency. By dividing the rendering depth interval according to the distance and using shadow maps of different resolutions, unnecessary calculations can be reduced and rendering efficiency can be improved, especially when dealing with large-scale scenes. Improve shadow quality. Through depth offset correction and bilinear interpolation of transition bands, artifacts such as shadow acne and aliasing at the edges of shadows can be effectively reduced, improving shadow quality and making the scene more realistic. Enhance the dynamic performance of the scene. By dynamically updating the shadow map according to the scene change value and smoothly transitioning the shadow when the viewpoint moves, the dynamic performance of the scene can be made more natural and smooth.

[0110] In an optional implementation, using a physically based rendering method to calculate direct illumination and indirect illumination to obtain illumination calculation results, fusing the illumination calculation results with the scene light and shadow synthesis data to generate dynamic illumination rendering data; constructing a deferred rendering pipeline based on the dynamic illumination rendering data, and performing illumination processing on visible pixels of the current viewpoint includes:

[0111] Construct a physically based lighting calculation model to decompose lighting information into direct lighting and indirect lighting; construct a ray tracing path from the camera viewpoint, record the intersection position of the light and the scene object, the surface normal, and the material properties; calculate the light source visibility based on the intersection position, calculate the lighting incident angle using the surface normal, and calculate the direct lighting contribution value in combination with the material properties;

[0112] A secondary light is emitted to the intersection position, and an indirect lighting contribution value is calculated by Monte Carlo integration; the indirect lighting contribution value is decomposed into a diffuse reflection component and a specular reflection component according to the material properties; the direction of the secondary light is determined by importance sampling, and the ambient lighting information is collected along the direction of the secondary light;

[0113] The direct illumination contribution value and the indirect illumination contribution value are weightedly superimposed to obtain an illumination calculation result; the illumination calculation result is accumulated in the time domain, and the time domain jitter is eliminated by using an exponential weighted average method; a mixing weight coefficient is calculated based on the material properties, and the illumination calculation result is merged with the scene light and shadow synthesis data according to the mixing weight coefficient to generate dynamic illumination rendering data;

[0114] Writing the position information, normal information, depth information, and material information of the scene geometry into the deferred rendering buffer; constructing a dynamic light map containing the dynamic lighting rendering data, and binding the dynamic light map to the deferred rendering buffer;

[0115] The depth information in the deferred rendering buffer is analyzed to determine visible pixels of the current viewpoint; a spatial relationship between the visible pixels and a light source is calculated, and illumination processing is performed on the visible pixels based on the spatial relationship.

[0116] The physically based dynamic lighting rendering method achieves high-quality real-time rendering effects by combining ray tracing and scene light and shadow synthesis data.

[0117] First, perform scene preprocessing. Model all objects in the scene and assign corresponding material properties, including diffuse reflectance, specular reflectance, roughness, etc. For example, for a red plastic ball, the diffuse reflectance can be set to 0.8, the specular reflectance to 0.2, and the roughness to 0.1. At the same time, record the light source information in the scene, including the light source type, position, color, intensity, etc. For example, a point light source has a position of (0,5,0), a color of white, and an intensity of 1000.

[0118] Next, build a physically based lighting calculation model. For each pixel in the scene, a ray tracing path is constructed starting from the camera viewpoint. This path records the intersection information between the light and the scene objects, including the intersection position, surface normal, material properties, etc. For example, the light intersects with the red plastic ball at point (1,1,1), and the surface normal is (0,1,0). Then, the light source visibility is calculated based on the intersection position. If the light source is blocked by other objects, the direct lighting contribution of the pixel is zero. Otherwise, the surface normal and the light source direction are used to calculate the lighting incident angle, and the direct lighting contribution is calculated by combining the material properties and the light source intensity. For example, the light source direction is (0,-1,0), the incident angle is 0 degrees, and the direct lighting contribution is 800.

[0119] At the same time, multiple secondary rays are emitted to the intersection position, and the indirect lighting contribution is calculated using the Monte Carlo integration method. The direction of the secondary light is determined by importance sampling. For example, for a rough surface, the direction of the secondary light is more scattered; for a smooth surface, the direction of the secondary light is more concentrated. Ambient lighting information is collected along the direction of the secondary light, and the indirect lighting contribution is decomposed into diffuse and specular components according to the material properties. For example, the indirect lighting contribution of a red plastic ball is 200, of which the diffuse component is 160 and the specular component is 40.

[0120] The direct illumination contribution value and the indirect illumination contribution value are weighted and superimposed to obtain the illumination calculation result of the pixel. In order to eliminate temporal jitter, the illumination calculation result is accumulated in the temporal domain and the exponential weighted average method is used. For example, if the illumination calculation result of the current frame is 1000, the illumination calculation result of the previous frame is 900, and the weighted average coefficient is 0.9, the final illumination result is 990.

[0121] Then, the lighting calculation results are merged with the pre-calculated scene light and shadow synthesis data. The mixing weight coefficient is calculated based on the material properties. For example, for opaque objects, the mixing weight coefficient is 1, and the lighting calculation results are fully used; for translucent objects, the mixing weight coefficient is less than 1, and the lighting calculation results are mixed with the scene light and shadow synthesis data. Finally, dynamic lighting rendering data is generated.

[0122] Finally, build a deferred rendering pipeline. Write the position information, normal information, depth information, and material information of the scene geometry into the deferred rendering buffer. Build the dynamic lighting rendering data into a dynamic light map and bind it to the deferred rendering buffer. Analyze the depth information in the deferred rendering buffer to determine the visible pixels of the current viewpoint. Calculate the spatial relationship between the visible pixels and the light source, perform lighting processing on the visible pixels based on the spatial relationship, and finally render the image.

[0123] The solution of this application can:

[0124] Higher rendering efficiency: Through deferred rendering technology, lighting calculations can be separated from geometric processing to improve rendering efficiency. More realistic lighting effects: Physically based rendering methods can more accurately simulate the propagation and interaction of light, thereby generating more realistic lighting effects. Stronger artistic expression: By integrating scene light and shadow synthesis data, richer artistic effects can be achieved, such as global illumination, ambient occlusion, etc.

[0125] In an optional implementation, real-time acquisition of spatial position and posture data of a virtual reality head display, determining current viewpoint parameters according to the spatial position and posture data; performing frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; performing GPU instantiation rendering on the visible building data to generate an initial rendered image includes:

[0126] Acquire posture data and spatial position data of a virtual reality head display; use an extended Kalman filter to perform posture fusion on the posture data to obtain a head display posture quaternion; process the spatial position data based on a real-time positioning and map building algorithm to obtain a head display spatial coordinate; combine the head display posture quaternion with the head display spatial coordinate to generate a head display spatial position matrix;

[0127] Extracting the head display position data as the viewpoint position according to the head display spatial posture matrix; converting the head display posture quaternion into a sight direction vector; constructing a view cone equation based on the viewpoint position and the sight direction vector; generating a projection matrix in combination with the head display field of view angle; combining the view cone equation with the projection matrix to form a viewpoint culling parameter;

[0128] Constructing a hierarchical bounding box for the buildings in the scene; performing intersection detection on the hierarchical bounding box and the viewpoint culling parameter to obtain a set of candidate buildings that intersect with the view frustum equation; calculating the depth values ​​of the buildings in the candidate building set relative to the viewpoint position; constructing an occlusion map based on the depth values, culling completely occluded buildings, and obtaining a set of visible buildings;

[0129] Classifying the visible building set according to building type to obtain building type groups; extracting shared geometry data and material data for each building type group; uploading the geometry data and the material data to a GPU storage buffer; calculating an instantiation transformation matrix for each building according to the head display spatial pose matrix; and organizing the instantiation transformation matrix into an instantiation data buffer;

[0130] The geometric data and the material data are read from the GPU storage buffer; the instantiated transformation matrix in the instantiated data buffer is applied to the geometric data; the visible building set is rendered in parallel using the GPU instantiated rendering pipeline; and the rendering result is synthesized with the background image to generate an initial rendered image of the virtual reality scene.

[0131] The virtual reality scene rendering method is used to efficiently render large-scale urban building models and provide an immersive virtual reality experience. The core of this method is to use the spatial positioning and posture information of the virtual reality headset, combined with the frustum culling and occlusion culling technology, to perform GPU instanced rendering of visible buildings, thereby achieving smooth virtual reality scene display.

[0132] First, obtain the posture data and spatial position data of the virtual reality headset. The posture data of the headset contains the rotation angle information of the headset, and the spatial position data describes the three-dimensional coordinates of the headset in the real world. For example, the posture data can be expressed as the angle of rotation around the X, Y, and Z axes, and the spatial position data can be expressed as the X, Y, and Z coordinate values, such as (1.2, 2.5, 0.8).

[0133] Then, the acquired posture data is subjected to posture fusion. The extended Kalman filter is used to process the posture data of the head display to obtain more accurate and smooth posture information, which is represented by quaternions. For example, the fused posture quaternion can be (0.7, 0.2, 0.1, 0.6).

[0134] At the same time, the acquired spatial position data is processed. The spatial position data of the headset is processed using the simultaneous localization and mapping algorithm (SLAM) to obtain the precise spatial coordinates of the headset in the virtual scene. For example, the processed spatial coordinates may be (10.5, 25.2, 8.7).

[0135] Next, the fused attitude quaternion and the processed spatial coordinates are combined to generate the headset spatial pose matrix. This matrix describes the position and orientation of the headset in the virtual scene.

[0136] The headset position data is extracted as the viewpoint position according to the headset spatial pose matrix, and the headset pose quaternion is converted into the sight direction vector. For example, the viewpoint position can be (10.5, 25.2, 8.7) and the sight direction vector can be (0.3, 0.6, 0.7).

[0137] Based on the viewpoint position and the sight direction vector, combined with the field of view angle of the headset, the view cone equation is constructed and the projection matrix is ​​generated. The view cone equation defines the visible area in the virtual scene, and the projection matrix is ​​used to project the three-dimensional scene onto the two-dimensional screen. For example, the field of view angle can be 90 degrees.

[0138] Construct a hierarchical bounding box for the buildings in the scene. A hierarchical bounding box is a data structure used for fast collision detection, which wraps the building model with multiple bounding boxes of different levels.

[0139] Perform intersection check on the constructed hierarchical bounding box and the viewpoint culling parameters. Get a set of candidate buildings that intersect with the view frustum equation. This means that these buildings are likely to be within the user's field of view.

[0140] Calculate the depth value of the building in the candidate building set relative to the viewpoint. The depth value indicates the distance between the building and the viewpoint.

[0141] An occlusion map is constructed based on the depth values, and buildings that are completely occluded are removed to obtain the final set of visible buildings, which means that these buildings are within the user's field of view and are not occluded by other buildings.

[0142] The visible building set is classified according to the building type to obtain a building type group. For example, the buildings can be divided into residential, commercial, industrial and other types.

[0143] For each building type group, extract the shared geometry and material data. For example, all res ident ial buildings may share the same roof geometry and material.

[0144] Upload the extracted geometry and material data to the GPU storage buffer.

[0145] The instantiation transformation matrix of each building is calculated based on the HMD space pose matrix. This matrix describes the position, rotation, and scale of each building in the virtual scene.

[0146] Organize the computed instanced transformation matrices into instanced data buffers.

[0147] Reads geometry and material data from GPU storage buffers.

[0148] Applies the instanced transformation matrix in the instanced data buffer to the geometry data.

[0149] The GPU instanced rendering pipeline is used to render the visible building collection in parallel.

[0150] The rendering result is synthesized with the background image to generate the initial rendering image of the virtual reality scene, which is finally presented in the user's virtual reality headset.

[0151] The solution of this application can:

[0152] Improve rendering efficiency: Through the frustum culling and occlusion culling technology, the number of buildings that need to be rendered is effectively reduced, thereby significantly improving rendering efficiency and making real-time rendering of large-scale urban building models possible. Enhance immersion: By real-time acquisition and processing of the spatial position and posture data of the headset, the user's perspective is updated in real time, which enhances the user's immersion and provides a more realistic virtual reality experience. Reduce hardware pressure: Through GPU instantiation rendering technology, the GPU's computing burden is reduced, the requirements for hardware devices are reduced, and this method can run on a wider range of hardware platforms.

[0153] Figure 2 FIG. 1 is a structural diagram of a virtual reality building roaming system based on scene rendering according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0154] The first unit is used to collect building three-dimensional model data, divide the building three-dimensional model data into multiple hierarchical blocks by using an octree space segmentation algorithm; calculate the viewpoint relative distance for the hierarchical blocks, and determine the building detail level threshold according to the viewpoint relative distance; construct a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing far and near level information; calculate the illumination distribution parameters of the optimized building model data by using a pre-calculated irradiance transmission technology, and bake the illumination distribution parameters to generate a basic illumination map;

[0155] The second unit is used to divide a plurality of rendering depth intervals based on the optimized building model data, and set a shadow map of a corresponding resolution for each rendering depth interval; synthesize the basic light map and the shadow map in real time to obtain scene light and shadow synthesis data; calculate direct lighting and indirect lighting using a physically based rendering method, merge the lighting calculation results with the scene light and shadow synthesis data, and generate dynamic lighting rendering data; construct a deferred rendering pipeline according to the dynamic lighting rendering data, and perform lighting processing on pixels visible from a current viewpoint;

[0156] The third unit is used to collect the spatial position and posture data of the virtual reality head display in real time, determine the current viewpoint parameters according to the spatial position and posture data; perform frustum culling and occlusion culling based on the current viewpoint parameters to screen out visible building data; perform GPU instantiation rendering on the visible building data to generate an initial rendered image; input the initial rendered image into a high dynamic range processing unit, use an adaptive exposure adjustment algorithm to optimize the scene light and dark contrast, and obtain an optimized rendered image; apply the temporal anti-aliasing technology to the optimized rendered image to eliminate geometric edges and temporal domain flicker, and generate a final scene image in combination with an atmospheric scattering algorithm based on image space.

[0157] According to a third aspect of the embodiments of the present invention,

[0158] An electronic device is provided, comprising:

[0159] processor;

[0160] a memory for storing processor-executable instructions;

[0161] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0162] A fourth aspect of the embodiments of the present invention is:

[0163] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0164] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual reality architectural roaming method based on scene rendering, characterized in that: include: Collecting building three-dimensional model data, dividing the building three-dimensional model data into multiple hierarchical blocks by using an octree space segmentation algorithm; calculating the viewpoint relative distance for the hierarchical blocks, and determining the building detail level threshold according to the viewpoint relative distance; constructing a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing near and far level information; calculating the illumination distribution parameters of the optimized building model data by using a pre-calculated irradiance transmission technology, and baking the illumination distribution parameters to generate a basic illumination map; Divide a plurality of rendering depth intervals based on the optimized building model data, and set a shadow map of corresponding resolution for each rendering depth interval; The basic light map and the shadow map are synthesized in real time to obtain scene light and shadow synthesis data; Calculating direct lighting and indirect lighting using a physically based rendering method, fusing the lighting calculation results with the scene light and shadow synthesis data to generate dynamic lighting rendering data; Constructing a deferred rendering pipeline according to the dynamic lighting rendering data, and performing lighting processing on pixels visible from the current viewpoint; Collecting spatial position and posture data of a virtual reality head display in real time, and determining current viewpoint parameters according to the spatial position and posture data; Perform frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; GPU instantiation rendering is performed on the visible building data to generate an initial rendering image; the initial rendering image is input into a high dynamic range processing unit, and an adaptive exposure adjustment algorithm is used to optimize the scene light and dark contrast to obtain an optimized rendering image; the temporal anti-aliasing technology is applied to the optimized rendering image to eliminate geometric edges and temporal domain flicker, and a final scene image is generated in combination with an atmospheric scattering algorithm based on image space.

2. The method according to claim 1, characterized in that Collecting building three-dimensional model data, dividing the building three-dimensional model data into multiple hierarchical blocks by using an octree space segmentation algorithm; calculating viewpoint relative distances for the hierarchical blocks, and determining building detail level thresholds according to the viewpoint relative distances, including: A ground laser scanner is used to obtain first point cloud data ranging from the bottom of the building to a height of 20 meters, with a scanning resolution of 1.6 millimeters per 10 meters; a drone is used to obtain second point cloud data of the upper part of the building and the roof structure, with a flight altitude of 30 meters from the top of the building; feature points are extracted from the first point cloud data and the second point cloud data, a registration matrix is ​​calculated based on the feature points, and the first point cloud data and the second point cloud data are registered and fused to obtain the three-dimensional model data of the building; A scene bounding box is constructed for the building three-dimensional model data, the scene bounding box is used as a root node, and the diagonal length of the scene bounding box is calculated as a reference scale; the face density of the root node is calculated according to the building three-dimensional model data, and when the face density is greater than a preset density threshold and the node size is greater than one sixteenth of the reference scale, the optimal segmentation plane is determined by principal component analysis; the building three-dimensional model data is spatially segmented based on the optimal segmentation plane to obtain a plurality of child nodes; the centroid position of the triangular facets crossing the optimal segmentation plane is calculated, and the triangular facets are assigned to corresponding child nodes to form a plurality of hierarchical blocks; a link table is established between adjacent nodes of the hierarchical block to record shared facet information; Discretizing the hierarchical blocks into voxel grids, calculating the shortest distance from each voxel in the voxel grid to the data surface of the three-dimensional building model to form a distance field; compressing and storing the distance field using a third-order spherical harmonic function to obtain spherical harmonic coefficients; querying the spherical harmonic coefficients of the corresponding voxels according to the viewpoint position, and reconstructing the viewpoint relative distance through the spherical harmonic coefficients; An architectural detail level threshold is set for the hierarchical block based on the viewpoint relative distance. When the viewpoint relative distance is less than or equal to a first distance threshold, the architectural detail level threshold is set to the first threshold and the original precision of the hierarchical block is maintained; when the viewpoint relative distance is greater than the first distance threshold and less than or equal to a second distance threshold, the architectural detail level threshold is set to the second threshold, and the hierarchical block is simplified by applying an edge collapse algorithm; when the viewpoint relative distance is greater than the second distance threshold and less than or equal to a third distance threshold, the architectural detail level threshold is set to the third threshold, and coplanar triangles are merged for the hierarchical block; when the viewpoint relative distance is greater than the third distance threshold, the architectural detail level threshold is set to the fourth threshold, and the hierarchical block is simplified with contour preservation.

3. The method according to claim 1, characterized in that Based on the building detail level threshold, a multi-level detail model is constructed to obtain optimized building model data containing near and far level information; using the pre-calculated irradiance transmission technology to calculate the illumination distribution parameters of the optimized building model data, and baking the illumination distribution parameters to generate a basic illumination map, including: According to the building detail level threshold, vertex curvature values ​​and boundary weights are respectively calculated, and a comprehensive error metric function is constructed with the vertex curvature values ​​and the boundary weights; based on the comprehensive error metric function, edge collapse operations, coplanar merging operations, and contour preservation operations are performed on the building three-dimensional model data to generate simplified mesh models with different precisions; Establishing an octree spatial structure, storing the normal information of the building three-dimensional model data in the octree spatial structure; mapping the normal information to the simplified mesh model to generate a multi-level detail model with normal information; Constructing a texture space parameterized grid, calculating the triangle area and the stretching coefficient based on the multi-level detail model with normal information; constructing a texture distortion energy function by multiplying the triangle area and the stretching coefficient; converging the texture distortion energy function by iteratively optimizing UV coordinates, and generating optimized building model data; Expanding incident illumination and bidirectional reflectance distribution functions into spherical harmonic basis functions; sampling multiple incident directions for the vertices of the optimized building model data; calculating the integral of the visibility function and the spherical harmonic basis functions to obtain a transmission matrix; and calculating illumination distribution parameters based on the transmission matrix; Construct a uniformly distributed sampling point grid, calculate irradiance values ​​for the sampling points of the sampling point grid; use radial basis function to spatially interpolate the irradiance values; divide the UV space based on the illumination distribution parameters and perform texture optimization; divide the UV space into pixel blocks of fixed size; extract the primary and secondary illumination directions of the pixel blocks, use principal component analysis to compress them, and perform anisotropic filtering to generate a basic light map.

4. The method according to claim 1, characterized in that: Divide a plurality of rendering depth intervals based on the optimized building model data, and set a shadow map of corresponding resolution for each rendering depth interval; The basic light map and the shadow map are synthesized in real time to obtain scene light and shadow synthesis data, including: Calculating a rendering depth range of the optimized building model data, dividing the rendering depth range into a near-view rendering depth interval, a mid-view rendering depth interval, and a far-view rendering depth interval, wherein a depth value of the near-view rendering depth interval is from zero to fifty meters, a depth value of the mid-view rendering depth interval is from fifty to two hundred meters, and a depth value of the far-view rendering depth interval is greater than two hundred meters; Generate a first-resolution shadow map for the near-view rendering depth interval, generate a second-resolution shadow map for the mid-view rendering depth interval, and generate a third-resolution shadow map for the far-view rendering depth interval, wherein the resolution of the first-resolution shadow map is 2048 by 2048, the resolution of the second-resolution shadow map is 1024 by 1024, and the resolution of the third-resolution shadow map is 512 by 512; Extracting a diffuse lighting component from the basic light map; applying a four-by-four pixel percentage proximity filter sampling to the diffuse lighting component for hardware filtering; calculating a depth offset value based on a normal vector of the optimized building model data, and correcting the diffuse lighting component using the depth offset value; A first transition zone is constructed at a boundary between the near-view rendering depth interval and the mid-view rendering depth interval, and a second transition zone is constructed at a boundary between the mid-view rendering depth interval and the far-view rendering depth interval; a first bilinear interpolation is performed on the first-resolution shadow map and the second-resolution shadow map within the first transition zone, and a second bilinear interpolation is performed on the second-resolution shadow map and the third-resolution shadow map within the second transition zone; Constructing a deferred rendering buffer, storing the position data, normal data, and depth data of the optimized building model data in the deferred rendering buffer; extracting the position data from the deferred rendering buffer, and determining the rendering depth intervals to which each part of the scene belongs according to the position data; calculating a scene change value of the rendering depth interval, and setting a shadow map update frequency based on the scene change value; asynchronously updating the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map according to the shadow map update frequency; When the viewpoint position moves, the dividing boundary of the rendering depth interval is recalculated, and the dividing boundary is filtered by applying a time domain smoothing coefficient; when the rendering depth interval is switched, a cross-gradient synthesis is performed on the shadow maps of adjacent rendering depth intervals; the diffuse reflection lighting component of the basic light map is synthesized in real time with the first resolution shadow map, the second resolution shadow map, and the third resolution shadow map to generate scene light and shadow synthesis data.

5. The method according to claim 1, characterized in that Calculating direct lighting and indirect lighting using a physically based rendering method, fusing the lighting calculation results with the scene light and shadow synthesis data to generate dynamic lighting rendering data; Constructing a deferred rendering pipeline according to the dynamic lighting rendering data, and performing lighting processing on the visible pixels of the current viewpoint includes: Construct a physically based lighting calculation model to decompose lighting information into direct lighting and indirect lighting; construct a ray tracing path from the camera viewpoint, record the intersection position of the light and the scene object, the surface normal, and the material properties; calculate the light source visibility based on the intersection position, calculate the lighting incident angle using the surface normal, and calculate the direct lighting contribution value in combination with the material properties; A secondary light is emitted to the intersection position, and an indirect lighting contribution value is calculated by Monte Carlo integration; the indirect lighting contribution value is decomposed into a diffuse reflection component and a specular reflection component according to the material properties; the direction of the secondary light is determined by importance sampling, and the ambient lighting information is collected along the direction of the secondary light; The direct illumination contribution value and the indirect illumination contribution value are weightedly superimposed to obtain an illumination calculation result; the illumination calculation result is accumulated in the time domain, and the time domain jitter is eliminated by using an exponential weighted average method; a mixing weight coefficient is calculated based on the material properties, and the illumination calculation result is merged with the scene light and shadow synthesis data according to the mixing weight coefficient to generate dynamic illumination rendering data; Writing the position information, normal information, depth information, and material information of the scene geometry into the deferred rendering buffer; constructing a dynamic light map containing the dynamic lighting rendering data, and binding the dynamic light map to the deferred rendering buffer; The depth information in the deferred rendering buffer is analyzed to determine visible pixels of the current viewpoint; a spatial relationship between the visible pixels and a light source is calculated, and illumination processing is performed on the visible pixels based on the spatial relationship.

6. The method according to claim 1, characterized in that Collecting spatial position and posture data of a virtual reality head display in real time, and determining current viewpoint parameters according to the spatial position and posture data; Perform frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; Performing GPU instantiation rendering on the visible building data to generate an initial rendering image includes: Acquire posture data and spatial position data of a virtual reality head display; use an extended Kalman filter to perform posture fusion on the posture data to obtain a head display posture quaternion; process the spatial position data based on a SLAM algorithm to obtain a head display spatial coordinate; combine the head display posture quaternion with the head display spatial coordinate to generate a head display spatial posture matrix; Extracting the head display position data as the viewpoint position according to the head display spatial posture matrix; converting the head display posture quaternion into a sight direction vector; constructing a view cone equation based on the viewpoint position and the sight direction vector; generating a projection matrix in combination with the head display field of view angle; combining the view cone equation with the projection matrix to form a viewpoint culling parameter; Constructing a hierarchical bounding box for the buildings in the scene; performing intersection detection on the hierarchical bounding box and the viewpoint culling parameter to obtain a set of candidate buildings that intersect with the view frustum equation; calculating the depth values ​​of the buildings in the candidate building set relative to the viewpoint position; constructing an occlusion map based on the depth values, culling completely occluded buildings, and obtaining a set of visible buildings; Classifying the visible building set according to building type to obtain building type groups; extracting shared geometry data and material data for each building type group; uploading the geometry data and the material data to a GPU storage buffer; calculating an instantiation transformation matrix for each building according to the head display spatial pose matrix; and organizing the instantiation transformation matrix into an instantiation data buffer; The geometric data and the material data are read from the GPU storage buffer; the instantiated transformation matrix in the instantiated data buffer is applied to the geometric data; the visible building set is rendered in parallel using the GPU instantiated rendering pipeline; and the rendering result is synthesized with the background image to generate an initial rendered image of the virtual reality scene.

7. A virtual reality architectural roaming system based on scene rendering, used to implement the method described in any one of claims 1 to 6, characterized in that: include: The first unit is used to collect building three-dimensional model data, divide the building three-dimensional model data into multiple hierarchical blocks by using an octree space segmentation algorithm; calculate the viewpoint relative distance for the hierarchical blocks, and determine the building detail level threshold according to the viewpoint relative distance; construct a multi-level detail model based on the building detail level threshold to obtain optimized building model data containing far and near level information; calculate the illumination distribution parameters of the optimized building model data by using a pre-calculated irradiance transmission technology, and bake the illumination distribution parameters to generate a basic illumination map; A second unit is used to divide a plurality of rendering depth intervals based on the optimized building model data, and set a shadow map of a corresponding resolution for each rendering depth interval; The basic light map and the shadow map are synthesized in real time to obtain scene light and shadow synthesis data; Calculating direct lighting and indirect lighting using a physically based rendering method, fusing the lighting calculation results with the scene light and shadow synthesis data to generate dynamic lighting rendering data; Constructing a deferred rendering pipeline according to the dynamic lighting rendering data, and performing lighting processing on pixels visible from the current viewpoint; The third unit is used to collect the spatial position and posture data of the virtual reality head display in real time, and determine the current viewpoint parameters according to the spatial position and posture data; Perform frustum culling and occlusion culling based on the current viewpoint parameters to filter out visible building data; GPU instantiation rendering is performed on the visible building data to generate an initial rendering image; the initial rendering image is input into a high dynamic range processing unit, and an adaptive exposure adjustment algorithm is used to optimize the scene light and dark contrast to obtain an optimized rendering image; the temporal anti-aliasing technology is applied to the optimized rendering image to eliminate geometric edges and temporal domain flicker, and a final scene image is generated in combination with an atmospheric scattering algorithm based on image space.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images