High-fidelity large-scale scene rendering method, system and equipment based on Gaussian splashing and medium

Abstract

The invention discloses a high-fidelity large-scale scene rendering method, system and device based on Gaussian splashing and a medium. The method comprises the steps of S1, collecting high-precision point cloud data of a large-scale scene; s2, generating an initial three-dimensional grid model, and introducing depth prior information to reduce floating artifacts in a rendering process; s3, constructing a multi-resolution three-dimensional Gaussian expression model, and adopting a self-adaptive scaling factor and random resolution level training scheme in each level; s4, in the rendering process, a hierarchical selection strategy is adopted based on the distance between the three-dimensional Gaussian and a rendering viewpoint, and high-quality and high-speed rendering on high-performance equipment is achieved; and S5, developing a lightweight renderer, and completing efficient rendering on a lightweight mobile device. According to the invention, by introducing a depth regularization technology, high-precision point cloud data and three-dimensional Gaussian representation are combined; the three-dimensional Gaussian model is optimized by using depth prior information, floating artifacts possibly occurring in the rendering process are reduced, and the rendering quality is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics processing, and in particular to a high-fidelity large-scale scene rendering method, system, device and medium based on Gaussian splashing. Background Art

[0002] With the growing demand for 3D scene rendering technology in fields such as virtual reality (VR), augmented reality (AR), and film and television entertainment, how to efficiently render large-scale scenes with high fidelity has become an important research direction in academia and industry. Traditional 3D scene rendering technology relies on the representation of polygonal meshes. Although significant progress has been made in many applications, these methods face many challenges in rendering performance and quality when faced with large-scale scenes containing a large number of details and complex geometric structures.

[0003] First, traditional polygonal mesh representation methods have limited expressiveness for complex geometric structures in large-scale scenes. Under the requirements of multi-view rendering, highly detailed modeling of the scene is usually required, which often leads to the generation of a large number of triangular patches, thus significantly increasing the computational burden of rendering. In order to achieve high-quality rendering effects in complex scenes, a large amount of computing resources and time are usually required, which is unacceptable for real-time applications.

[0004] Secondly, it is particularly difficult to achieve high-fidelity large-scale scene rendering on lightweight devices such as mobile devices. The computing power and storage space of mobile devices are far inferior to high-performance desktop devices, which often leads to performance bottlenecks when rendering complex 3D scenes on these devices. Traditional rendering methods cannot fully utilize the limited hardware resources of mobile devices while ensuring rendering quality, resulting in low rendering efficiency and unsatisfactory rendering results.

[0005] Furthermore, although the existing Level-of-Detail (LOD) technology can optimize the rendering performance of large-scale scenes to a certain extent, its performance at multiple resolutions is often not stable enough. Specifically, traditional LOD technology relies on manual or semi-automatic methods to divide and manage resolution levels, which may cause obvious level switching artifacts in the rendering effect when the scene changes dynamically or the viewpoint moves quickly, affecting visual continuity. In addition, LOD technology usually requires pre-generation of corresponding geometric models for different resolution levels, which increases the complexity of preprocessing and storage requirements.

[0006] On the other hand, rendering methods based on deep learning and point cloud data have gradually emerged in recent years, attempting to optimize the representation and rendering process of three-dimensional scenes through neural networks and deep prior information. However, existing deep learning methods often face problems such as high model complexity and long training and inference time when dealing with large-scale scenes. In particular, in the reconstruction and rendering process of high-precision point cloud data, how to effectively reduce floating artifacts and improve the realism of the rendering results is still an unresolved problem.

[0007] In addition, although the existing WebGL rendering technology provides the ability to perform hardware accelerated rendering in a browser environment, it still has performance bottlenecks when processing large-scale, complex geometric scenes. Especially on mobile devices, due to the limitations of computing and rendering capabilities, WebGL renderers often find it difficult to handle multi-resolution scenes while ensuring real-time and high-quality rendering.

[0008] In summary, the existing technologies have significant deficiencies in high-fidelity rendering of large-scale scenes, especially in real-time and adaptability to mobile devices. Therefore, there is an urgent need for a new rendering method that can efficiently process complex large-scale 3D scenes under multi-view and multi-resolution conditions, make full use of the hardware resources of lightweight devices, and achieve high-quality real-time rendering effects. Summary of the invention

[0009] The present invention aims to solve the existing problems and to provide a high-fidelity large-scale scene rendering method, system, device and medium based on Gaussian splashing.

[0010] In order to achieve the above object, the technical solution adopted by the present invention provides a high-fidelity large-scale scene rendering method based on Gaussian splashing, comprising the following steps:

[0011] S1. Collect high-precision point cloud data of large-scale scenes;

[0012] S2, generate an initial 3D mesh model based on high-precision point cloud data, and introduce depth prior information to reduce floating artifacts in the rendering process;

[0013] S3, construct a multi-resolution three-dimensional Gaussian expression model, using adaptive scaling factors and random resolution level training schemes in each level to optimize Gaussian distribution at different resolutions;

[0014] S4. During the rendering process, a hierarchical selection strategy is adopted based on the distance between the 3D Gaussian and the rendering viewpoint to ensure high-quality and high-speed rendering on high-performance devices when rendering large-scale scenes;

[0015] S5. Develop a lightweight renderer so that the multi-resolution three-dimensional Gaussian expression obtained based on S1-4 can be efficiently rendered on lightweight mobile devices including mobile phones, tablets, and laptops, thereby achieving real-time, high-fidelity rendering of large-scale scenes.

[0016] In S1 of some embodiments, multi-view RGBD data is also synchronously collected and fused with high-precision point cloud data at each collection point to ensure spatial alignment and geometric consistency of data at different viewing angles.

[0017] Furthermore, multi-view RGBD data and high-precision point cloud data are collected for large-scale scenes along the predetermined route, and multi-view scanning is adopted to capture the details of different geometric structures.

[0018] In S1 of some embodiments, a large-scale scene is divided into multiple sub-areas, and the data in each sub-area is fused through point cloud stitching and posture calibration to form a complete scene data set.

[0019] In S2 of some embodiments, a Poisson reconstruction method is used to generate an initial three-dimensional mesh model to generate a continuous and closed three-dimensional mesh.

[0020] In S2 of some embodiments, the depth prior information is introduced through the depth regularization technology to construct the depth regularization loss function L depth :

[0021]

[0022] Among them, D i Represents the true depth value of the i-th point in the point cloud data, represents the depth value of the i-th point predicted by the optimized three-dimensional Gaussian representation, and N is the total number of points in the point cloud data.

[0023] In S2 of some embodiments, an image reconstruction loss function L is also constructed. rgb , which is used to measure the pixel difference between the predicted image and the real image, is defined as:

[0024]

[0025] Where M represents the total number of pixels in the image, I j represents the RGB value of the jth pixel in the real image, Represents the RGB value of the j-th pixel predicted by the three-dimensional Gaussian representation model.

[0026] In S2 of some embodiments, the depth regularization loss function L depth And the image reconstruction loss function L rgb Combined to form the total loss function Ltotal , used to optimize the three-dimensional Gaussian representation model, the total loss function is defined as follows:

[0027] L total =L rgb +λ depth L depth ;

[0028] Among them, λ depth is the weight coefficient used to balance the contribution of depth regularization and image reconstruction.

[0029] In S3 of some embodiments, a multi-resolution three-dimensional Gaussian expression model is constructed to achieve a Level-of-Detail effect in large-scale scene rendering, which includes multiple resolution levels, each level corresponding to a different Gaussian distribution resolution; and an adaptive scaling factor s is defined. i ,s i is the scaling factor of the i-th level, which is used to adjust the scale of Gaussians at each level during training:

[0030] s i =α i β i ;

[0031] Among them, α i represents the scaling parameter related to scene depth, β i Represents the scaling parameter related to the viewpoint distance, and i is the index of the resolution level.

[0032] In S3 of some embodiments, the random resolution hierarchy training scheme includes:

[0033] Initialize the Gaussian distribution of each resolution level, and generate the initial point cloud according to the Poisson sampling method during the initialization process;

[0034] Randomly select a resolution level for training, using the adaptive scaling factor s i Adjust the resolution of Gaussian distribution to ensure the consistency of Gaussian expression at different resolutions;

[0035] The resolution levels are dynamically adjusted in each training iteration to optimize the balance between the levels, reducing over-smoothing of low-resolution layers and over-refinement of high-resolution layers.

[0036] In S3 of some embodiments, during the Level-of-Detail (LOD) rendering process, a level selection strategy is adopted based on the distance between the rendering viewpoint and the three-dimensional Gaussian distribution to select the most suitable resolution level according to the following formula:

[0037]

[0038] Among them, L(d) is the selected resolution level, L max is the highest resolution level, d is the distance between the current viewpoint and the Gaussian distribution, d max The maximum viewing distance.

[0039] In S3 of some embodiments, the rendering level is dynamically adjusted during the rendering process to avoid rendering artifacts and detail loss caused by resolution mismatch.

[0040] In S4 of some embodiments, in each rendering iteration, the rendering result is calculated based on the real-time position of the current viewpoint and the dynamically adjusted rendering level, and the result is compared with the previous frame to ensure a smooth transition between consecutive frames and reduce rendering artifacts caused by resolution level switching.

[0041] In S4 of some embodiments, the device performance adjustment factor P is also calculated. factor Adjust the maximum level and viewing distance range:

[0042]

[0043] Based on this, the maximum level and viewing distance range are dynamically adjusted to adapt to the computing power and scene requirements of different devices.

[0044] In S4 of some embodiments, a depth regularization module is introduced to perform depth constraints on the Gaussian expression in the rendering process to reduce the generation of floating artifacts.

[0045] In S5 of some embodiments, a lightweight renderer is developed based on WebGL, and rendering pipelines of different resolution levels are defined for a multi-resolution 3D Gaussian expression model, each rendering pipeline corresponding to a different LOD level, specifically including:

[0046] Create a separate shader program for each LOD level, written in WebGL's shader language, to process three-dimensional Gaussian expressions at different resolutions;

[0047] Define the input and output of each shader program, including vertex data, normals, texture coordinates, and lighting information, and automatically select the appropriate LOD level based on device performance;

[0048] Use FrameBuffer Objects for multi-level rendering to keep the rendering results consistent and smooth between different resolution levels.

[0049] In S5 of some embodiments, the lightweight renderer implements an optimization strategy based on deferred rendering, specifically including:

[0050] Calculate and cache the scene's global illumination information during initial rendering to reduce real-time computing overhead;

[0051] Implement screen-space-based shadow and reflection effects to improve visual effects while reducing the amount of calculation; for each frame of rendering, dynamically adjust the rendering resolution based on the current performance of the device to ensure smooth frame rates on different devices.

[0052] The present invention also provides a high-fidelity large-scale scene rendering system based on Gaussian splashing, comprising:

[0053] Data acquisition module: used to collect multi-view RGBD data and high-precision point cloud data of large-scale scenes, and ensure the spatial alignment and precise geometric representation of the collected data;

[0054] Data processing module: Generates the initial 3D mesh model based on high-precision point cloud data reconstruction; introduces depth prior information through deep regularization technology and constructs deep regularization loss function to reduce the generation of floating artifacts; Multi-resolution 3D Gaussian expression module: introduces adaptive scaling factor and random resolution hierarchical training scheme to optimize Gaussian distribution at different viewing distances;

[0055] Rendering control module: Based on the distance between the rendering viewpoint and the 3D Gaussian distribution, a dynamic layer selection strategy is used to adjust the rendering layer in real time, calculate the distance and select the optimal resolution layer; combined with the device performance adjustment factor, the rendering resolution and viewing distance range are dynamically adapted;

[0056] Renderer module: Develop a lightweight renderer based on WebGL to achieve efficient rendering on lightweight mobile devices; use a multi-level rendering pipeline and deferred rendering strategy, combined with double buffering technology, to achieve high-resolution and smooth visual experience.

[0057] In some embodiments, a display module is also included: used to display the rendering results and support user interaction operations.

[0058] The present invention also provides a high-fidelity large-scale scene rendering device, including an acquisition device, a memory and a processor. The acquisition device is used to acquire multi-perspective RGBD data and high-precision point cloud data. The memory includes a computer program that can be run thereon. When the processor executes the computer program, any of the rendering methods described above is implemented.

[0059] The present invention also provides a chip, comprising one or more processors, for calling and running a computer program from a memory, so that a device equipped with the chip executes the rendering method.

[0060] The present invention also provides a computer-readable storage medium, wherein the computer-executable instructions are used to execute the rendering method when executed by a computer processor.

[0061] Compared with the prior art, the present invention collects multi-view RGBD data and high-precision point cloud data along a predetermined route for large-scale scenes, and combines IMU data with multi-view RGBD data to ensure the spatial alignment and geometric consistency of data at different viewpoints. The Poisson reconstruction method is used to generate the initial three-dimensional mesh model, which can accurately capture the complex geometric structure in the scene. By introducing deep regularization technology, high-precision point cloud data is combined with three-dimensional Gaussian representation, and deep prior information is used to optimize the three-dimensional Gaussian model, which reduces the floating artifacts that may appear during the rendering process and significantly improves the rendering quality.

[0062] In summary, the method of the present invention significantly improves the rendering quality and efficiency of large-scale scenes through innovative designs in data collection, model expression, algorithm optimization, and lightweight rendering, and is particularly practical on mobile devices. The data in the table further confirms the powerful performance and broad application prospects of the present invention in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure is a flowchart illustrating an embodiment of the method of the present invention.

[0064] Figure 2 The figure is a comparison diagram of the effects of the embodiments of the method of the present invention.

[0065] Figure 3 Schematic diagram of the structure of the multi-resolution three-dimensional Gaussian expression model. DETAILED DESCRIPTION

[0066] The present invention will now be further described with reference to the accompanying drawings.

[0067] See also Figure 1 , Figure 1 An embodiment of the present invention is shown. The first embodiment of the present invention provides a high-fidelity large-scale scene rendering method based on Gaussian splashing, comprising the following steps:

[0068] S1. Collect multi-view RGBD data and high-precision point cloud data for large-scale scenes along a predetermined route to generate scene data with a variety of different geometric structures;

[0069] S2. Based on the collected high-precision point cloud data, the Poisson reconstruction method is used to generate the initial mesh, and the depth prior information is introduced through the deep regularization technology, which is used as a supervisory signal to optimize the three-dimensional Gaussian representation, reduce floating artifacts in the rendering process, and improve the rendering quality;

[0070] S3. Develop and apply a multi-resolution 3D Gaussian representation model that includes different levels of resolution, using adaptive scaling factors and random resolution level training schemes for each level to optimize Gaussian representations at different resolutions during Level-of-Detail rendering.

[0071] S4. During the rendering process, a hierarchical selection strategy is adopted based on the distance between the 3D Gaussian and the rendering viewpoint to ensure high-quality and high-speed rendering on high-performance devices when rendering large-scale scenes;

[0072] S5, a lightweight renderer developed based on WebGL, enables efficient rendering of multi-resolution three-dimensional Gaussian expressions on lightweight mobile devices including mobile phones, tablets, and laptops, thereby achieving real-time, high-fidelity rendering of large-scale scenes.

[0073] The "large-scale scene" referred to in the present invention is an indoor or outdoor scene with an area greater than 50 square meters, such as a large / multi-story garage. In actual situations, the "large-scale scene" to which the method of the present invention is applicable can also be a scene of several thousand to tens of thousands of square meters, provided that the corresponding existing equipment is configured.

[0074] In the implementation manner of the present invention, S1 may also include: S11, using a handheld scanning device equipped with an IMU, a LiDAR and a fisheye camera to collect multi-perspective RGBD data of a large-scale scene along a predetermined route, the handheld scanning device collects RGB images with a resolution of 6K at a speed of 30 frames per second, and synchronously records depth data associated with each perspective, the depth data is collected by a LiDAR sensor, and high-precision point cloud data is generated at a rate of 2.6 million points per second.

[0075] Furthermore, S1 may also include: S12. During the data collection process, the handheld scanning device runs along a predetermined route, including multi-perspective scanning in four directions: forward, reverse, left, and right, to ensure full coverage of the scene. The predetermined route is designed to capture details of various different geometric structures in large-scale scenes.

[0076] Furthermore, S1 may also include: S13, obtaining the posture data of the device through the IMU sensor of the handheld scanning device, which is used to calculate the spatial position of each collection point, and fusing it with the synchronous data of the LiDAR and the fisheye camera, so that the RGBD data and point cloud data collected at each viewing angle are spatially aligned.

[0077] Furthermore, S1 may also include: S14, adjusting the exposure parameters and acquisition speed of the handheld scanning device at the junction of a partially open area and a closed area, and in an area where light changes greatly, wherein when the handheld scanning device passes through the junction of light and dark light, data acquisition is suspended to allow the sensor to adapt to light changes and to ensure uniformity of RGBD data in the area.

[0078] Furthermore, S1 may also include: S15. For large-scale scenes, the scene is divided into multiple sub-areas, and data collection is performed in sequence. The data in each sub-area is fused through point cloud stitching and posture calibration to form a complete scene data set.

[0079] In S1, multi-view RGBD data and high-precision point cloud data are collected along a predetermined route for a large-scale scene by using a handheld scanning device equipped with an IMU, a LiDAR, and a fisheye camera. By combining the IMU data with the multi-view RGBD data, the spatial alignment and geometric consistency of the data at different viewpoints are ensured. In an embodiment of the present invention, S2 may also include: S21, based on the collected high-precision point cloud data, an initial three-dimensional mesh model is generated using a Poisson reconstruction method, and the Poisson reconstruction method globally optimizes the point cloud data to generate a continuous, closed three-dimensional mesh that can accurately capture the geometric shape of the scene.

[0080] Furthermore, S2 may also include: S22, after generating the initial grid, introducing the depth prior information through the depth regularization technology, and constructing the depth regularization loss function L depth :

[0081]

[0082] Among them, D i Represents the true depth value of the i-th point in the point cloud data, represents the depth value of the i-th point predicted by the optimized three-dimensional Gaussian representation, and N is the total number of points in the point cloud data.

[0083] On this basis, S2 can also include: S23, at the same time, construct the image reconstruction loss function L rgb , which is used to measure the pixel difference between the predicted image and the real image, is defined as:

[0084]

[0085] Where M represents the total number of pixels in the image, I j represents the RGB value of the jth pixel in the real image, represents the RGB value of the j-th pixel predicted by the three-dimensional Gaussian representation model;

[0086] Then S2 includes: S24, the depth regularization loss function L depth And the image reconstruction loss function L rgb Combined to form the total loss function L total , used to optimize the three-dimensional Gaussian representation model, the total loss function is defined as follows:

[0087] L total =L rgb +λ depth L depth ;

[0088] Among them, λ depth is the weight coefficient used to balance the contribution of depth regularization and image reconstruction;

[0089] Furthermore, S2 may also include: S25. During the optimization process, various parameters of the three-dimensional Gaussian representation are iteratively adjusted to reduce the occurrence of floating artifacts, and ultimately generate a rendering result with high fidelity, ensuring that the rendering result is visually highly consistent with the real scene.

[0090] In S2, the Poisson reconstruction method is used to generate the initial 3D mesh model, which can accurately capture the complex geometric structure in the scene. By introducing deep regularization technology, high-precision point cloud data is combined with 3D Gaussian representation, and deep prior information is used to optimize the 3D Gaussian model, which reduces the floating artifacts that may appear during the rendering process and significantly improves the rendering quality.

[0091] In the implementation manner of the present invention, S3 may also include: S31, developing a multi-resolution three-dimensional Gaussian expression model for achieving Level-of-Detail effects in large-scale scene rendering. Figure 3 ,The multi-resolution three-dimensional Gaussian expression model contains multiple resolution levels, each level corresponds to a different Gaussian distribution resolution, Figure 3 As the distance changes from far to near, the Gaussian resolution changes from fine to coarse; the model transforms the multi-resolution Gaussian into an octree structure, and the multi-resolution Gaussian representation is very suitable for storage in the octree structure, and each level of the Gaussian representation is effectively stored in the corresponding depth node of the octree. To adapt to different viewing distances and scene complexities.

[0092] Further, S3 may also include: S32, defining an adaptive scaling factor s for multiple resolution levels i , where s i is the scaling factor of the i-th level, which is used to adjust the scale of Gaussians at each level during training:

[0093] s i =α i β i ;

[0094] Among them, α i represents the scaling parameter related to scene depth, β i Represents the scaling parameter related to the viewpoint distance, and i is the index of the resolution level.

[0095] Further, S3 may also include: S33, in the model training process, adopting a random resolution hierarchical training scheme, specifically including the following steps:

[0096] Initialize the Gaussian distribution of each resolution level, and generate the initial point cloud according to the Poisson sampling method during the initialization process;

[0097] Randomly select a resolution level for training, using an adaptive scaling factor s i Adjust the resolution of Gaussian distribution to ensure the consistency of Gaussian expression at different resolutions;

[0098] In each training iteration, the resolution levels are dynamically adjusted to optimize the balance between the levels, reducing over-smoothing of low-resolution levels and over-refinement of high-resolution levels;

[0099] Further, S3 may also include: S34, in the Level-of-Detail (LOD) rendering process, based on the distance between the rendering viewpoint and the three-dimensional Gaussian distribution, adopting a level selection strategy, and selecting the most suitable resolution level according to the following formula:

[0100]

[0101] Among them, L(d) is the selected resolution level, L max is the highest resolution level, d is the distance between the current viewpoint and the Gaussian distribution, d max is the maximum viewing distance;

[0102] Furthermore, S3 may also include: S35. According to the level selection strategy, dynamically adjust the rendering level during the rendering process to ensure high-quality and high-speed rendering effects on high-performance devices, avoid rendering artifacts and detail loss caused by resolution mismatch, and thus optimize the overall rendering quality and efficiency.

[0103] A multi-resolution 3D Gaussian expression model has been developed in S3, which is specially designed for Level-of-Detail (LOD) rendering. By introducing adaptive scaling factors and random resolution level training schemes, it can effectively optimize Gaussian distribution at different resolution levels. During the rendering process, the most suitable resolution level is dynamically selected based on the distance between the rendering viewpoint and the 3D Gaussian distribution, thereby achieving a significant increase in rendering speed while ensuring high rendering quality, avoiding the common level switching artifact problem in traditional LOD methods, and ensuring the continuity and consistency of the rendering results, especially when the scene changes dynamically.

[0104] In this implementation, S4 may further include: S41, in each rendering iteration, based on the real-time position of the current viewpoint and the dynamically adjusted rendering level, calculating the rendering result, and comparing the result with the previous frame to ensure a smooth transition between consecutive frames and reduce rendering artifacts caused by resolution level switching;

[0105] Furthermore, S4 may also include: S42, to further optimize the rendering performance, calculate the device performance adjustment factor P in combination with the hardware capability of the rendering device. factor Used to adjust the maximum level and viewing distance range:

[0106]

[0107] The maximum level and viewing distance range are dynamically adjusted accordingly to adapt to the computing capabilities and scene requirements of different devices. Further, S4 may also include: S43, by introducing a depth regularization module, the Gaussian expression in the rendering process is depth constrained to reduce the generation of floating artifacts and improve the rendering quality.

[0108] Furthermore, S4 may also include: S44. Through a dynamic layer selection strategy, it is ensured that when rendering large-scale scenes, no matter how the viewpoint moves, an optimal balance can be achieved between rendering quality and rendering speed to achieve a high-fidelity rendering effect.

[0109] In S4, deep regularization technology is introduced into the rendering algorithm, and it is combined with the image reconstruction loss function to form a comprehensive loss function framework. Specifically, by defining the deep regularization loss function and the image reconstruction loss function, and combining the two to form a total loss function, the generation of floating artifacts in the rendering process is effectively reduced, and the realism of the rendering results is improved. By iteratively optimizing the parameters of the three-dimensional Gaussian representation model, higher accuracy and robustness than traditional methods are shown in high-precision rendering of large-scale scenes. In this implementation, S5 may include: S51, developing a lightweight renderer based on WebGL, so that multi-resolution three-dimensional Gaussian expression can be efficiently rendered on lightweight mobile devices including mobile phones, tablets, and laptops. The renderer is developed using JavaScript and WebGLAPI to ensure hardware-accelerated rendering on lightweight devices;

[0110] Furthermore, S5 may also include: S52, when developing a renderer, for a multi-resolution three-dimensional Gaussian expression model, defining rendering pipelines of different resolution levels, each rendering pipeline corresponding to a different LOD level, specifically including the following steps:

[0111] Create a separate shader program for each LOD level, written in WebGL's shader language, to process three-dimensional Gaussian expressions at different resolutions;

[0112] Define the input and output of each shader program, including vertex data, normals, texture coordinates, and lighting information, and automatically select the appropriate LOD level based on device performance;

[0113] Use FrameBuffer Objects for multi-level rendering to keep the rendering results consistent and smooth between different resolution levels.

[0114] Furthermore, S5 may also include: S53, in order to adapt to the computing resource limitation of the lightweight device, the renderer implements an optimization strategy based on delayed rendering, which specifically includes the following steps:

[0115] Calculate and cache the scene's global illumination information during initial rendering to reduce real-time computing overhead;

[0116] Implement screen-space-based shadow and reflection effects to improve visual effects while reducing the amount of calculation; for each frame of rendering, dynamically adjust the rendering resolution based on the current performance of the device to ensure smooth frame rates on different devices.

[0117] Furthermore, S5 may also include: S54, the rendering result of the renderer is directly displayed on the screen of the device through the WebGL drawing buffer, and double buffering technology is used to reduce screen tearing to ensure a smooth visual experience at high resolution.

[0118] Furthermore, S5 may also include: S55. Through the implementation of a multi-resolution three-dimensional Gaussian expression model of the renderer, it is possible to complete real-time rendering of large-scale scenes on lightweight devices with lower computing resources, thereby achieving high-fidelity rendering effects. At the same time, it supports real-time interaction and dynamic scene changes, allowing users to obtain a rendering experience on mobile devices that is similar to that of high-performance devices.

[0119] In S5, through a lightweight renderer developed based on WebGL, multi-resolution 3D Gaussian expressions can be efficiently rendered on lightweight mobile devices including mobile phones, tablets and laptops. Thanks to the hardware acceleration capabilities of WebGL and the optimized design of multi-resolution 3D Gaussian expressions in the renderer, high-quality real-time rendering can be achieved on lightweight devices, ensuring the smoothness and fidelity of large-scale scene rendering. The renderer uses a multi-level rendering pipeline and combines a deferred rendering strategy to achieve the best rendering effect on different devices through dynamic adaptation of device performance.

[0120] Based on the above method, this example uses the above method (Ours (LOD-LiDAR-GS)) to render the large-scale garage scene in the GarageWorld dataset with high fidelity and runs it on an NVIDIA RTXA6000 GPU using the PyTorch framework. Figure 2 , by comparing with other existing methods (such as 3DGS, Mip-Splatting, F2-NeRF, etc.), the present invention significantly improves the image quality. In terms of indicators such as PSNR, SSIM and LPIPS, the performance of the present invention is particularly outstanding. For example, in terms of the PSNR indicator, the method of the present invention reached 25.77, which is higher than most of the comparison methods; in terms of the SSIM indicator, it also achieved 0.812, second only to 3DGS (0.822); in terms of the LPIPS indicator, the present invention even reached 0.210, showing an advantage over other methods in visual perception, and can better preserve image details and structures.

[0121] By comparison, it can be seen that other methods, such as 3DGS, although they perform well in PSNR and SSIM, are relatively poor in LPIPS (0.412), which shows that they are insufficient in preserving high-frequency details. The method of the present invention finds a better balance between structural similarity (SSIM) and perceptual loss (LPIPS), especially in the highly complex and detailed garage scene, showing excellent rendering quality.

[0122] This embodiment also uses the above method to optimize the real-time rendering of ScanNet++ scenes. In the ScanNet++ dataset, the dynamic resolution selection strategy of the present invention effectively optimizes the rendering effect in multi-resolution scenes. By dynamically adjusting the rendering level based on the distance between the viewpoint and the three-dimensional Gaussian distribution, the method of the present invention achieved 29.19 on PSNR, 0.927 on SSIM, and a minimum value of 0.064 on LPIPS. These results show that the present invention not only performs well in pixel-level accuracy, but also far exceeds the existing technology in the fidelity of visual perception.

[0123] It is worth noting that although OCT-GS has a slightly higher PSNR performance (31.55), the present invention performs better on LPIPS (0.064), which indicates that the present invention achieves a better balance between overall perceptual quality and performance.

[0124] This embodiment also uses the above method to perform real-time rendering of lightweight devices in the KITTI-360 scenario. In the KITTI-360 dataset, this embodiment demonstrates its powerful rendering capabilities on large-scale outdoor scenes. On lightweight devices, the method of the present invention achieves excellent performance of PSNR of 24.53, SSIM of 0.811, and LPIPS of 0.167. This achieves rendering quality close to that of high-performance devices on mobile devices, especially in complex outdoor scenes, where the present invention achieves a good balance between fidelity and computational efficiency. In contrast, other methods such as NGP and F2-NeRF perform relatively poorly on KITTI-360, especially in the SSIM and LPIPS indicators, which are 0.586 and 0.247, respectively, which are significantly lower than the method of the present invention. This shows that the present invention not only has advantages in rendering speed when processing large-scale, complex geometric scenes, but also ensures the fidelity of visual quality.

[0125] This embodiment also uses the above method to implement a lightweight WebGL renderer. This embodiment describes in detail the implementation process of a lightweight renderer based on WebGL. The renderer is developed based on JavaScript and WebGL API, and is optimized specifically for multi-resolution three-dimensional Gaussian expression models, enabling efficient rendering on lightweight mobile devices including mobile phones, tablets, and laptops. Through a multi-level rendering pipeline and dynamic rendering strategy, the renderer can dynamically adjust the rendering resolution according to the performance of the current device to ensure a smooth user experience.

[0126] In addition, the renderer uses double buffering technology to reduce screen tearing and ensure a smooth visual experience at high resolution. Through this optimized design, the renderer of the present invention achieves rendering quality close to that of high-performance devices on lightweight devices and performs well in real-time rendering of large-scale scenes.

[0127] Table 1 Data comparison of Example 1-Example 3

[0128]

[0129] The present invention uses a handheld scanning device equipped with IMU, LiDAR and fisheye camera to collect multi-view RGBD data and high-precision point cloud data for large-scale scenes along a predetermined route. By combining IMU data with multi-view RGBD data, the spatial alignment and geometric consistency of data at different viewpoints are ensured. In particular, the Poisson reconstruction method is used to generate the initial three-dimensional mesh model, which can accurately capture the complex geometric structure in the scene. By introducing deep regularization technology, high-precision point cloud data is combined with three-dimensional Gaussian representation, and deep prior information is used to optimize the three-dimensional Gaussian model, which reduces the floating artifacts that may appear in the rendering process and significantly improves the rendering quality.

[0130] In summary, the method of the present invention significantly improves the rendering quality and efficiency of large-scale scenes through innovative designs in data collection, model expression, algorithm optimization, and lightweight rendering, and is particularly practical on mobile devices. The data in the table further confirms the powerful performance and broad application prospects of the present invention in practical applications.

[0131] The second embodiment of the present invention provides a high-fidelity large-scale scene rendering system based on Gaussian splashing, comprising the following modules:

[0132] Data acquisition module: used to collect multi-view RGBD data and high-precision point cloud data of large-scale scenes through handheld scanning devices equipped with IMU, LiDAR and fisheye cameras; the data acquisition module can generate scene data including a variety of different geometric structures, and obtain the posture data of the device through the IMU sensor to ensure the spatial alignment and precise geometric representation of the collected data.

[0133] Data processing module: Receives high-precision point cloud data generated by the data acquisition module, and uses the Poisson reconstruction method to generate an initial three-dimensional mesh model; introduces depth prior information through deep regularization technology, constructs a deep regularization loss function, and combines it with the image reconstruction loss function to optimize the three-dimensional Gaussian representation model, reduce the generation of floating artifacts during rendering, and improve rendering quality.

[0134] Multi-resolution 3D Gaussian expression module: Designed for Level-of-Detail rendering, it includes multiple resolution levels. It optimizes Gaussian distribution at different viewing distances by introducing adaptive scaling factors and random resolution level training schemes. It can dynamically select the most suitable resolution level to achieve high-quality rendering effects, especially in complex scenes and dynamic viewpoint conditions.

[0135] Rendering control module: Based on the distance between the rendering viewpoint and the three-dimensional Gaussian distribution, a dynamic level selection strategy is used to adjust the rendering level in real time; the rendering control module calculates the distance and selects the optimal resolution level to ensure the best balance between rendering quality and speed under different devices and scene complexities; the rendering control module further combines the device performance adjustment factor to dynamically adapt the rendering resolution and viewing distance range to ensure that the rendering effects on both high-performance devices and lightweight devices can meet real-time requirements.

[0136] Renderer module: A lightweight renderer developed based on WebGL. The renderer module can achieve efficient rendering on lightweight mobile devices including mobile phones, tablets, laptops, etc.; the renderer module uses a multi-level rendering pipeline and deferred rendering strategy, combined with double buffering technology, to ensure high-resolution and smooth visual experience when rendering large-scale scenes on mobile devices.

[0137] Preferably, a display module is also included: used to display the rendering results, which are directly displayed on the device screen through the WebGL drawing buffer to ensure the continuity and smoothness of the picture during real-time rendering; the display module supports user interactive operations such as viewpoint movement and scene zooming, providing intuitive visual feedback and high-fidelity rendering experience.

[0138] The third embodiment of the present invention also provides a high-fidelity dynamic character rendering device, including an acquisition device, a memory and a processor. The acquisition device is used to acquire multi-perspective RGBD data and high-precision point cloud data. The memory includes a computer program that can be run thereon. When the processor executes the computer program, the above-mentioned rendering method is implemented.

[0139] The memory may include a random access memory, a flash memory, a read-only memory, a programmable read-only memory, a non-volatile memory or a register, etc. The processor may be a central processing unit (CPU), etc. or a graphic processing unit (GPU). The memory may store executable instructions. The processor may execute the executable instructions stored in the memory to implement the various processes described herein.

[0140] It can be understood that the memory in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a ROM (Read-Only Memory), a PROM (Programmable ROM), an EPROM (Erasable PROM), an EEPROM (Electrically EPROM), or a flash memory. The volatile memory can be a RAM (Random Access Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), DDRSDRAM (Double Data Rate SDRAM), ESDRAM (Enhanced SDRAM), SLDRAM (Synchlink DRAM), and DRRAM (Direct Rambus RAM). The memory described herein is intended to include, but is not limited to, these and any other suitable types of memory. In some embodiments, the memory stores the following elements, upgrade packages, executable units, or data structures, or a subset thereof, or an extended set thereof: an operating system and an application.

[0141] Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program includes various application programs for implementing various application services. The program for implementing the method of the embodiment of the present invention may be included in the application program. In the embodiment of the present invention, the processor calls the program or instruction stored in the memory, specifically, the program or instruction stored in the application program, and the processor is used to execute the rendering method steps provided in the first embodiment.

[0142] The fourth embodiment of the present invention further provides a chip for executing the rendering method in the first aspect. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip is used to execute the rendering method in the first embodiment.

[0143] In addition, in a fifth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the rendering method in the first embodiment of the present invention are implemented. For example, the machine-readable storage medium may include, but is not limited to, various known and unknown types of non-volatile memories.

[0144] A sixth embodiment of the present invention further provides a computer program product, including computer program instructions, which enable a computer to execute the above rendering method.

[0145] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A high-fidelity large-scale scene rendering method based on Gaussian splashing, characterized in that include: S1. Collect high-precision point cloud data of large-scale scenes; S2, generate an initial 3D mesh model based on high-precision point cloud data, and introduce depth prior information to reduce floating artifacts in the rendering process; S3, construct a multi-resolution three-dimensional Gaussian expression model, using adaptive scaling factors and random resolution level training schemes in each level to optimize Gaussian distribution at different resolutions; S4. During the rendering process, a hierarchical selection strategy is adopted based on the distance between the 3D Gaussian and the rendering viewpoint to achieve high-quality and high-speed rendering on high-performance devices; S5. Develop a lightweight renderer so that the multi-resolution three-dimensional Gaussian expression obtained based on S1-4 can be efficiently rendered on lightweight mobile devices.

2. The method according to claim 1, characterized in that: In S2, the depth prior information is introduced through the deep regularization technology to construct the deep regularization loss function L depth : Among them, D i Represents the true depth value of the i-th point in the point cloud data, represents the depth value of the i-th point predicted by the optimized three-dimensional Gaussian representation, and N is the total number of points in the point cloud data.

3. The method according to claim 1, 5 or 6, characterized in that: In S2, the image reconstruction loss function L is also constructed rgb , which is used to measure the pixel difference between the predicted image and the real image, is defined as: Where M represents the total number of pixels in the image, I j represents the RGB value of the jth pixel in the real image, Represents the RGB value of the j-th pixel predicted by the three-dimensional Gaussian representation model.

4. The method according to claim 3, characterized in that: In S2, the depth regularization loss function L depth And the image reconstruction loss function L rgb Combined to form the total loss function L total , used to optimize the three-dimensional Gaussian representation model, the total loss function is defined as follows: L total =L rgb +λ depth L depth ; Among them, λ depth is the weight coefficient.

5. The method according to claim 1, characterized in that: In S3, during the LOD rendering process, based on the distance between the rendering viewpoint and the three-dimensional Gaussian distribution, a level selection strategy is adopted to select the most suitable resolution level according to the following formula: Among them, L(d) is the selected resolution level, L max is the highest resolution level, d is the distance between the current viewpoint and the Gaussian distribution, d max The maximum viewing distance.

6. The method according to claim 1, characterized in that: In S5, a lightweight renderer is developed based on WebGL. For the multi-resolution 3D Gaussian expression model, rendering pipelines with different resolution levels are defined. Each rendering pipeline corresponds to a different LOD level, including: Create independent shaders for each LOD level to process 3D Gaussian expressions at different resolutions; Define the input and output of each shader, including vertex data, normals, texture coordinates, and lighting information, and automatically select the appropriate LOD level based on device performance; Multi-level rendering ensures that rendering results remain consistent and smooth between different resolution levels.

7. A high-fidelity large-scale scene rendering system based on Gaussian splashing, characterized by include: Data acquisition module: used to collect multi-view RGBD data and high-precision point cloud data of large-scale scenes, and ensure the spatial alignment and precise geometric representation of the collected data; Data processing module: Generate an initial 3D mesh model based on high-precision point cloud data reconstruction; introduce depth prior information, construct a depth regularization loss function, and reduce the generation of floating artifacts; Multi-resolution 3D Gaussian expression module: introduces adaptive scaling factors and random resolution level training scheme to optimize Gaussian distribution at different viewing distances; Rendering control module: Based on the distance between the rendering viewpoint and the 3D Gaussian distribution, a dynamic layer selection strategy is used to adjust the rendering layer in real time, calculate the distance and select the optimal resolution layer; combined with the device performance adjustment factor, the rendering resolution and viewing distance range are dynamically adapted; Renderer module: Develop a lightweight renderer based on WebGL to achieve efficient rendering on lightweight mobile devices; use a multi-level rendering pipeline and deferred rendering strategy, combined with double buffering technology, to achieve high-resolution and smooth visual experience.

8. A high-fidelity large-scale scene rendering device, characterized in that: It includes an acquisition device, a memory and a processor. The acquisition device is used to acquire multi-view RGBD data and high-precision point cloud data. The memory includes a computer program that can be run thereon. When the processor executes the computer program, it implements the rendering method described in any one of claims 1 to 6.

9. A chip, characterized in that: It includes one or more processors, which are used to call and run a computer program from a memory, so that a device equipped with the chip executes the rendering method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer executable instructions are used to execute the rendering method according to any one of claims 1 to 6 when executed by a computer processor.

Images