A method and device for re-lighting rendering based on a three-dimensional Gaussian spattering model

Abstract

This application provides a relighting rendering method and apparatus for a 3D Gaussian splash model. The method includes: reconstructing a 3DGS model to obtain a Mesh model; setting the material of the Mesh model and adding the material to the Mesh model to obtain a materialized Mesh model; setting the lighting effect parameters of the scene and rendering the materialized Mesh model using a rendering engine to obtain a lighting image; performing radiative field self-illumination rendering on the 3DGS model to obtain a texture image, wherein the texture image and the lighting image have the same camera pose; and fusing the texture image and the lighting image to obtain a rendered image with lighting effects. This application reconstructs the 3DGS model into a Mesh model and integrates advanced lighting effects such as global illumination into the 3DGS model using the Mesh model, thereby obtaining an image with both advanced lighting effects and high-quality textures.

Description

Technical Field

[0001] This application relates to the field of 3D scene rendering technology, and in particular to a relighting rendering method and apparatus for 3D Gaussian splash models. Background Technology

[0002] For 3D Gaussian splash models, existing relighting rendering methods adjust the relighting effect by changing the color representation function of the 3D Gaussian ellipsoid or by combining depth information to estimate the lighting. These methods require redesigning a corresponding calculation method for different complex lighting effects, making it difficult to comprehensively cover the various existing lighting effects. Summary of the Invention

[0003] In view of this, this application provides a relighting rendering method and apparatus for three-dimensional Gaussian splash models to solve the above-mentioned technical problems.

[0004] In a first aspect, embodiments of this application provide a relighting rendering method for a 3D Gaussian splash model, including: The 3DGS model is reconstructed to obtain the Mesh model; Set the material of the Mesh model, and add the material to the Mesh model to obtain a Mesh model with material; Set the lighting effect parameters for the scene, and use the rendering engine to render the Mesh model with materials to obtain the lighting image; A radiative field self-illumination rendering was performed on the 3DGS model to obtain a texture image, wherein the texture image and the illumination image have the same camera pose; The texture image and the lighting image are fused to obtain a rendered image with lighting effects.

[0005] In one possible implementation, the 3DGS model is reconstructed to obtain a Mesh model; including: For any Gaussian ellipsoid in a 3DGS model, its corresponding Gaussian distribution function for: in, For random variables, The center point of the Gaussian ellipsoid, Let be the covariance matrix of the Gaussian ellipsoid. For transpose; Gaussian distribution function As a spatial density distribution function, multiple three-dimensional points that conform to the spatial density distribution function are randomly generated as part of the three-dimensional point cloud data; Generate 3D point cloud data based on partial 3D point cloud data of the Gaussian ellipsoid of all 3DGS models; The 3D point cloud data is processed using the Poisson surface reconstruction algorithm, Marching Cubes algorithm, or Ball Pivoting algorithm to obtain a Mesh model.

[0006] In one possible implementation, the material of the mesh model is set, and the material is added to the mesh model to obtain a mesh model with material; including: Create materials and set material parameters; the materials include: metallic materials, mirror materials, and rough materials; the material parameters include: color, metallicity, roughness, normal direction, transparency, and self-illumination intensity; Add the material to the Mesh model to obtain a Mesh model with material.

[0007] In one possible implementation, the 3DGS model is reconstructed to obtain a Mesh model; including: For any Gaussian ellipsoid in a 3DGS model, its corresponding Gaussian distribution function for: in, For random variables, The center point of the Gaussian ellipsoid, Let be the covariance matrix of the Gaussian ellipsoid. For transpose; Gaussian distribution function As a spatial density distribution function, multiple three-dimensional points that conform to the spatial density distribution function are randomly generated as part of the three-dimensional point cloud data; Color parameters are set for partial 3D point cloud data based on the spherical harmonic function of the Gaussian ellipsoid, resulting in colored partial 3D point cloud data. Generate colored 3D point cloud data based on all the colored partial 3D point cloud data. Using PointNet++ or Point Transformer models, semantic segmentation is performed on colored 3D point cloud data to obtain semantically meaningful 3D point cloud data. The semantic 3D point cloud data can be reconstructed using Poisson surface reconstruction algorithm, Marching Cubes algorithm, or Ball Pivoting algorithm to obtain semantic Mesh model.

[0008] In one possible implementation, the material of the mesh model is set, and the material is added to the mesh model to obtain a mesh model with material; including: Based on the semantically meaningful Mesh model, determine the material of the Mesh model; Add materials to the Mesh model to obtain a Mesh model with materials.

[0009] In one possible implementation, the rendering engine includes Open3D and Blender; Set the scene's lighting parameters, and use the rendering engine to render the mesh model with materials to obtain the lighting image; including: Set the scene's lighting mode: global illumination or direct illumination; set the global illumination parameters, including: global illumination mode, ray length, diffuse bounce count, specular bounce count, transmission bounce count, number of samples, and final acquisition quality; Create a light source and set its parameters; the light source includes: sunlight source, point light source, line light source, rectangular light source, and spotlight light source; the light source parameters include: light source intensity, light source position, light source direction angle, and light source color; Set the shadow parameters in the scene, including: shadow type, shadow mapping method, shadow map resolution, shadow edge anti-aliasing method, and number of shadow sampling points; Set the volumetric fog parameters in the scene, including: albedo, extinction coefficient, scattering coefficient, density scaling, and ambient light scaling; Based on the preset camera pose, the rendering engine renders the materialized Mesh model using the lighting mode, light source, shadow, and volumetric fog to obtain the lighting image.

[0010] In one possible implementation, the 3DGS model is rendered without lighting to obtain a texture image, including: At the preset camera pose, the 3DGS model is rendered using the 3DGS rendering tool library to obtain a texture image.

[0011] In one possible implementation, the texture image and the lighting image are fused to obtain a rendered image with lighting effects; including: Pixels of a rendered image with lighting effects pixel values The calculation formula is: in, pixels of a texture image pixel values, The weights of the texture image; pixels of a lit image pixel values, The weights of the texture image.

[0012] Secondly, embodiments of this application provide a relighting rendering apparatus for a three-dimensional Gaussian splash model, comprising: The reconstruction unit is used to reconstruct the 3DGS model to obtain the Mesh model; The material addition unit is used to set the material of the Mesh model and add the material to the Mesh model to obtain a Mesh model with material; The first processing unit is used to set the lighting effect parameters of the scene and use the rendering engine to render the Mesh model with materials to obtain the lighting image. The second processing unit is used to perform radiation field self-illumination rendering on the 3DGS model to obtain a texture image, wherein the texture image and the illumination image have the same camera pose. The fusion unit is used to fuse the texture image and the lighting image to obtain a rendered image with lighting effects.

[0013] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the methods of embodiments of this application.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the method of embodiments of this application.

[0016] This application reconstructs a 3DGS model into a Mesh model, and then uses the Mesh model to incorporate advanced lighting effects such as global illumination into the 3DGS model, thereby obtaining an image that has both advanced lighting effects and high-quality textures. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A schematic diagram illustrating how a 3DGS model provided in this application is projected onto a camera in different poses and rendered into a new perspective image; Figure 2A flowchart of a relighting rendering method for a 3D Gaussian splash model provided in an embodiment of this application; Figure 3 A flowchart illustrating the generation of a new relit image using dual-path rendering fusion, provided in an embodiment of this application; Figure 4 A flowchart illustrating the generation of a new relit image through dual-path rendering fusion after incorporating automatic material functionality, as provided in an embodiment of this application. Figure 5 A functional structure diagram of a relighting rendering apparatus for a 3D Gaussian splash model provided in an embodiment of this application; Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0022] 3D Gaussian Splatting (hereinafter referred to as 3DGS) is a geometric model used to represent 3D scenes / objects. 3DGS consists of a series of 3D Gaussian distribution functions. Each 3D Gaussian ellipsoid, in addition to having its position and shape described by different probability distribution functions, also possesses opacity α and color SH. Therefore, each Gaussian ellipsoid has four properties: position (mean of the 3D Gaussian distribution function), shape (covariance matrix of the 3D Gaussian distribution function), opacity, and color. These four properties are generally different for different Gaussian ellipsoids.

[0023] When an imaging viewpoint is selected, one can imagine a camera at a corresponding position and pose (hereinafter referred to as pose). Multiple three-dimensional Gaussian ellipsoids in space will be projected onto the camera's imaging plane to form an image. This image formation process is commonly known as rendering. For a pixel on the camera's imaging plane, the rendered color is obtained by superimposing the colors of all the Gaussian ellipsoids projected onto that pixel: the corresponding three-dimensional Gaussian distribution function of the Gaussian ellipsoid on the projection ray (… Figure 1 The probability of the color (as indicated by the blue arrow in the diagram) is multiplied by its opacity α, then multiplied by its color SH, and then the results from multiple Gaussian ellipsoids are summed. The color SH of each Gaussian ellipsoid is not a fixed value; SH is a spherical harmonic function SH(θ,φ) that varies with the angle. Therefore, the color seen by the camera may differ depending on its pose. This can be used to describe mirror-like surfaces, where the color (especially the light intensity) seen from different angles can be significantly different for mirror-like materials.

[0024] 3DGS models are primarily obtained through training. The learning process is similar to that of deep neural networks, but unlike deep learning which trains neural network weights, 3DGS mainly trains the number of Gaussian ellipsoids and the four properties of each Gaussian ellipsoid. SfM (Structure from Motion) is a classic method for estimating three-dimensional points and the corresponding camera pose from two-dimensional image points. Through training with input image sequences, 3DGS establishes the relationship between images captured in a specific scene and their corresponding camera poses. Therefore, after training, given a camera pose, 3DGS can generate a photograph taken in that pose. The training process can be considered as inputting "old images" and "old poses"; after training, inputting a "new pose" to 3DGS can generate a "new image." Automatic 3D reconstruction methods refer to any approach that constructs a 3D model from multi-view images captured by a camera. The most traditional method, such as using images taken by drones, employs only SfM and MVS (Multi-View Stereo) methods to estimate dense 3D point clouds and ultimately construct a 3D mesh model. However, unlike large-scale AI models that estimate 3D models from images, current estimation methods for large-scale AI models are not based on real physical models, and therefore lack realism and reliability in many aspects.

[0025] 3DGS has advantages in 3D modeling and generating images from new perspectives, but the lighting environment of the newly generated images is difficult to change. For example, if the image was taken on a cloudy day, it is difficult to generate a new image of a sunny day after training. This is because the image information input for training is the color of the cloudy lighting, so the color of the resulting 3D Gaussian ellipsoid is also the color of the cloudy lighting. To generate new images under different lighting conditions, it is necessary to change the color of the already trained 3D Gaussian ellipsoid, which brings many troubles, especially when trying to change the lighting environment of the original shooting scene with new and complex light sources. This invention mainly addresses how to efficiently change the lighting to generate new images under new lighting conditions.

[0026] There are several techniques available that can alter lighting conditions (often called relighting) to generate new images under new lighting conditions. Three of these techniques are highlighted below.

[0027] (1) Spherical Harmonic Function (SH) Decoding Method. This method directly alters the angular distribution of colors in the pre-trained three-dimensional Gaussian ellipsoids. The original color angular distribution is represented by spherical harmonic functions. When relighting, the coefficients and other parameters of these spherical harmonic functions are adjusted to describe the new lighting environment. However, this SH decoding method is difficult to simulate lighting effects such as shadows and metallic highlights because the colors expressed by SH in the three-dimensional Gaussian ellipsoids only contain the lighting information received by the ellipsoids, making it difficult to simulate information such as occlusion and reflection. Moreover, the SH decoding method is difficult to simulate dynamically changing lighting environments and requires continuous calculation of new spherical harmonic function parameters.

[0028] Other methods employ more complex functions instead of spherical harmonics to characterize the color of the 3D Gaussian ellipsoid, and combine this with more complex rendering steps to generate the lighting for the final image. Regardless, these methods continuously adjust the color representation function of the 3D Gaussian ellipsoid, whether or not it is a spherical harmonic.

[0029] (2) GASLIGHT method. The light source is also modeled using a 3D Gaussian ellipsoid, and then the Gaussian ellipsoid of the light source is projected onto the Gaussian ellipsoid of the scene to change its original color, thereby adjusting the lighting. The final newly generated image is based on the new color of the Gaussian ellipsoid affected by the light source, and the lighting adjustment occurs on the Gaussian ellipsoid. It can be said that compared to the first method, it indirectly changes the color representation function of the Gaussian ellipsoid (by projecting the Gaussian ellipsoid of the light source).

[0030] (3) Depth map-based rendering. Estimate the depth of objects in the 3D scene, and then calculate various lighting effects such as shadows by using depth information and contrast.

[0031] Given that all lighting effects on existing rendering platforms are based on mesh models, using mesh model relighting to adjust lighting effects eliminates the need to develop specific calculation methods for different lighting effects. Other 3DGS relighting methods, however, relight on the 3DGS model itself, redesigning the color representation function of the 3D Gaussian ellipsoid to achieve different lighting effects. This requires complex design and struggles to comprehensively cover all lighting effects. This application, by utilizing the mesh model, achieves existing lighting effects without requiring redesign.

[0032] Compatibility and Promotion Difficulty: Given that all existing game and film platforms are based on mesh rendering, the coupling between modules in this application's mesh-based lighting re-blending 3DGS process is relatively low. Unlike other methods that require significant algorithmic modifications to the underlying rendering platform, this approach only performs fusion at a higher level—the final image rendering result—greatly reducing the difficulty of integrating 3DGS with existing rendering platforms. It also allows for the easy addition of new mesh models to the scene. The impact of these new mesh models on the original scene's lighting effects can be directly calculated by the rendering platform without any additional consideration. Other methods of relighting on 3DGS models require an additional calculation method to account for the lighting effects of newly added mesh models.

[0033] Considering execution efficiency: Directly relighting a 3DGS model requires additional color calculations and compositing. The computational complexity for different lighting effects is difficult to estimate, and the computational cost is at least on the order of magnitude of the original 3DGS. This application utilizes mesh rendering, where even the most complex lighting effects are still on the order of magnitude of mesh rendering. Furthermore, the computational cost of mesh rendering is significantly lower than that of 3DGS in most cases.

[0034] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0035] Current mainstream 3D rendering platforms, including those for game and film production, are primarily based on 3D mesh models for rendering. All rendering methods are optimized for mesh models, including various lighting effects such as global illumination, lighting for complex materials (metal, human skin), and high-quality shadows. However, to obtain high-quality mesh models, artists need to manually construct the meshes. Mesh models obtained directly from automated 3D modeling are often insufficiently precise, resulting in poor-quality rendered images. 3DGS models, on the other hand, can render high-quality images. This application combines the advantages of both to simultaneously obtain complex lighting effects and high-quality image textures; that is, relighting the mesh model to obtain adjusted lighting, and obtaining fine image textures on the 3DGS model.

[0036] like Figure 2 As shown in the embodiment of this application, a relighting rendering method for a 3D Gaussian splash model is provided, including: Step 101: Reconstruct the 3DGS model to obtain the Mesh model; Step 102: Set the material of the Mesh model and add the material to the Mesh model to obtain a Mesh model with material; Step 103: Set the lighting effect parameters for the scene, and use the rendering engine to render the Mesh model with materials to obtain the lighting image; Step 104: Perform radiation field self-illumination rendering on the 3DGS model to obtain a texture image, wherein the texture image and the illumination image have the same camera pose; Step 105: Fuse the texture image and the lighting image to obtain a rendered image with lighting effects.

[0037] A significant technical problem with existing 3DGS model relighting methods is their inability to achieve global illumination and its detailed settings. This embodiment addresses this by reconstructing the 3DGS model as a mesh model, which allows for the integration of advanced lighting effects like global illumination into the 3DGS model.

[0038] The method in this embodiment merges the rendering result of 3DGS with the rendering result after relighting on the Mesh, thereby directly utilizing various existing lighting effects and then merging them with the generated image of 3DGS to obtain an image that has both advanced lighting effects and high-quality textures.

[0039] In some embodiments, the 3DGS model is reconstructed to obtain a Mesh model; including: For any Gaussian ellipsoid in a 3DGS model, its corresponding Gaussian distribution function for: in, For random variables, The center point of the Gaussian ellipsoid, Let be the covariance matrix of the Gaussian ellipsoid. For transpose; Gaussian distribution function As a spatial density distribution function, multiple three-dimensional points that conform to the spatial density distribution function are randomly generated as part of the three-dimensional point cloud data; Generate 3D point cloud data based on partial 3D point cloud data of the Gaussian ellipsoid of all 3DGS models; The 3D point cloud data is processed using the Poisson surface reconstruction algorithm, Marching Cubes algorithm, or Ball Pivoting algorithm to obtain a Mesh model.

[0040] Furthermore, the covariance matrix of the 3D Gaussian ellipsoids in the 3DGS model can be used to help estimate the normal vectors of the 3D point cloud, thereby better generating the mesh model; alternatively, the center point of each ellipsoid in 3DGS can be used as the mesh vertex, and the distance between the centers of multiple ellipsoids and the covariance matrix of the ellipsoids can determine whether these mesh vertices are connected. Information such as the transparency and probability of multiple 3D Gaussian ellipsoids can also be used to correct the position of the 3D point cloud and filter out noisy 3D points, etc.

[0041] In some embodiments, the material of the mesh model is set, and the material is added to the mesh model to obtain a mesh model with material; including: Create materials and set material parameters; the materials include: metallic materials, mirror materials, and rough materials; the material parameters include: color, metallicity, roughness, normal direction, transparency, and self-illumination intensity; Add the material to the Mesh model to obtain a Mesh model with material.

[0042] In some embodiments, the 3DGS model is reconstructed to obtain a Mesh model; including: For any Gaussian ellipsoid in a 3DGS model, its corresponding Gaussian distribution function for: in, For random variables, The center point of the Gaussian ellipsoid, Let be the covariance matrix of the Gaussian ellipsoid. For transpose; Gaussian distribution function As a spatial density distribution function, multiple three-dimensional points that conform to the spatial density distribution function are randomly generated as part of the three-dimensional point cloud data; Color parameters are set for partial 3D point cloud data based on the spherical harmonic function of the Gaussian ellipsoid, resulting in colored partial 3D point cloud data. Generate colored 3D point cloud data based on all the colored partial 3D point cloud data. Using PointNet++ or Point Transformer models, semantic segmentation is performed on colored 3D point cloud data to obtain semantically meaningful 3D point cloud data. The semantic 3D point cloud data can be reconstructed using Poisson surface reconstruction algorithm, Marching Cubes algorithm, or Ball Pivoting algorithm to obtain semantic Mesh model.

[0043] In some embodiments, the material of the mesh model is set, and the material is added to the mesh model to obtain a mesh model with material; including: Based on the semantically meaningful Mesh model, determine the material of the Mesh model; Add materials to the Mesh model to obtain a Mesh model with materials.

[0044] In some embodiments, the rendering engine includes Open3D and Blender; Set the scene's lighting parameters, and use the rendering engine to render the mesh model with materials to obtain the lighting image; including: Set the scene's lighting mode: global illumination or direct illumination; set the global illumination parameters, including: global illumination mode, ray length, diffuse bounce count, specular bounce count, transmission bounce count, number of samples, and final acquisition quality; Create a light source and set its parameters; the light source includes: sunlight source, point light source, line light source, rectangular light source, and spotlight light source; the light source parameters include: light source intensity, light source position, light source direction angle, and light source color; Set the shadow parameters in the scene, including: shadow type, shadow mapping method, shadow map resolution, shadow edge anti-aliasing method, and number of shadow sampling points; Set the volumetric fog parameters in the scene, including: albedo, extinction coefficient, scattering coefficient, density scaling, and ambient light scaling; Based on the preset camera pose, the rendering engine renders the materialized Mesh model using the lighting mode, light source, shadow, and volumetric fog to obtain the lighting image.

[0045] Currently, all rendering engines are optimized for mesh models, allowing all lighting effects to be applied to them, such as global illumination and dynamic lights. Various shadow effects can then be achieved. To consider only the intensity of the light and eliminate interference from other colors, a white material is typically used. More diverse lighting effects are obtained by adjusting the material type: for example, using a metallic material for metallic highlights, a specular material for specular reflection, and a rough material for diffuse reflection. After setting the mesh material (color and type) and arranging the light sources in the scene (sunlight, point lights, rectangular lights, etc.), the corresponding new image can be directly rendered at the target pose, such as... Figure 3 As shown. This new image includes lighting effects, but because it uses a white mesh, there is no image texture.

[0046] For example, we can segment the image into patches representing the road surface, leaves, cars, and pedestrians. This results in multiple semantically meaningful mesh models, such as one mesh representing the road surface, and others representing leaves, cars, and pedestrians. Then, in a rendering engine like Open3D, we assign corresponding material types to these semantically meaningful meshes (the parameters are only type-related; the color remains white as before): the road surface is assigned a concrete material (rough, weak reflection), the leaves a leaf material (medium smooth, medium reflection), the cars a metal material (medium smooth, strong reflection), the glass a mirror material (smooth and strong reflection), and so on. Alternatively, we can add more complex material settings beyond simple roughness and reflectivity, such as transparency, self-illumination, and double reflection. This achieves automatic material type assignment programmatically, allowing us to render images with more realistic and complex lighting effects, which can then be blended with the texture images of the 3DGS model. Figure 4 As shown.

[0047] In some embodiments, the 3DGS model is rendered without lighting to obtain a texture image, including: At the preset camera pose, the 3DGS model is rendered using the 3DGS rendering tool library to obtain a texture image.

[0048] Specifically, it uses the 3DGS rendering library provided by the official 3DGS library. This library is based on PyTorch + CUDA and has a dedicated rendering workflow. This rendering library is included in the overall 3DGS library suite and is a part of 3DGS. It can be compared to the CUDA and cuDNN libraries used for inference in deep learning, but it is a dedicated rendering library developed by the inventors of 3DGS.

[0049] Alternatively, you can use rendering libraries developed by other third-party developers, such as the gsplat rendering library developed by the Nerfstudio team, which is also basically based on CUDA.

[0050] In some embodiments, the texture image and the lighting image are fused to obtain a rendered image with lighting effects; including: Pixels of a rendered image with lighting effects pixel values The calculation formula is: in, pixels of a texture image pixel values, The weights of the texture image; pixels of a lit image pixel values, The weights of the texture image.

[0051] Based on the same inventive concept, this application provides a relighting rendering apparatus for a 3D Gaussian splash model, see reference. Figure 5 As shown, the relighting rendering apparatus 200 for a 3D Gaussian splash model provided in this application embodiment includes at least: Reconstruction unit 201 is used to reconstruct the 3DGS model to obtain the Mesh model; The material adding unit 202 is used to set the material of the Mesh model and add the material to the Mesh model to obtain a Mesh model with material; The first processing unit 203 is used to set the lighting effect parameters of the scene and render the Mesh model with materials using the rendering engine to obtain the lighting image. The second processing unit 204 is used to perform radiation field self-illumination rendering on the 3DGS model to obtain a texture image, wherein the texture image and the illumination image have the same camera pose. The fusion unit 205 is used to fuse the texture image and the lighting image to obtain a rendered image with lighting effects.

[0052] It should be noted that the principle of the relighting rendering device 200 for three-dimensional Gaussian splash models provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the relighting rendering device 200 for three-dimensional Gaussian splash models provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0053] Based on the same inventive concept, embodiments of this application also provide an electronic device, such as... Figure 6As shown, it includes a memory and a processor. The memory stores an executable program, and the processor executes the executable program to implement the steps of the relighting rendering method for a three-dimensional Gaussian splash model provided in the above embodiment.

[0054] The aforementioned processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] Since the electronic device described in this application embodiment is an electronic device equipped with a memory for implementing the relighting rendering method for a three-dimensional Gaussian splash model disclosed in this application embodiment, those skilled in the art can understand the structure and variations of the electronic device described in this application embodiment based on the relighting rendering method for a three-dimensional Gaussian splash model described in this application embodiment, and therefore will not be described again here.

[0056] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is run by a processor, it implements the steps of the relighting rendering method for a three-dimensional Gaussian splash model provided in the above embodiments.

[0057] The storage medium in this embodiment may be included in an electronic device; or it may exist independently and not be assembled into an electronic device. The storage medium carries one or more computer programs, which, when executed, implement the steps of the relighting rendering method for a three-dimensional Gaussian splash model provided in the above embodiment.

[0058] It should be understood that the various solutions in this embodiment have the same technical effects as those in the above method embodiments, and will not be repeated here.

[0059] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. Optionally, specific examples in this embodiment can refer to the examples described in any embodiment of this application, which will not be repeated here. Obviously, those skilled in the art should understand that the various modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular hardware and software combination.

[0060] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the relighting rendering method for a three-dimensional Gaussian splash model provided in the above embodiments.

[0061] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions targeted in the blocks may occur in a different order than those targeted in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0062] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0063] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

Claims

1. A relighting rendering method for a 3D Gaussian splash model, characterized in that, include: The 3DGS model is reconstructed to obtain the Mesh model; Set the material of the Mesh model, and add the material to the Mesh model to obtain a Mesh model with material; Set the lighting effect parameters for the scene, and use the rendering engine to render the Mesh model with materials to obtain the lighting image; A radiative field self-illumination rendering was performed on the 3DGS model to obtain a texture image, wherein the texture image and the illumination image have the same camera pose; The texture image and the lighting image are fused to obtain a rendered image with lighting effects.

2. The method according to claim 1, characterized in that, The 3DGS model is reconstructed to obtain the Mesh model; including: For any Gaussian ellipsoid in a 3DGS model, its corresponding Gaussian distribution function for: in, For random variables, The center point of the Gaussian ellipsoid, Let be the covariance matrix of the Gaussian ellipsoid. For transpose; Gaussian distribution function As a spatial density distribution function, multiple three-dimensional points that conform to the spatial density distribution function are randomly generated as part of the three-dimensional point cloud data; Generate 3D point cloud data based on partial 3D point cloud data of the Gaussian ellipsoid of all 3DGS models; The 3D point cloud data is processed using the Poisson surface reconstruction algorithm, Marching Cubes algorithm, or Ball Pivoting algorithm to obtain a Mesh model.

3. The method according to claim 2, characterized in that, Setting the material of the mesh model and adding the material to the mesh model results in a mesh model with materials; including: Create materials and set material parameters; the materials include: metallic materials, mirror materials, and rough materials; the material parameters include: color, metallicity, roughness, normal direction, transparency, and self-illumination intensity; Add the material to the Mesh model to obtain a Mesh model with material.

4. The method according to claim 1, characterized in that, The 3DGS model is reconstructed to obtain the Mesh model; including: For any Gaussian ellipsoid in a 3DGS model, its corresponding Gaussian distribution function for: in, For random variables, The center point of the Gaussian ellipsoid, Let be the covariance matrix of the Gaussian ellipsoid. For transpose; Gaussian distribution function As a spatial density distribution function, multiple three-dimensional points that conform to the spatial density distribution function are randomly generated as part of the three-dimensional point cloud data; Color parameters are set for partial 3D point cloud data based on the spherical harmonic function of the Gaussian ellipsoid, resulting in colored partial 3D point cloud data. Generate colored 3D point cloud data based on all the colored partial 3D point cloud data. Using PointNet++ or Point Transformer models, semantic segmentation is performed on colored 3D point cloud data to obtain semantically meaningful 3D point cloud data. The semantic 3D point cloud data can be reconstructed using Poisson surface reconstruction algorithm, Marching Cubes algorithm, or Ball Pivoting algorithm to obtain semantic Mesh model.

5. The method according to claim 4, characterized in that, Setting the material of the mesh model and adding the material to the mesh model results in a mesh model with materials; including: Based on the semantically meaningful Mesh model, determine the material of the Mesh model; Add materials to the Mesh model to obtain a Mesh model with materials.

6. The method according to claim 3 or 5, characterized in that, The rendering engines include Open3D and Blender; Set the scene's lighting parameters, and use the rendering engine to render the mesh model with materials to obtain the lighting image; including: Set the scene's lighting mode: global illumination or direct illumination; set the global illumination parameters, including: global illumination mode, ray length, diffuse bounce count, specular bounce count, transmission bounce count, number of samples, and final acquisition quality; Create a light source and set its parameters; the light source includes: sunlight source, point light source, line light source, rectangular light source, and spotlight light source; the light source parameters include: light source intensity, light source position, light source direction angle, and light source color; Set the shadow parameters in the scene, including: shadow type, shadow mapping method, shadow map resolution, shadow edge anti-aliasing method, and number of shadow sampling points; Set the volumetric fog parameters in the scene, including: albedo, extinction coefficient, scattering coefficient, density scaling, and ambient light scaling; Based on the preset camera pose, the rendering engine renders the materialized Mesh model using the lighting mode, light source, shadow, and volumetric fog to obtain the lighting image.

7. The method according to claim 1, characterized in that, The 3DGS model is rendered without lighting to obtain a texture image, including: At the preset camera pose, the 3DGS model is rendered using the 3DGS rendering tool library to obtain a texture image.

8. The method according to claim 1, characterized in that, The texture image and the lighting image are fused to obtain a rendered image with lighting effects; including: Pixels of a rendered image with lighting effects pixel values The calculation formula is: in, pixels of a texture image pixel values, The weights of the texture image; pixels of a lit image pixel values, The weights of the texture image.

9. A relighting rendering device for a 3D Gaussian splash model, characterized in that, include: The reconstruction unit is used to reconstruct the 3DGS model to obtain the Mesh model; The material addition unit is used to set the material of the Mesh model and add the material to the Mesh model to obtain a Mesh model with material; The first processing unit is used to set the lighting effect parameters of the scene and use the rendering engine to render the Mesh model with materials to obtain the lighting image. The second processing unit is used to perform radiation field self-illumination rendering on the 3DGS model to obtain a texture image, wherein the texture image and the illumination image have the same camera pose. The fusion unit is used to fuse the texture image and the lighting image to obtain a rendered image with lighting effects.

10. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-8.

Images