Gaussian spatter-based three-dimensional scene relighting and self-emission editing method and device, and storage medium
By constructing a 3D Gaussian model and optimizing its self-illuminating properties, the problems of unreal-time light source editing and lack of physical realism in 3D scene reconstruction were solved. Controllable editing of self-illuminating objects and physically consistent indirect lighting were achieved, improving the realism of the reconstruction and editing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV
- Filing Date
- 2026-03-27
- Publication Date
- 2026-06-26
Smart Images
Figure CN122289504A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision and 3D scene reconstruction technology, specifically to a method, apparatus, and storage medium for 3D scene relighting and self-illumination editing based on Gaussian splashing. Background Technology
[0002] In 3D scene reconstruction or editing tasks, it is often necessary to reconstruct and edit complex 3D scenes containing locally self-illuminating objects such as indoor lamps, computer screens, and outdoor neon billboards. This is especially true for scenes that require independent control of these light sources, such as turning lights on and off, or changing the color or intensity of the light. Existing 3D reconstruction and inverse rendering technologies face a dilemma between efficiency and editability when handling such scenes, and cannot meet the needs of real-time interaction. Specifically, the following key issues exist: Limitations of lighting models: Most existing methods assume that the scene is only illuminated by ambient light (Environment Map) at infinity, ignoring local light sources within the scene, which makes it impossible to accurately simulate the light and shadow interaction between the lights and the surrounding objects.
[0003] The limitations of Neural Radiance Fields (NeRF) methods: Although they can achieve lighting decoupling, they rely on voxel ray tracing, which has extremely high computational costs, slow training and rendering speeds, and cannot support real-time editing.
[0004] The shortcomings of existing 3D Gaussian Splatting (3DGS) methods include: while offering high efficiency in real-time rendering, they lack explicit modeling of self-illumination properties. They often incorrectly "bake" the illumination effect into the object's surface color, assuming the light bulb is inherently bright white rather than emitting light, making it impossible to independently edit the light source—for example, changing a red light to green without disrupting object textures. Because Gaussian primitives typically mix surface reflectivity and illumination, it's difficult to adjust the color or intensity of the luminescent object independently; for example, dimming a light source without affecting the object's color. Furthermore, the lack of indirect illumination means that after editing the light source, the shadows and shading of surrounding objects don't change accordingly, resulting in a lack of physical realism.
[0005] In summary, current 3D scene reconstruction technologies suffer from technical problems such as the inability to edit lighting and self-illuminating objects in real time and independently, and a lack of physical realism after editing the light sources. Summary of the Invention
[0006] The purpose of this invention is to provide a method, apparatus, and storage medium for relighting and self-illuminating editing of 3D scenes based on Gaussian splashing, in order to solve the technical problems in existing 3D scene reconstruction where lighting and self-illuminating bodies cannot be edited in real time and independently, and where the edited light sources lack physical realism.
[0007] In a first aspect, the present invention provides an autonomous driving scene reconstruction method based on spatiotemporal consistency constraints, comprising the following steps: Images of the luminous object not emitting light and images of the luminous object emitting light are obtained from multiple perspectives of the target scene, and luminous image sets and non-luminous image sets are constructed. Construct a 3D Gaussian model of the target scene and train the 3D Gaussian model based on a set of non-illuminated images to obtain the geometric and material information of the 3D Gaussian model; The 3D Gaussian model is optimized based on the luminescent and non-luminescent image sets to obtain the Gaussian point set with self-luminescent properties. A self-illuminating map is obtained by rendering based on geometric and material information and Gaussian point sets with self-illumination properties, and the lighting effect of the self-illuminating map is modified according to the Gaussian point sets with self-illumination properties.
[0008] The significant advantages of this invention are as follows: This solution achieves controllable self-illumination editing. By explicitly modeling the self-illumination properties of the 3D Gaussian model, users can freely adjust the color and intensity of the light source, achieving true relighting and solving the problem of traditional 3DGS baking light into color. By optimizing the geometric and material information and the Gaussian point set of the self-illumination properties of the 3D Gaussian model respectively, after editing the light source, other objects in the scene can correctly reflect the new light color and intensity, greatly improving realism and achieving physically consistent indirect lighting.
[0009] Furthermore, the step of training a 3D Gaussian model based on a set of non-illuminated images to obtain the geometric and material information of the 3D Gaussian model includes: Calculate the weighted average depth of visible Gaussians on each ray in a 3D Gaussian model based on a set of non-emitting images; The pseudo-normal map is calculated based on the weighted average depth of the visible Gaussians on each ray, and the normal map is obtained by supervising the rendering of the normals of each Gaussian point based on the pseudo-normal map. Albedo map, roughness map and metallicity map are generated based on 3D Gaussian model, and the geometric and material information of 3D Gaussian model is obtained by combining normal map, albedo map, roughness map and metallicity map.
[0010] The beneficial effects of the above technical solution are as follows: By calculating the weighted average depth of visible Gaussian planes to generate a pseudo-normal map, and using this to supervise the generation of normals and various physical property maps (albedo map, roughness map, and metallicity map), the deficiency of the original 3D Gaussian model (3DGS) in lacking explicit geometric expression is effectively compensated for. This step achieves the effective extraction and decoupling of the accurate 3D geometric structure and physical material properties of the scene, laying a precise data foundation for subsequent physically based realistic lighting calculations.
[0011] Furthermore, after the step of generating the albedo map, roughness map, and metallicity map based on the 3D Gaussian model, the method further includes: Construct a physically based rendering environment lighting model, and obtain environment maps based on the physically based rendering environment lighting model; Predictive images are generated based on normal maps, albedo maps, roughness maps, metallicity maps, and environment maps; The photometric loss is calculated by comparing the predicted image with a set of non-illuminated images, and the albedo, roughness, metallicity, and environment map parameters of each Gaussian point are updated based on the photometric loss.
[0012] The beneficial effects of the above technical solution are as follows: By introducing a physically based rendering environment lighting model and combining photometric loss to compare and reverse-optimize the generated image with the real non-illuminating image, accurate separation of ambient lighting and the inherent material parameters of objects is achieved. This not only improves the physical accuracy of material properties such as albedo and roughness at each Gaussian point, but also ensures that the scene still possesses a high degree of intrinsic realism after the self-illuminating body lighting is stripped away.
[0013] Further, the step of optimizing the 3D Gaussian model based on the luminescent image set and the non-luminescent image set to obtain the self-luminescent attribute Gaussian point set includes: The brightness difference of the self-luminous body was calculated based on the luminous image set and the non-luminous image set; A self-illumination attribute is constructed for each Gaussian point, and the predicted self-illumination image is rendered based on the self-illumination attribute. A mask loss function is constructed based on the brightness difference results of the self-illuminating body and the self-illuminating image, and the self-illuminating properties are optimized based on the predicted self-illuminating image and the mask loss function; Construct a Gaussian point set of self-illuminating properties based on the optimized self-illuminating properties.
[0014] The beneficial effects of the above technical solution are as follows: by utilizing image difference technology and mask loss function, it is possible to accurately locate and extract self-illuminating objects in the scene, and assign explicit self-illuminating properties to specific Gaussian points. This step effectively avoids the confusion between self-illuminating information and the basic material information of the scene, realizing independent decoupling and high-precision modeling of light source objects, which is a core prerequisite for subsequent independent editing of light sources.
[0015] Furthermore, the self-illuminating map obtained by rendering based on geometric and material information and the Gaussian point set of self-illumination properties includes: Calculate the step size for each shading pixel to move towards the luminous Gaussian point; Calculate the ray visibility of each shading pixel as it moves toward the emitting Gaussian point; A rendering equation is constructed based on the visibility of light, and the distance attenuation and bidirectional reflection distribution of each visible light point to a pixel are calculated based on the rendering equation. The self-illumination effect is calculated based on the Gaussian point set of self-illumination properties and the rendering equation, and the self-illumination map is rendered based on the self-illumination effect.
[0016] The beneficial effects of the above technical solution are as follows: Based on stepped visibility, distance attenuation, and bidirectional reflection distribution functions, the rendering equation can be constructed to strictly follow the laws of physical light propagation and accurately calculate the illumination effect of the self-illuminating object on different surrounding material surfaces (i.e., indirect lighting). This gives the heavy lighting effect a high degree of physical realism and ensures the correct interaction between light and scene materials.
[0017] Furthermore, the calculation of the ray visibility during the stepping process of each tinted pixel toward the emitting Gaussian point includes: Multiple sampling points are obtained by stepping along the direction of the light rays during the stepping process from the color pixel to the luminous Gaussian point; The depth buffer value and cumulative opacity of the Gaussian field for each sampling point are obtained by rendering the trained 3D Gaussian model. The overall occlusion value for each sampling point is calculated based on the depth buffer value and the cumulative opacity of the Gaussian field. The light visibility of the tinted pixel during its step towards the luminous Gaussian point is calculated by accumulating the combined occlusion values of all sampling points.
[0018] The beneficial effects of the above technical solution are as follows: by fully utilizing the depth buffer value and accumulated opacity information of the trained 3D Gaussian for multi-point sampling and comprehensive occlusion calculation, it can accurately simulate the spatial occlusion relationship during light propagation. This method can adaptively generate physically correct soft or hard shadows, significantly enhancing the spatial three-dimensionality and lighting realism of heavily lit scenes.
[0019] Furthermore, the calculation of the self-illumination effect based on the Gaussian point set of self-illumination properties and the rendering equation includes: The self-illumination term is obtained by rendering Gaussian point sets based on geometric and material information and self-illumination properties. Indirect lighting is calculated based on the rendering equation; The self-illuminating effect is obtained by adding the self-illuminating term and the indirect illumination.
[0020] The beneficial effects of the above technical solution are as follows: by physically superimposing the direct light emission (self-emission) of the light source itself with the illumination effect of the light source on the environment (indirect lighting), the overall visual performance of the self-emission body in the scene is fully and completely restored. This ensures a high degree of physical consistency between the high brightness of the light source itself and the diffuse halo effect it creates in the scene.
[0021] Furthermore, when modifying the lighting effect of the self-illumination map, the input adjustment target is obtained to modify the self-illumination property, and the self-illumination map is rendered according to the modified self-illumination property.
[0022] The beneficial effects of the above technical solution are: it gives users a high degree of editing freedom and real-time interactive capabilities. Thanks to the complete decoupling of the pre-processed material and self-illumination properties, when users modify the color, intensity and other self-illumination properties of the target light source, they do not need to retrain the entire 3D Gaussian model. They can render physically correct and globally consistent relighting effects in real time, which completely breaks through the technical bottleneck of the lighting being "baked" and solidified in traditional methods.
[0023] Secondly, the present invention also provides a three-dimensional scene relighting and self-illuminating editing device based on Gaussian splashing, including a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned three-dimensional scene relighting and self-illuminating editing method based on Gaussian splashing.
[0024] Thirdly, the present invention also provides a computer-readable storage medium containing a computer program, wherein the computer program is stored thereon, and when the computer program is executed by one or more processors, the above-described method for relighting and self-illuminating a three-dimensional scene based on Gaussian splashing is implemented. Attached Figure Description
[0025] Figure 1 This is a flowchart of the three-dimensional scene relighting and self-illuminating editing method based on Gaussian splashing in an embodiment of the present invention; Figure 2 This is a diagram illustrating the lighting effect of a modified self-emission image in an embodiment of the present invention. Figure 3 This is a comparison diagram of the present invention and the prior art when modifying the color of the light source. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.
[0027] See appendix Figure 1 The illustrated method for relighting and self-illumination editing of a 3D scene based on Gaussian splashing includes the following steps: S1. Acquire images of the target scene from multiple perspectives: images of the luminous elements not emitting light and images of the luminous elements emitting light. Construct a luminous image set based on the luminous images and a non-luminous image set based on the non-luminous images. Both the luminous and non-luminous image sets contain camera pose information at the time of image acquisition. Specifically, when acquiring images of the target scene's luminous elements not emitting light and images of the luminous elements emitting light, images of the luminous components in the target scene with their light sources turned off or on can be acquired using image acquisition devices such as cameras. Combine images of the luminous components turned off from multiple perspectives in the target scene to obtain a non-luminous image set, and combine images of the luminous components turned on from multiple perspectives in the target scene to obtain a luminous image set. The specific details of image acquisition and image set division in the target scene are existing technologies and will not be elaborated here.
[0028] S2. Construct a 3D Gaussian model of the target scene and train the 3D Gaussian model based on a set of non-illuminated images to obtain the geometric and material information of the 3D Gaussian model. When constructing the 3D Gaussian model of the target scene, each Gaussian point in the 3D Gaussian model includes standard Gaussian properties and explicit physical properties. The standard Gaussian properties include the position, covariance, and opacity of the Gaussian point, while the explicit physical properties include: Normals are stored as independent, optimizable properties, rather than being implicitly derived solely from geometry; Basic color and albedo are used to represent the inherent color of an object's surface; Roughness is used to describe the degree of microscopic smoothness of an object's surface; Metallicity is used to describe the metallic optical characteristics of an object's surface. The self-illumination property is used to represent the light-emitting property of an object.
[0029] Specifically, the calculation of geometry and material information includes the following steps: A1. Calculate the weighted average depth of the visible Gaussian rays on each ray in the 3D Gaussian model of the target scene based on a set of non-emitting images:
[0030]
[0031] In the formula, The depth is the weighted average depth of the visible Gaussian. This represents the number of Gaussian points on the light source that participate in the rendering. For the first The depth values of a visible Gaussian point in the camera coordinate system For the first The normalized weight of each Gaussian point relative to the corresponding pixel of the ray is typically determined by opacity and transmittance. For the light to reach the first Cumulative transmittance at a Gaussian point For the first Opacity of one Gaussian point For the first light The cumulative transmittance at each Gaussian point involved in the rendering; in the summation term of the denominator, it is used to calculate the total contribution of all visible points along the entire ray to the pixel color. For the first Opacity of one Gaussian point.
[0032] A2. A pseudo-normal map is calculated based on the weighted average depth of the visible Gaussian rays along each ray. The normals at each Gaussian point are then rendered under supervision based on the pseudo-normal map, and the normal map is obtained from the normals at each Gaussian point. Specifically, a normal vector is explicitly stored at each 3D Gaussian point. These normals are rendered to obtain the rendered normal map, and the rendered normal map is required to be as consistent as possible with the pseudo-normal map calculated based on depth. The specific details of calculating the pseudo-normal map based on depth information and rendering the normals at each Gaussian point are existing technologies and will not be elaborated upon here.
[0033] When rendering normals, the smoothness of the normals is constrained using the total variational loss function, which is expressed as follows:
[0034]
[0035] In the formula, Optimize the total loss function for the normal vector. The normal prediction loss measures the degree of difference between the rendered normal map and a reference normal map (such as a pseudo-normal map). These are variational regularization weights used to balance the ratio between the loss of the normal data term and the loss of the smoothing term. The Total Variation Loss is used to constrain the continuity of normals between adjacent Gaussian points or pixels, thereby suppressing noise and enhancing surface smoothness. The square of the L2 norm. The predicted normal vectors obtained for rendering are normal information generated by the 3D Gaussian model through the current geometric properties. Using the reference normal vector, the geometric normal calculated based on the weighted average depth map is used as the ground truth to supervise the properties of the Gaussian point.
[0036] A3. Generate albedo, roughness, and metallicity maps based on a 3D Gaussian model. Specifically, when generating the albedo, roughness, and metallicity maps, an improved differentiable Gaussian rasterizer is used to project Gaussian primitives from the scene onto the screen space. Unlike traditional 3DGS which directly mixes colors, this step involves modifying the Gaussian properties... The mixture is combined with the normal map, albedo map, roughness map and metallicity map to construct a G-buffer containing screen space geometry and material information; the specific content of the albedo map, roughness map and metallicity map of the 3D Gaussian model generated by the differentiable Gaussian rasterizer is existing technology and will not be described in detail here.
[0037] A4. Construct a physically based rendering environment lighting model, and obtain a learnable environment map based on the physically based rendering environment lighting model to represent the global incident radiance of the scene in a non-emitting state; the specific content of generating the environment map through the physically based rendering environment lighting model is existing technology and will not be described in detail here.
[0038] A5. Generate a predicted image based on the geometry buffer and environment map. Specifically, the predicted image is synthesized by applying rendering equations pixel by pixel in screen space based on the geometry buffer and environment map. When generating the predicted image, the emitted radiance is calculated based on the albedo map, roughness map, metallicity map, and normal map.
[0039]
[0040]
[0041] In the formula, for Radiance emitted in the line of sight direction, This refers to diffuse reflected irradiance. This is an albedo map. The irradiance map stores data along the normal. The total amount of incident ambient light is the directional integral. For specular reflection of emitted irradiance, The pre-filtered environment map performs pre-convolution processing on the ambient light according to different roughness levels to obtain the specular reflection direction. The intensity of the incident light, This is an element-wise multiplication operation. This is a BRDF pre-integration lookup table that stores the integration results of Fresnel and geometric occlusion terms under different roughness and viewing angles. The unit normal vector, To observe the direction, Unit reflection vector; A6. Calculate the photometric loss by comparing the predicted image with the non-illuminated image set, and update the albedo, roughness, metallicity and environment map parameters of each Gaussian point according to the photometric loss.
[0042] The photometric loss function is:
[0043] In the formula, For luminance loss, for Norm, To predict the image, An image of a luminous object that does not emit light; When updating the albedo, roughness, metallicity, and environment map parameters of Gaussian points, the generation process propagates the photometric loss direction to the geometry buffer, thereby updating these parameters. This update process is implemented through an iterative optimization loop, which monitors the current iteration count in real time. When the number of iterations reaches the preset maximum number of iterations, the loop stops and the current optimal parameters are output.
[0044] S3. Freeze the geometric and material information obtained in step S2, and optimize the 3D Gaussian model using the luminescent and non-luminescent image sets to obtain a Gaussian point set with self-illuminating properties. Specifically, in step S3, freeze the albedo, roughness, metallicity, and environment mapping parameters optimized in step S2, and optimize the self-illuminating properties of the 3D Gaussian model using the luminescent image set. This includes the following steps: S301. Calculate the brightness difference result of the self-emitting body based on the luminous image set and the non-luminous image set:
[0045] In the formula, The brightness difference results generated by the self-emissive body. Image of a luminescent body emitting light. An image of a luminous object that does not emit light; This eliminates ambient background light in the brightness difference results, including only the direct and indirect light contributions caused by the luminous body.
[0046] S302. Construct a self-illumination attribute for each Gaussian point, and render the predicted self-illumination image based on the self-illumination attribute; S303. Construct a mask loss function, and train and optimize the self-illumination properties based on the predicted self-illumination image and the mask loss function; wherein the mask loss function is:
[0047] In the formula, For mask loss, The mask corresponding to the light-emitting area. To predict self-illuminating images, The brightness difference results generated by the self-emissive body. These are two-dimensional pixel coordinates, representing the location of a specific pixel in the image where the loss is to be calculated. S304. Construct a Gaussian point set of self-illumination attributes based on the optimized self-illumination attributes of each Gaussian point.
[0048] S4. A self-illuminating map is obtained by rendering Gaussian point sets based on geometric and material information and self-illumination properties, and the lighting effect of the self-illuminating map is modified according to the self-illumination properties. In this embodiment, to address the illumination effect of the luminescent body on the surrounding environment when rendering the self-illuminating map, a screen-space global illumination (SSGI) module is introduced into the deferred rendering pipeline. The screen-space global illumination module is located in the lighting calculation stage of the deferred rendering pipeline. It calculates the Monte Carlo integration of each shading pixel on the screen in screen space, samples multiple luminescent Gaussian points, and calculates the radiative contribution of each luminescent Gaussian point to the shading pixel to obtain the indirect lighting intensity. The indirect lighting intensity is then superimposed with the direct lighting intensity, and the final image is output after tone mapping. Specifically, the following steps are included: S401. Calculate the step size for each shading pixel to move towards the emitting Gaussian point. To balance computational efficiency and near-field detail, the step size is dynamically adjusted as the light travels. Specifically, a smaller step size is used when approaching the shading pixel surface (starting point) to capture contact shadows, and the step size increases with distance. The formula for calculating the step size is:
[0049] In the formula, For the first The step size of the next step. This is the sampling point index during the ray's movement; it represents the current sampling point on the path from the shaded pixel to the emitting Gaussian point. Each sampling stage, For the minimum step size, For the maximum step size, In order to be with the first The normalized function related to the step distance; S402. Calculate the ray visibility of each shading pixel during the stepping process towards the emitting Gaussian point; that is, calculate the occlusion relationship by combining the cumulative opacity of the Gaussian field and the depth buffer during the stepping process to determine whether the light is blocked; specifically, step sampling is performed along the ray direction during the stepping process of each shading pixel towards each emitting Gaussian point to obtain multiple sampling points, and the local occlusion value of each sampling point is calculated. The castability of all sampling points is accumulated to obtain the ray visibility of each shading pixel during the stepping process towards the emitting Gaussian point. The calculation formula is as follows:
[0050] In the formula, For the visibility of light during the stepping process of a colored pixel towards the emitting Gaussian point. This represents the overall occlusion value of the sampling points. It is the sampling point number along the light stepping path. It represents each discrete spatial position traversed when performing linear stepping sampling from the colored pixel to the target emitting Gaussian point. In step S402, the depth buffer value and Gaussian field cumulative opacity are obtained by rendering the trained 3D Gaussian model. Specifically, the depth buffer value of the visible surface of the scene can be obtained by rasterizing the trained 3D Gaussian model at the current viewpoint, which stores the depth value of the visible surface. The Gaussian field cumulative opacity can be obtained by rasterizing the trained 3D Gaussian model, or by querying the Gaussian density field at the projection position during the light stepping process. The Gaussian field cumulative opacity represents the degree to which the spatial location is not only occluded by the visible surface but also by a semi-transparent Gaussian cloud. The specific acquisition process of the depth buffer value and Gaussian field cumulative opacity is existing technology and will not be elaborated here.
[0051] The overall occlusion value of the sampling point is calculated based on the depth buffer value and the cumulative opacity of the Gaussian field:
[0052] In the formula, This represents the overall occlusion value of the sampling points. Accumulate opacity for the Gaussian field. These are the projected coordinates of the sampling point on the screen. The Gaussian opacity at this location is used to handle semi-transparent occlusions. For indicator functions, The depth of the sampling point. This is the depth buffer value for the visible surface of the scene. This is a small offset used to prevent self-occlusion, depending on the depth of the sampling point. Greater than the depth buffer value When the indicator function is 1, it means that the light ray has passed through the interior of the geometric surface and occlusion has occurred; S403. Construct a rendering equation based on ray visibility, and calculate the distance attenuation and bidirectional reflection distribution of each visible ray point to a pixel based on the rendering equation; where the rendering equation is:
[0053]
[0054] In the formula, For the first The contribution brightness of each luminous Gaussian point to the current pixel. From the current colored pixel to the 1st pixel The ray visibility between several luminous Gaussian points is used to characterize spatial occlusion. For the j-th luminous Gaussian point The intensity of self-luminescence, For distance attenuation, For the current colored pixel and the first The physical distance between the luminous Gaussian points The constant coefficient for distance, It is a constant. This refers to the bidirectional reflectance distribution function term in physically based rendering (PBR), which includes diffuse and specular reflections. To point from the colored pixel to the first The direction vector of the unit incident light at each luminous Gaussian point This is the unit view direction vector pointing from the colored pixel to the camera. Let be the unit surface normal vector at the shaded pixel. It is a function with maximum value. The cosine term of the incident light and the surface normal; The formula for calculating indirect lighting is:
[0055] In the formula, Color pixels Indirect light component at the location, For pixels The total number of luminescent Gaussian points that contribute to effective illumination. For the first A pair of luminous Gaussian points for each pixel The resulting contribution brightness, The coordinates of the pixel to be colored; S404. The self-illumination effect is calculated based on the Gaussian point set of self-illumination properties and indirect lighting. The specific formula for calculating the self-illumination effect is as follows:
[0056] In the formula, Color pixels The total self-illumination effect value at the location, For direct self-illumination, it means that given the self-illumination property parameters... In this case, the corresponding 3D Gaussian points are projected onto the pixels using microspatial sputtering technology. The initial radiance value generated at that location, It has a self-illuminating property. This refers to the pixel position coordinates in the image coordinate system. Specifically, during the geometry buffer rasterization stage, not only are the normals and materials (Albedo, Roughness, Metallic) rendered, but also the self-illumination properties of Gaussian points are projected and blended onto the screen pixels using Differentiable Splatting technology to obtain the direct self-illumination term. Indirect lighting; In step S4, when modifying the lighting effect of the self-illumination map based on its self-illumination properties, the self-illumination properties are modified by obtaining the user-inputted adjustment targets, such as adjusting hue, saturation, and brightness. The self-illumination map is then rendered based on the modified self-illumination properties and the screen space global illumination module, resulting in a physically consistent new lighting effect. For example, the rendered result of the self-illuminating body has hue, saturation, and brightness parameters consistent with the modification targets, and the hue, saturation, and brightness parameters of the surrounding environment's reflection of the self-illuminating body in the rendered image also match the self-illumination effect. The lighting effect of the self-illumination map is shown in the attached figure. Figure 2 As shown.
[0057] Table 1 shows a comparison of the performance metrics of this solution with existing technologies under different lighting editing modes: Table 1. Comparison of performance indicators under different lighting editing modes
[0058] As shown in the table, the method in this solution performs excellently in several core performance indicators, including: In terms of rendering accuracy and material reproduction capability, in the basic New View Composition (NV) experiment, the structural similarity index SSIM (0.9790) of this application outperformed all the comparison methods, and the peak signal-to-noise ratio PSNR (36.78) also remained at an extremely high level. This proves that by explicitly modeling geometric and material information (albedo, roughness, metallicity), this application can accurately reproduce the fine structure and intrinsic materials of the target scene, laying a high-quality geometric foundation for subsequent editing.
[0059] Among the advantages of complex relighting editing, the advantages of this application are most significant in experiments involving intensity editing (NV+I), color editing (NV+C), and intensity + color composite editing (NV+I+C): The peak signal-to-noise ratio (PSNR) is significantly superior: Under composite editing, the PSNR of this application reaches 31.93, far exceeding the 23.52 of GS-IR (3D Gaussian splash reverse rendering) and the 23.90 of the R3DG model. This indicates that by decoupling the self-illumination property and introducing physically consistent indirect lighting calculations, this scheme not only ensures accurate light source color in the relit scene but also makes the reflection effects of surrounding objects more physically realistic.
[0060] The visual perception quality is better. The LPIPS index of the learning-perceived image patch similarity in editing mode is significantly lower than other Gaussian splashing-based methods (GS-IR, R3DG, DeferredGS), which means that the rendering result is closer to the real image in human visual perception.
[0061] In terms of runtime, this scheme demonstrates strong practicality. Specifically, the training time of this scheme can be completed in only 25 minutes, which is an order of magnitude improvement compared to the 5 hours of the self-luminous source reconstruction method ESR-NeRF based on neural radiation field, and is also superior to similar methods such as R3DG.
[0062] In terms of editing timeliness, this solution takes less than 5 minutes to edit, achieving near real-time lighting adjustment feedback.
[0063] Comparison of this solution with existing technologies in modifying light source color: Figure 3 As shown in the attached figure, the rendering results of this solution effectively eliminate noise and rendering artifacts. In contrast, the traditional ESR-NeRF method based on the NeRF architecture exhibits significant image blurring, noise, and geometric artifacts when rendering new lighting scenes, such as the obvious blocky anomalies appearing on the bottom light-emitting plate in the Cube scene. Furthermore, the Deferred Shading Gaussian Splashing (DeferredGS) method produces unnatural dark spots and projection errors on the surface of the gift box in the Gift scene. In comparison, the image generated by this solution is clear and clean, without introducing any obvious rendering artifacts.
[0064] Furthermore, this solution can accurately reproduce indirect lighting and material reflection. Existing 3D Gaussian-based methods (such as GS-IR, R3DG, and DeferredGS) perform poorly in handling the interaction between light and materials, making it difficult to generate physically correct indirect lighting. For example, in a magnified view of the Cube scene, R3DG and DeferredGS failed to calculate the diffuse reflection of the red background light on the bottom of the cube, resulting in the bottom of the cube remaining dark; in a magnified view of the Book scene, R3DG failed to correctly illuminate the surrounding wooden table texture and books with red light. This solution successfully decouples lighting and materials, and can simulate the diffuse and specular reflection effects of the new light source on the surrounding environment very naturally, allowing the red light to be correctly mapped onto the bottom of the cube and the surface of the wooden table.
[0065] Highly realistic physical world: By combining global and local detail representation of various scenes, this solution renders light and shadow transitions, material textures, and color changes of illuminated surfaces that are closest to real reference images, solving the problems of light and shadow separation and physical distortion caused by traditional methods when editing light source colors.
[0066] Compared to existing technologies, this solution achieves controllable self-illumination editing. By explicitly modeling self-illumination properties, users can freely adjust the color and intensity of the light source, achieving true heavy lighting and solving the problem of traditional 3DGS "baking" light into color.
[0067] This approach achieves physically consistent indirect lighting by introducing a Screen Space Global Illumination (SSGI) module. This ensures that after editing light sources, other objects in the scene can correctly reflect the new light color and intensity, significantly enhancing realism. Furthermore, this solution inherits the efficiency of 3DGS, offering training speeds several times faster than NeRF-like methods and supporting real-time rendering. This enables efficient training and rendering while avoiding costly volumetric ray tracing calculations.
[0068] The present invention also provides a three-dimensional scene relighting and self-illumination editing device based on Gaussian splashing, including a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described three-dimensional scene relighting and self-illumination editing method based on Gaussian splashing.
[0069] The present invention also provides a computer-readable storage medium containing a computer program, wherein the computer program is stored thereon, and when the computer program is executed by one or more processors, it implements the above-described method for relighting and self-illuminating a 3D scene based on Gaussian splashing.
[0070] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the scope of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for relighting and self-illuminating 3D scenes based on Gaussian splashing, characterized in that, include: Images of non-emitting and emitting objects from multiple perspectives in the target scene are acquired separately, and sets of emitting and non-emitting images are constructed. Construct a 3D Gaussian model of the target scene, train the 3D Gaussian model based on a set of non-illuminated images, and obtain the geometric and material information of the 3D Gaussian model; The 3D Gaussian model is optimized based on the luminescent and non-luminescent image sets to obtain the Gaussian point set with self-luminescent properties. A self-illuminating map is obtained by rendering based on geometric and material information and Gaussian point sets with self-illumination properties, and the lighting effect of the self-illuminating map is modified according to the Gaussian point sets with self-illumination properties.
2. The relighting and self-illuminating editing method according to claim 1, characterized in that, The step of training a 3D Gaussian model based on a set of non-illuminated images to obtain the geometric and material information of the 3D Gaussian model includes: Calculate the weighted average depth of the visible Gaussian rays for each ray in the 3D Gaussian model based on a set of non-emitting images; The pseudo-normal map is calculated based on the weighted average depth of the visible Gaussian rays for each ray, and the normal map is obtained by supervising the rendering of the normals of each Gaussian point based on the pseudo-normal map. Albedo map, roughness map and metallicity map are generated based on 3D Gaussian model, and the geometric and material information of 3D Gaussian model is obtained by combining normal map, albedo map, roughness map and metallicity map.
3. The relighting and self-illuminating editing method according to claim 2, characterized in that, Following the steps of generating the albedo map, roughness map, and metallicity map based on the 3D Gaussian model, the method further includes: Construct a physically based rendering environment lighting model, and obtain environment maps based on the physically based rendering environment lighting model; Predictive images are generated based on normal maps, albedo maps, roughness maps, metallicity maps, and environment maps; The photometric loss is calculated by comparing the predicted image with a set of non-illuminated images, and the albedo, roughness, metallicity, and environment map parameters of each Gaussian point are updated based on the photometric loss.
4. The relighting and self-illuminating editing method according to claim 1, characterized in that, The step of optimizing the 3D Gaussian model based on the luminescent and non-luminescent image sets to obtain the self-luminescent Gaussian point set includes: The brightness difference of the self-luminous body was calculated based on the luminous image set and the non-luminous image set. A self-illumination attribute is constructed for each Gaussian point, and a predicted self-illumination image is rendered based on the self-illumination attribute. A mask loss function is constructed based on the brightness difference results of the self-illuminating body and the predicted self-illuminating image, and the self-illuminating properties are optimized based on the predicted self-illuminating image and the mask function. Construct a Gaussian point set of self-illuminating properties based on the optimized self-illuminating properties.
5. The relighting and self-illuminating editing method according to claim 1, characterized in that, The self-illuminating map obtained by rendering Gaussian point sets based on geometric and material information and self-illumination properties includes: Calculate the step size for each shading pixel to move towards the luminous Gaussian point; Calculate the ray visibility of each shading pixel as it moves toward the emitting Gaussian point; A rendering equation is constructed based on the visibility of light, and the distance attenuation and bidirectional reflection distribution of each visible light point to a pixel are calculated based on the rendering equation. The self-illumination effect is calculated based on the Gaussian point set of self-illumination properties and the rendering equation, and the self-illumination map is rendered based on the self-illumination effect.
6. The relighting and self-illuminating editing method according to claim 5, characterized in that, The calculation of the ray visibility of each tinted pixel during its step towards the emitting Gaussian point includes: Multiple sampling points are obtained by stepping along the direction of the light rays during the stepping process from the color pixel to the luminous Gaussian point; The depth buffer value and cumulative opacity of the Gaussian field for each sampling point are obtained by rendering the trained 3D Gaussian model. The overall occlusion value for each sampling point is calculated based on the depth buffer value and the cumulative opacity of the Gaussian field. The overall occlusion value of all sampling points is accumulated and calculated to obtain the light visibility of the colored pixel during the stepping process towards the emitting Gaussian point.
7. The relighting and self-illuminating editing method according to claim 5, characterized in that, The self-illumination effect calculated based on the Gaussian point set of self-illumination properties and the rendering equation includes: The self-illumination term is obtained by rendering Gaussian point sets based on geometric and material information and self-illumination properties. Indirect lighting is calculated based on the rendering equation; The self-luminescence effect is calculated based on the self-luminescence term and indirect illumination.
8. The relighting and self-illuminating editing method according to claim 1, characterized in that, When modifying the lighting effect of the self-illumination map, obtain the Gaussian point set of the self-illumination attribute to be modified as input, and render the self-illumination map based on the modified Gaussian point set of the self-illumination attribute.
9. A 3D scene relighting and self-illuminating editing device based on Gaussian splashing, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the three-dimensional scene relighting and self-illumination editing method based on Gaussian splashing as described in any one of claims 1-8.
10. A computer-readable storage medium containing a computer program, wherein the computer program is stored thereon, characterized in that, When the computer program is executed by one or more processors, it implements the three-dimensional scene relighting and self-illumination editing method based on Gaussian splashing as described in any one of claims 1-8.