Three-dimensional scene relighting system and method based on Gaussian Phong model

Abstract

The invention provides a three-dimensional scene relighting system and method based on a Gaussian Phong model, and the system comprises a Gaussian Phong rendering module which is provided with an ambient light unit for calculating the ambient light color of a Gaussian point; the diffuse reflection light unit is used for calculating a diffuse reflection color based on a Blinn-Phong reflection model; the specular reflection light unit is used for calculating a specular reflection color based on a Blinn-Phong reflection model; the fusion unit is used for adding the ambient light color, the diffuse reflection color and the specular reflection color to obtain the final color of the Gaussian point; and the output unit is used for mapping the final colors of all Gaussian points to a camera plane by utilizing a Gaussian splashing technology to obtain a rendering result under an observation visual angle. According to the method, the Blinn-Phong reflection model is introduced into a differentiable rendering framework of the 3DGS, various illumination in a scene can be naturally decomposed and decoupled, and understanding more conforming to physical prior is provided for interaction of objects and illumination in training data.

Description

Technical Field

[0001] The present invention relates to the fields of computer image processing and machine learning, and in particular to a three-dimensional scene re-illumination system and method based on a Gaussian Phong model. Background Art

[0002] Relighting technology is widely used in computer graphics, virtual reality, movie special effects, and 3D scanning. Its main task is to change the lighting conditions in the scene while keeping the geometric structure of the object unchanged. Traditional relighting methods usually rely on multi-camera and multi-light source settings, that is, using multiple synchronized cameras and light sources in different positions to capture the scene, and perform relighting calculations from multiple perspectives and lighting conditions. This means that under each lighting condition, the scene needs to be captured synchronously by multiple cameras. Although this method can provide high-quality rendering effects, it requires a large amount of hardware equipment and complex synchronization settings, which makes it costly and cumbersome to operate in practical applications. Especially under dynamic lighting conditions, the data collection process is often very cumbersome and time-consuming.

[0003] In recent years, the One Light At a Time (OLAT) task has become a research focus in the field of re-illumination. The core idea of ​​the OLAT setting is to use a single point light source at a time during training and rely on a single camera to capture images, which greatly simplifies the data acquisition process. This method collects data by moving the light source and camera, enabling efficient training at a low cost and applicable to dynamic and complex lighting change scenes. Compared with traditional multi-light source and multi-camera re-illumination methods, the OLAT setting greatly reduces hardware requirements, typically requiring only a single camera and a single point light source, which gives it significant advantages in dynamic lighting changes and cost control.

[0004] The training setting of OLAT is usually to collect data by gradually changing the position of the point light source and the camera's viewing angle to build an image dataset containing the light source position and camera configuration. Each frame of the image is obtained by the triplet O t ,V t ,P t Indicates that O t is the observed image, V t For camera configuration, P t is the light source position. This approach makes data collection more flexible and can handle more complex lighting variations, while avoiding the challenges of synchronizing multiple light sources and cameras in traditional methods. In this setup, the model no longer relies on a traditional multi-light environment. Instead, it learns the scene's geometry and material information by gradually varying the light source position, achieving high-quality relighting effects.

[0005] Overall, the OLAT method has become a key research direction in the field of re-illumination due to its simplified data acquisition process, low hardware requirements, and efficient training model. It has not only attracted widespread attention in academia but also provided a more feasible solution for practical applications, showing great potential in scenarios such as dynamic lighting and real-time rendering. However, current OLAT methods have the following two main limitations in terms of generalization:

[0006] First, limited training data makes geometric learning difficult.

[0007] Due to the limited training data available, coupled with the movement of both the light source and the camera, the illumination and perspective constantly change over time, and the entanglement of geometric and illumination information increases the difficulty of learning the true geometry of objects. With such a limited training corpus, the model often struggles to accurately capture complex geometric details, and thus cannot maintain good reconstruction and rendering results in new environments or perspectives.

[0008] Second, it has poor adaptability to changes in the distribution of test light sources.

[0009] Existing methods rely heavily on specific lighting distributions. When encountering a completely different light distribution during testing than during training (i.e., an out-of-distribution lighting setup), the model is prone to overfitting, making it difficult to produce realistic relighting renderings. This often manifests as blurring of highlights and shadows under the new lighting conditions, resulting in reflections and shadows that are inconsistent with real-world scenes. This indicates that the model has failed to truly learn universal reflection features, but instead has primarily memorized the lighting patterns in the training set. Summary of the Invention

[0010] In view of the defects in the prior art, the present invention aims to provide a three-dimensional scene re-illumination system based on the Gaussian Phong model.

[0011] According to one aspect of the present invention, a three-dimensional scene relighting system based on a Gaussian Phong model is provided, comprising a Gaussian Phong rendering module, the module comprising:

[0012] An input unit is configured to obtain Gaussian point cloud parameters, an observation angle, and a light source position. The Gaussian point cloud parameters include a zero-order spherical harmonic coefficient, a diffuse reflection coefficient, a normal vector direction, and a specular reflection coefficient.

[0013] An ambient light unit calculates the ambient light color of a Gaussian point using the zero-order spherical harmonic coefficients;

[0014] A diffuse reflection light unit, based on a Blinn-Phong reflection model, uses the diffuse reflection coefficient, the normal vector direction, and the light source position to calculate the diffuse reflection color of a Gaussian point;

[0015] A specular reflection light unit, based on a Blinn-Phong reflection model, uses the normal vector direction, the light source position, the specular reflection coefficient, and the observation angle to calculate the specular reflection color;

[0016] A fusion unit, which adds the ambient light color, diffuse reflection color, and specular reflection color to obtain a final color of the Gaussian point;

[0017] The output unit maps the final colors of all the Gaussian points to the camera plane using the Gaussian splashing technology to obtain a rendering result under the observation angle.

[0018] Preferably, the fusion process of the fusion unit is:

[0019]

[0020] Among them, L a The ambient light color is directly calculated from the zero-order spherical harmonic coefficients. is the light intensity received by the Gaussian point, L d Represents the final color of diffuse light, which is composed of the diffuse RGB color k d and diffuse intensity I d Multiply to get; L s Represents the final color of the specular reflected light, which is the specular reflection RGB color k s and diffuse intensity I s Multiply them together to get .

[0021] Preferably, a meta-learning training module is also included, which implements the loss function calculation and gradient update of the Gaussian Phong rendering module through a meta-learning method; the module takes the Gaussian point cloud parameter set as input and outputs a trained Gaussian Phong rendering module.

[0022] Preferably, the meta-learning training module regards re-illumination as a multi-task learning problem, and rendering under different lighting positions as independent tasks, and adopts a two-layer meta-learning optimization method to achieve consistent learning of geometric and color information.

[0023] Preferably, the meta-learning training module includes:

[0024] Meta-learning inner loop unit, using the support set to train the Gaussian Phong rendering module: starting from the same parameters θ, it is trained on m support samples in sequence to independently generate m sub-models θ' i , i=1…m, perform rendering tasks under different lighting positions;

[0025] Meta-learning outer loop unit, using the query set to calculate the loss function and formally update the model gradient for the Gaussian Phong rendering module: all sub-models θ' trained in the inner loop i Test on m query samples in turn, add the calculated loss functions of the m query samples to get the overall loss function value of the Gaussian Phong rendering module, and use it to update the initial θ.

[0026] Preferably, the overall loss function value of the Gaussian Phong rendering module is:

[0027]

[0028] in, represents the mean square error between the predicted rendered image and the real image; represents the sparse loss, It is the loss related to the normal vector; is the Gaussian flat loss is the normal vector smoothing loss;

[0029] α in the middle is the opacity of the Gaussian point;

[0030] Include normal vector loss and normal vector residual loss λ n ,λ res Represents the corresponding weight; n is the normal vector direction, It is the normal vector estimated from the depth map output by the Gaussian point cloud;

[0031] Where s1, s2, and s3 are the scaling coefficients in the three directions of the Gaussian point;

[0032] Calculated by rendering the two-dimensional normal map, sg(.) means no gradient is passed, and g represents a Gaussian kernel of size k.

[0033] According to a second aspect of the present invention, a three-dimensional scene relighting method based on a Gaussian Phong model is provided, comprising:

[0034] Obtaining Gaussian point cloud parameters, observation angle, and light source position, wherein the Gaussian point cloud parameters include zero-order spherical harmonic coefficients, diffuse reflection coefficients, normal vector directions, and specular reflection coefficients;

[0035] Calculate the ambient light color of the Gaussian point using the zero-order spherical harmonic coefficients;

[0036] Based on the Blinn-Phong reflection model, the diffuse reflection color of the Gaussian point is calculated using the diffuse reflection coefficient, the normal vector direction, and the light source position;

[0037] Based on the Blinn-Phong reflection model, the specular reflection color is calculated using the normal vector direction, the light source position, the specular reflection coefficient and the observation angle;

[0038] Add the ambient light color, diffuse reflection color and specular reflection color to obtain the final color of the Gaussian point;

[0039] The final colors of all the Gaussian points are mapped to the camera plane using the Gaussian splashing technique to obtain a rendering result under the observation angle.

[0040] Preferably, the process of obtaining the rendering result is also included, including:

[0041] The loss function calculation and gradient update of Gaussian rendering are realized by a meta-learning method. The meta-learning algorithm is based on the Gaussian point cloud parameter set {θ k} as input and output the trained Gaussian point cloud parameter set.

[0042] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the system or the method when executing the program.

[0043] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to run the system or perform the method.

[0044] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0045] In an embodiment of the present invention, the Blinn-Phong reflection model is applied to a differentiable rendering framework of three-dimensional Gaussian sputtering (3DGS). The various lighting components in the scene can be effectively decoupled through the ambient light unit, diffuse reflection unit, and specular reflection unit. In addition, it also promotes the decoupling of geometry and material by integrating the fusion unit to fuse the colors of ambient light, diffuse reflection, and specular reflection, thereby providing a more physically priori understanding of the interaction between objects and lighting in the training data. Therefore, it can solve the geometric learning problem caused by limited training data in the prior art.

[0046] The Gaussian Phong model-based three-dimensional scene relighting system in the embodiment of the present invention explores the use of meta-learning methods to train the relighting model, regards OLAT relighting as a multi-task learning problem, and regards rendering under different lighting positions as independent tasks. A two-layer meta-learning optimization method is used to achieve consistent learning of geometric and color information.

[0047] The Gaussian Phong model-based 3D scene relighting system in this embodiment combines a rendering framework based on the Blinn-Phong reflectance model with a meta-learning training method to achieve efficient generalization under distributed external light sources. This system effectively addresses the limitations of existing technologies in adapting to changes in test light source distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0049] Figure 1 4 is a structural diagram of a three-dimensional scene re-illumination system based on a Gaussian Phong model in one embodiment of the present invention;

[0050] Figure 2 This is an overall flow chart of the Gaussian Phong rendering module in one embodiment of the present invention;

[0051] Figure 3 This is an algorithm flow chart of the meta-learning training module in a preferred embodiment of the present invention;

[0052] Figure 4 4 is a flowchart of a three-dimensional scene re-illumination method based on a Gaussian Phong model in one embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0054] like Figure 1 and Figure 2 As shown, in one embodiment of the present invention, a three-dimensional scene re-illumination system based on the Gaussian Phong model includes a Gaussian Phong rendering module, which inputs: Gaussian point cloud parameters θ k =(x,R,S,α,f0,n,k d ,k s ) k, observation angle v, light source position l, where Gaussian point position x, rotation matrix R, scaling matrix S, opacity α, zero-order spherical harmonic coefficient f0, diffuse reflection coefficient k d and the specular reflection coefficient k s , n normal vector direction; output the rendered image obtained by the Gaussian splash method.

[0055] The Gaussian Phong rendering module includes:

[0056] Input unit, obtains Gaussian point cloud parameters, observation angle and light source position, Gaussian point cloud parameters zero-order spherical harmonic coefficient, diffuse reflection coefficient, normal vector direction and specular reflection coefficient;

[0057] The ambient light unit uses the zero-order spherical harmonic coefficients to calculate the ambient light color of the Gaussian point;

[0058] The diffuse reflection unit is based on the Blinn-Phong reflection model and uses the diffuse reflection coefficient, normal vector direction, and light source position to calculate the diffuse reflection color of the Gaussian point;

[0059] The specular reflection light unit is based on the Blinn-Phong reflection model and uses the normal vector direction, light source position, specular reflection coefficient and observation angle to calculate the specular reflection color;

[0060] The fusion unit adds the ambient light color, diffuse reflection color and specular reflection color to obtain the final color of the Gaussian point;

[0061] The output unit uses the Gaussian splashing technology to map the final colors of all Gaussian points to the camera plane to obtain the rendering results under the observation perspective.

[0062] The Gaussian Phong model is an illumination model that combines the Gaussian distribution and the Phong reflection model. It is mainly used to simulate the lighting effects of objects in computer graphics. This model not only takes into account the diffuse reflection caused by the rough surface, but also takes into account the effects of highlights and ambient light caused by the mirror reflection of the surface. The basic parameters of the differentiable rendering framework of the Gaussian point cloud 3DGS include the parameters x, R, S, α, f0 (learnable parameters) and the observation angle v (the input of the data set) of each Gaussian point. The Blinn-Phong reflection model is constructed on the 3DGS framework. In addition to the basic parameters contained in the 3DGS framework, the following parameters are added: the parameter diffuse reflection color k of each Gaussian point d , specular color k s, the normal vector direction n (a learnable parameter), and the light source position l. Thus, the above embodiment introduces the Blinn-Phong reflection model into the differentiable rendering framework of Gaussian point cloud 3DGS. This fusion method can naturally decouple the various lighting components in the scene (such as diffuse reflection color and specular reflection color), thereby providing a more physically consistent understanding of the interaction between objects and lighting in the training data.

[0063] In a preferred embodiment, the ambient light unit calculates the ambient light color of the Gaussian point using zero-order spherical harmonic coefficients;

[0064] I a =C0·f0

[0065] Where C0 is the normalized coefficient of the zero-order spherical harmonic basis function, which is a constant, and f0 is the zero-order spherical harmonic coefficient of the Gaussian point.

[0066] Ambient light units simulate constant ambient lighting. They represent the uniform brightness of light after multiple scatterings, ensuring that even shadows are never completely dark. Ambient light units provide a baseline brightness, ensuring that there are no completely black areas in the scene, thereby enhancing visual realism. They have a low overall computational cost and effectively avoid completely dark areas, making the scene appear more natural.

[0067] In a preferred embodiment, the diffuse reflection light unit calculates the diffuse reflection color of the Gaussian point using the diffuse reflection coefficient, the normal vector direction, and the light source position based on the algorithm of the Blinn-Phong reflection model;

[0068]

[0069] Among them, n is the direction of the normal vector, l is the position of the light source, specifically the incident direction of the light source, I is the intensity of the light source, and r is the distance between the Gaussian point and the point light source.

[0070] The diffuse reflection unit follows Lambert's law, simulating the multi-directional scattering of light upon striking a rough surface. Its function is to calculate brightness based on the angle between the light and the surface normal, brightening areas facing the light source and darkening areas away from it, resulting in a soft, non-glossy effect. The diffuse reflection unit realistically simulates the lighting characteristics of most natural surfaces and is suitable for most materials, such as walls and fabrics.

[0071] In a preferred embodiment, the specular reflection light unit calculates the specular reflection color based on an algorithm of the Blinn-Phong reflection model using the normal vector direction, light source position, specular reflection coefficient, and observation angle;

[0072]

[0073] Where h is the average value (half-range vector) of the viewing angle direction v and the incident direction of the light source l, p is an index representing the surface glossiness, and max represents the maximum value;

[0074] Specular lighting simulates specular reflections on smooth surfaces, calculating the intensity of specular reflections based on the angle between the light source and the viewing direction. Specular lighting creates a glossy, shiny surface, enhancing the smoothness and metallic feel of materials. This enhances the three-dimensionality and realism of objects and is particularly useful on glass, metal, and polished surfaces.

[0075] In a preferred embodiment, the fusion unit adds the ambient light color, the diffuse reflection color, and the specular reflection color to obtain the final color of the Gaussian point;

[0076]

[0077] Among them, L a The ambient light color is directly calculated from the zero-order spherical harmonic coefficients. is the light intensity received by the Gaussian point, L d Represents the final color of diffuse light, which is composed of the diffuse RGB color k d and diffuse intensity I d Multiply to get; L s Represents the final color of the specular reflected light, which is the specular reflection RGB color k s and diffuse intensity I s Multiply them together to get .

[0078] The fusion unit performs a weighted combination of the colors of ambient light, diffuse light, and specular light to calculate the final color of the Gaussian point. This is then used in Gaussian splatting to produce the final rendered image. Through this process, the fusion unit integrates the influence of different lighting components, ensuring that the rendered result not only maintains basic brightness but also captures object details, material properties, and lighting characteristics, resulting in a more realistic and layered image. The advantages of the fusion unit lie in its physical consistency, computational efficiency, and improved visual quality. First, it uses the Blinn-Phong reflectance model for lighting calculations, ensuring that the rendered result conforms to physical lighting laws, thereby enhancing image realism. Second, due to the simplicity of the Blinn-Phong reflectance model, the fusion unit has low computational overhead, making it suitable for real-time rendering and efficient computation. Finally, fusing different lighting components enhances the appearance of light and shadow on objects, enriching surface details while avoiding the unnatural effects associated with a single lighting model, thereby improving overall visual quality.

[0079] Although the OLAT setting simplifies data collection, actual training is challenging. When the illumination and perspective change simultaneously, the geometry and color information of the object are prone to change significantly, which increases the difficulty of model learning. Because the same position presents different colors at different perspectives, the model often overfits to a single perspective and learns incorrect geometry and color information. For example, it learns overly complex scene geometry to adapt to different perspective illumination phenomena, which ultimately makes the model lack consistency. In a preferred embodiment of the present invention, another meta-learning training module is designed. This module implements the model loss function calculation and gradient update through a meta-learning algorithm. This module uses the Gaussian point cloud parameter set {θ k} as input, and output is a set of trained Gaussian point cloud parameters. This training module primarily treats relighting as a multi-task learning problem, treating rendering under different lighting positions as independent tasks, and employing a two-layer meta-learning optimization method to achieve consistent learning of geometric and color information. Specifically, let's first introduce some concepts:

[0080] In meta-learning, the master model is the global model for meta-learning, storing common knowledge across tasks. In this application, the master model refers to the Gaussian Phong rendering module. Sub-models are task-specific instances derived from the master model, fine-tuned for a single task. The support set is the training dataset for the sub-model, used for rapid within-task adaptation. Each support example includes light source position, camera parameters, RGB images, and labels. The query set is the test dataset for the sub-model, used to evaluate task adaptation.

[0081] A single main model update will use m tasks, each task contains a support sample and a query sample, that is, a single main model update will use m support samples and m query samples.

[0082] In meta-learning, for each new task, we start with the initialization parameters of the main model, generate a sub-model, fine-tune it using the support set of the task, and evaluate the fine-tuned sub-model using the query set. Then, we aggregate all the query losses and update the main model parameters.

[0083] Specifically, the training process of this embodiment is divided into two parts: inner loop and outer loop. In the inner loop, the training process of the sub-model only includes a single step or a few steps of gradient optimization to achieve the goal of rapid adaptation to the task. After training, the sub-model will be evaluated on the query set, and its loss is used to guide the outer loop to update the initial parameters of the main model. Specifically, the meta-learning training module, such as Figure 3 As shown, including:

[0084] Meta-learning inner loop unit: In the inner loop, the model is trained using the support set. All sub-models start with the same parameters θ and are trained on m support samples in turn to calculate the loss function D supRepresents the support set of the current iteration, Represents a support set sample. To independently generate m sub-models θ′ i , i=1…m;

[0085] Meta-learning outer loop module: In the outer loop, for each query sample, the corresponding inner loop is used to optimize the model θ' i Calculating the loss function D query Represents the query set of the current iteration, Represents a query set sample. Then, aggregate m loss values And update the initial θ. This approach enables the model to learn representations with strong generalization capabilities under different lighting conditions through task-specific adaptation followed by global updates.

[0086] The loss function value of the aggregated main model is:

[0087] The overall loss function value of the Gaussian Phong rendering module is:

[0088]

[0089] in, represents the mean square error between the predicted rendered image and the real image; represents the sparse loss, It is the loss related to the normal vector; is the Gaussian flat loss is the normal vector smoothing loss;

[0090] α in the middle is the opacity of the Gaussian point;

[0091] Include normal vector loss and normal vector residual loss λ n ,λ res Represents the corresponding weight; n is the normal vector direction, It is the normal vector estimated from the depth map output by the Gaussian point cloud;

[0092] Where s1, s2, and s3 are the scaling coefficients in the three directions of the Gaussian point;

[0093] Calculated by rendering the two-dimensional normal map, sg(.) means no gradient is passed, and g represents a Gaussian kernel of size k.

[0094] The above embodiment carried out an early exploration of using meta-learning methods (Meta-Learning, also known as "learning to learn", whose core goal is to enable the model to have the ability to summarize general learning strategies from existing learning experience, so that it can quickly adapt and efficiently learn when faced with new tasks) to train re-lighting models. OLAT re-lighting is treated as a multi-task learning problem, and rendering under different lighting positions is regarded as an independent task. A two-layer meta-learning optimization method is used to ensure that the geometry and color information learned by the model are consistent, compatible and non-conflicting under different lighting conditions. Specifically, the learned geometry, as an inherent attribute of the object, does not depend on lighting changes; the learned color information can adapt to lighting changes and maintain reasonable performance under different lighting conditions, thereby achieving consistent learning of geometry and color information (consistency learning refers to: appearance consistency (no strange color jumps or unreasonable reflections)).

[0095] In addition, the Gaussian Phong rendering module in the embodiment of the present invention inherits the physical prior knowledge of the Blinn-Phong reflection model, ensuring that it can still generate rendering effects that conform to physical laws and have universal applicability under new lighting conditions, and the combination with the meta-learning training module further enhances the generalization ability of the model under atypical light source conditions. Specifically, the Gaussian Phong rendering module represents the three-dimensional scene with a Gaussian point cloud, and models the diffuse reflection and specular reflection characteristics of each Gaussian point through the improved Blinn-Phong model, introducing physical priors such as normal direction and lighting dependency to ensure the physical consistency of the rendering results under different lighting. This design provides clear physical constraints for meta-learning training, making the model optimization process more interpretable and convergent. At the same time, by constraining the optimization space, it ensures that the geometric and material information shared between different tasks has higher consistency, avoiding geometric drift or material distortion problems in cross-lighting tasks. On the other hand, the meta-learning training module models the heavy lighting problem as a multi-task learning scenario through a two-layer optimization strategy, with different lighting positions corresponding to different tasks. The Gaussian point cloud parameters are continuously optimized during training, enabling the rendering module to maintain physically reasonable rendering performance even when facing unseen lighting conditions, further improving the model's generalization ability for atypical light sources.

[0096] Based on the same inventive concept, other embodiments of the present invention further provide a three-dimensional scene re-illumination method based on the Gaussian Phong model, such as Figure 4 As shown, the following steps are included:

[0097] Step 1: Obtain Gaussian point cloud parameters, observation angle, and light source position, Gaussian point cloud parameters zero-order spherical harmonic coefficient, diffuse reflection coefficient, normal vector direction, and specular reflection coefficient;

[0098] Step 2, using the zero-order spherical harmonic coefficients, calculate the ambient light color of the Gaussian point;

[0099] Step 3: Based on the Blinn-Phong reflection model, the diffuse reflection color of the Gaussian point is calculated using the diffuse reflection coefficient, normal vector direction, and light source position.

[0100] Step 4: Based on the Blinn-Phong reflection model, the specular reflection color is calculated using the normal vector direction, light source position, specular reflection coefficient, and observation angle.

[0101] Step 5, add the ambient light color, diffuse reflection color and specular reflection color to get the final color of the Gaussian point;

[0102] Step 6: Use Gaussian splashing technology to map the final colors of all Gaussian points to the camera plane to obtain the rendering result under the observation perspective.

[0103] Each step in the above example of the present invention may refer to the implementation technology of each module / unit corresponding to the three-dimensional scene re-illumination system based on the Gaussian Phong model in the above embodiment, and will not be repeated here.

[0104] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the terminal can be used to execute the above-mentioned method, or to run the above-mentioned system.

[0105] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDRSDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories.

[0106] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0107] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0108] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the above method or run the above system.

[0109] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one location to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. Alternatively, the ASIC can be located in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0110] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0111] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0114] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A three-dimensional scene re-illumination system based on the Gaussian Phong model, characterized in that: Includes the Gaussian Phong rendering module, which includes: An input unit is configured to obtain Gaussian point cloud parameters, an observation angle, and a light source position. The Gaussian point cloud parameters include a zero-order spherical harmonic coefficient, a diffuse reflection coefficient, a normal vector direction, and a specular reflection coefficient. An ambient light unit calculates the ambient light color of a Gaussian point using the zero-order spherical harmonic coefficients; The diffuse reflection light unit calculates the diffuse reflection color of the Gaussian point based on the Blinn-Phong reflection model using the diffuse reflection coefficient, normal vector direction, and light source position; The specular reflection light unit calculates the specular reflection color based on the Blinn-Phong reflection model using the normal vector direction, light source position, specular reflection coefficient and observation angle; A fusion unit, which adds the ambient light color, diffuse reflection color, and specular reflection color to obtain a final color of the Gaussian point; The output unit maps the final colors of all the Gaussian points to the camera plane using the Gaussian splashing technology to obtain a rendering result under the observation angle.

2. A three-dimensional scene re-illumination system based on the Gaussian Phong model according to claim 1, characterized in that: The fusion process of the fusion unit is as follows: Among them, L a The ambient light color is directly calculated from the zero-order spherical harmonic coefficients. is the light intensity received by the Gaussian point, L d Represents the final color of diffuse light, which is composed of the diffuse RGB color k d and diffuse intensity I d Multiply to get; L s Represents the final color of the specular reflected light, which is the specular reflection RGB color k s and diffuse intensity I s Multiply them together and |lights| represents the number of light sources.

3. The three-dimensional scene re-illumination system based on the Gaussian Phong model according to claim 1, characterized in that: It also includes a meta-learning training module, which implements the loss function calculation and gradient update of the Gaussian Phong rendering module through a meta-learning method; the module takes a Gaussian point cloud parameter set as input and outputs a trained Gaussian Phong rendering module.

4. A three-dimensional scene re-illumination system based on the Gaussian Phong model according to claim 3, characterized in that: The meta-learning training module treats re-illumination as a multi-task learning problem and rendering under different lighting positions as independent tasks, and adopts a two-layer meta-learning optimization method to achieve consistent learning of geometric and color information.

5. The three-dimensional scene re-illumination system based on the Gaussian Phong model according to claim 4, characterized in that: The meta-learning training module includes: Meta-learning inner loop unit, using the support set to train the Gaussian Phong rendering module: starting from the same parameters θ, it is trained on m support samples in sequence to independently generate m sub-models θ' i , i=1…m, perform rendering tasks under different lighting positions; Meta-learning outer loop unit, using the query set to calculate the loss function and formally update the model gradient for the Gaussian Phong rendering module: all sub-models θ' trained in the inner loop i Test on m query samples in turn, add the calculated loss functions of the m query samples to get the overall loss function value of the Gaussian Phong rendering module, and use it to update the initial θ.

6. The three-dimensional scene re-illumination system based on the Gaussian Phong model according to claim 5, characterized in that: The overall loss function value of the Gaussian Phong rendering module is: in, represents the mean square error between the predicted rendered image and the real image; represents the sparse loss, It is the loss related to the normal vector; is the Gaussian flat loss is the normal vector smoothing loss; α in the middle is the opacity of the Gaussian point; Include normal vector loss and normal vector residual loss λ n ,λ res Represents the corresponding weight; n is the normal vector direction, It is the normal vector estimated from the depth map output by the Gaussian point cloud; Where s1, s2, and s3 are the scaling coefficients in the three directions of the Gaussian point; Calculated by rendering the two-dimensional normal map, sg(.) means no gradient is passed, and g represents a Gaussian kernel of size k.

7. A three-dimensional scene relighting method based on the Gaussian Phong model, characterized in that: include: Obtaining Gaussian point cloud parameters, observation angle, and light source position, wherein the Gaussian point cloud parameters include zero-order spherical harmonic coefficients, diffuse reflection coefficients, normal vector directions, and specular reflection coefficients; Calculate the ambient light color of the Gaussian point using the zero-order spherical harmonic coefficients; Based on the Blinn-Phong reflection model, the diffuse reflection color of the Gaussian point is calculated using the diffuse reflection coefficient, the normal vector direction, and the light source position; Based on the Blinn-Phong reflection model, the specular reflection color is calculated using the normal vector direction, the light source position, the specular reflection coefficient and the observation angle; Add the ambient light color, diffuse reflection color and specular reflection color to obtain the final color of the Gaussian point; The final colors of all the Gaussian points are mapped to the camera plane using the Gaussian splashing technique to obtain a rendering result under the observation angle.

8. The three-dimensional scene relighting method based on the Gaussian Phong model according to claim 7, characterized in that: It also includes training on the process of obtaining rendering results, including: The loss function calculation and gradient update of Gaussian rendering are realized by a meta-learning method. The meta-learning algorithm is based on the Gaussian point cloud parameter set {θ k } as input and output the trained Gaussian point cloud parameter set.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When executing the program, the processor can be used to run the system according to any one of claims 1 to 6, or to execute the method according to any one of claims 7 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to run the system according to any one of claims 1 to 6, or to execute the method according to any one of claims 7 to 8.

Images