A continuous indoor lighting modeling and estimation method based on three-dimensional gaussian sputtering

Abstract

The present application relates to the technical field of augmented reality, and particularly relates to a continuous indoor illumination modeling and estimation method based on three-dimensional Gaussian sputtering, which comprises collecting an indoor single frame image, generating an indoor illumination map by using the single frame image; extracting a depth map by using the indoor illumination map, inputting the depth map and the indoor illumination map into a three-dimensional mixed Gaussian illumination parameter estimation network, and outputting an illumination parameter estimation value of position, shape and radiance. The present application solves the problems of existing methods, such as serious discretization of illumination modeling mode, insufficient spatial consistency of illumination modeling result, and difficulty in accurately modeling occluded light and near-field light.

Description

Technical Field

[0001] This invention relates to the field of augmented reality technology, and in particular to a method for modeling and estimating continuous indoor lighting based on three-dimensional Gaussian sputtering. Background Technology

[0002] Indoor lighting modeling and estimation is one of the key foundational technologies for achieving virtual-real fusion, realistic rendering, and immersive interactive experiences. By accurately modeling and estimating the lighting in real indoor scenes, virtual objects can be provided with lighting conditions consistent with the real environment, thereby significantly improving visual realism.

[0003] Existing methods for indoor illumination estimation mainly include marker-based methods, physics-based optimization methods, and deep learning-based methods. With the development of deep learning technology, existing methods can predict indoor illumination information from single images or videos. However, existing technologies still generally have the following shortcomings: 1. The lighting modeling method suffers from severe discretization: Most methods use two-dimensional environment maps or regular three-dimensional voxel meshes to model indoor lighting, which is essentially a discrete representation and makes it difficult to accurately depict the continuous changes in indoor lighting in three-dimensional space.

[0004] 2. Insufficient spatial consistency in lighting modeling results: Existing methods typically estimate illumination independently at different spatial locations, lacking a unified continuous illumination model, which leads to inconsistent illumination results at different locations in the same scene. 3. Difficulty in accurately modeling occlusion and near-field lighting: It is difficult to accurately describe light source occlusion, window incident light, and complex near-field lighting structures based on low-dimensional lighting models or regular voxels. Summary of the Invention

[0005] To address the shortcomings of existing methods, this invention solves the problems of severe discretization in lighting modeling, insufficient spatial consistency of lighting modeling results, and difficulty in accurately modeling occlusion lighting and near-field lighting.

[0006] The technical solution adopted in this invention is: a method for continuous indoor illumination modeling and estimation based on three-dimensional Gaussian sputtering, comprising the following steps: Step 1: Acquire single-frame images of the indoor environment and generate an indoor illumination map using the single-frame images; Step 2: Extract depth map using indoor lighting map, input depth map and indoor lighting map into 3D Gaussian mixture lighting parameter estimation network, output lighting parameter estimates of position, shape and radiance; As a preferred embodiment of the present invention, the three-dimensional mixed Gaussian illumination parameter estimation network includes: Using encoder From indoor lighting diagram and depth map Extracting depth features from; Three Gaussian parameter decoders were used respectively. For each pixel predict The group includes Gaussian parameters such as position, shape, and radiance. Characterization of indoor lighting output ;in, It is the three-dimensional Gaussian opacity; and These are the mean and covariance matrices, respectively. It depends on the direction of light incidence. The radiance function.

[0007] As a preferred embodiment of the present invention, the mean The formula is:

[0008] in, It is the camera calibration matrix; In pixels; For depth; For displacement; ; For depth map The depth value at that location; This is for depth offset.

[0009] In a preferred embodiment of the present invention, the radiance function includes: Gaussian radiance of the light source, Gaussian radiance of the window, and other radiances.

[0010] In a preferred embodiment of the present invention, the formula for the Gaussian radiance of the light source is:

[0011]

[0012] in, It refers to the direction of the opening; Control the size of the opening; It refers to the brightness of the light source; It is a spherical harmonic function.

[0013] As a preferred embodiment of the present invention, the formula for the Gaussian radiance of a window is:

[0014] in, ; It is the direction and The cosine of the angle formed; It is the directional diffusion width; It is the brightness of the sun; It is the visibility term of the sun.

[0015] As a preferred embodiment of the present invention, a clipping function is used. radiance Compress to a low dynamic range.

[0016] In a preferred embodiment of the present invention, the formula for the clipping function is:

[0017] in, It's a hyperparameter.

[0018] In a preferred embodiment of the present invention, indoor illumination is characterized by the combination of an opacity function and an radiance function of the indoor illumination radiation field.

[0019] As a preferred embodiment of the present invention, a continuous indoor illumination modeling and estimation system based on three-dimensional Gaussian sputtering includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the continuous indoor illumination modeling and estimation method based on three-dimensional Gaussian sputtering.

[0020] The beneficial effects of this invention are: 1. This invention designs a continuous indoor illumination representation model based on three-dimensional Gaussian mixtures; and on this basis, proposes a single-frame indoor illumination estimation method, including generative indoor illumination map reconstruction and three-dimensional Gaussian mixture illumination parameter estimation; 2. The light source category of this invention can accurately describe the bright area near the light source and its spatial attenuation process, thereby significantly improving the accuracy and realism of indoor lighting modeling and estimation; 3. This invention designs window-type lighting to correlate radiance with the direction of solar incidence and independently models the high-intensity directional radiation components entering the room through the window. This allows for a more accurate characterization of the directional lighting effect formed by natural light entering through the window, while effectively reducing the impact of bright areas on the stability of the overall lighting modeling and estimation process. Compared to the modeling method using a unified spherical harmonic function, this invention effectively reduces systematic biases generated during window lighting modeling. 4. This invention proposes to use a pruning function based on an exponential function. The radiance of a three-dimensional Gaussian By compressing to a low dynamic range, the problem of gradient explosion or vanishing caused by the extremely uneven distribution of 3D Gaussian radiance data and the extreme values ​​in radiance affecting the training of neural networks is solved, which affects the convergence speed and final performance of the network. 5. The three-dimensional Gaussian mixture set constructed by this invention can be used to render high dynamic range lighting maps at any location indoors, providing support for virtual-real fusion, realistic rendering, and immersive interactive experiences. Attached Figure Description

[0021] Figure 1 This invention relates to the design of a three-dimensional hybrid Gaussian illumination parameter estimation network. Figure 2 This is the three-dimensional Gaussian mixture representation of indoor lighting according to the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0023] like Figure 1 As shown, a method for continuous indoor illumination modeling and estimation based on three-dimensional Gaussian sputtering includes the following steps: Step 1: Acquire a single frame image of the indoor space. Indoor lighting maps generated from single-frame images ; Given input video A single-frame image with a known pose and limited field of view at a given time. If we directly map the illumination to three-dimensional Gaussian mixture using a neural network, it is a severely unconstrained inverse problem. Therefore, this invention adopts a two-step approach of stepwise approximation: first, we perform generative reconstruction of the indoor illumination map, and then estimate the three-dimensional Gaussian mixture illumination. By learning the prior knowledge of illumination distribution during the generative reconstruction of the indoor illumination map, we can alleviate the ill-posedness of the illumination problem. Generative adversarial networks (GANs) can be used to reconstruct indoor lighting maps. The basic idea is to train the generator to produce images that resemble real lighting maps through generative adversarial training. Similar lighting maps; specifically, generators The optimization objective is: (1) The first term is the illumination map reconstruction error; the second term is the discrimination loss. It is an L1 norm; As the weight, it is set to 0.001 in this embodiment.

[0024] For discriminator The goal is to distinguish between generated lighting maps and real lighting maps: (2) After training, the generator Able to input image Convert to indoor lighting diagram .

[0025] In addition, diffusion models can also be used as an alternative method for generative reconstruction of indoor lighting maps.

[0026] Step 2: Utilize indoor lighting diagrams Extraction depth D , will depth D Indoor lighting diagram Input a 3D Gaussian mixture lighting parameter estimation network and output lighting parameter estimates for position, shape, and radiance. The goal of 3D Gaussian mixture illumination parameter estimation is to use indoor illumination maps As input, the three-dimensional Gaussian parameters of each pixel are estimated using a neural network to obtain the initial three-dimensional Gaussian mixture. ; Specifically, given an indoor lighting diagram and estimated depth map Estimated depth map Existing depth estimation networks can be used; First use the encoder from and Deep features are extracted and then processed by three Gaussian parameter decoders. For each pixel predict The group includes Gaussian parameters such as position, shape, and radiance. ; The appropriate option can be selected based on the complexity of the actual application scenario. In this embodiment... Set it to 3.

[0027] The encoder includes a multi-layer convolutional structure, which sequentially includes convolutional layers, normalization layers, and nonlinear activation layers. It extracts multi-scale features step by step through downsampling operations. Through the multi-layer convolutional structure, the encoder maps the input image into a low-resolution, high-channel-number depth feature representation to characterize the illumination distribution characteristics in indoor scenes.

[0028] The Gaussian parameter decoder predicts the parameters of multiple three-dimensional Gaussian lighting units based on the lighting features output by the encoder, so as to achieve continuous three-dimensional modeling of indoor lighting. In this embodiment, the Gaussian parameter decoder receives the depth features output by the encoder as input and maps the features to multiple parameter regression branches through at least one fully connected network. The multiple parameter regression branches are used to predict different parameters of the three-dimensional Gaussian lighting units.

[0029] The entire 3D Gaussian mixture lighting parameter estimation network comprises two key designs: first, it employs a layered Gaussian approach to simultaneously estimate both the observed indoor lighting and the occluded indoor lighting, i.e., for each pixel... predict Gaussian layer; Specifically, the first Mean of Gaussian layer Use depth and displacement Jointly stated: , It is the camera calibration matrix; the first Depth of Gaussian layer Then it can be given by the following formula: (3) in, For depth map Medium pixel The depth value at that location; It is the first Layer depth offset.

[0030] Clearly, for the depth offset of the first layer of Gaussians... Its value is 0, used to model observed indoor lighting; for occluded indoor lighting, the depth offset... Not negative, thus ensuring the first The layer Gaussian is always located behind the first layer Gaussian, enabling the estimation of shading illumination.

[0031] Secondly, it is proposed to design multiple independent three-dimensional Gaussian radiance parameter decoders. To achieve different radiation parameters Gaussian mixture estimation; 3D Gaussian Radiance Parameter Decoder Used according to decoder Output illumination characteristics and radiometric characteristics Predicting parameters of multiple 3D Gaussian illumination units to achieve continuous 3D modeling of indoor lighting; based on the current pixel illumination characteristics The displayed illumination type will show the radiance characteristics. The signal is fed to the Gaussian radiance parameter decoder under the corresponding illumination type to obtain the Gaussian radiance.

[0032] In this embodiment, and Similar in structure, take over The output depth features are used as input, and the features are mapped to radiative parameters through at least one fully connected network. .

[0033] Modeling three-dimensional continuous indoor illumination is the foundation and core of indoor illumination estimation. In order to achieve spatial continuity and high-precision representation of indoor illumination, this invention designs a continuous indoor illumination representation model based on three-dimensional Gaussian mixture. On this basis, a single-frame indoor illumination estimation method in step two is proposed, including generative indoor illumination map reconstruction and three-dimensional Gaussian mixture illumination parameter estimation.

[0034] Existing methods typically represent indoor illumination as three-dimensional uniform voxels. While this can improve the spatial consistency of indoor illumination estimation to some extent, it ignores the spatial continuity of indoor illumination and is not conducive to accurately representing the continuous spatial changes of dynamic illumination. like Figure 2 Therefore, the present invention intends to represent indoor lighting as... A set of three-dimensional Gaussians This allows for the representation of three-dimensional continuous indoor lighting, as shown in the formula: (4) in, It is the three-dimensional Gaussian opacity; and These are the mean and covariance matrices, respectively, controlling the three-dimensional Gaussian function. Location Rotation and scaling ; It depends on the direction of light incidence. Radiance function; This invention proposes to divide the three-dimensional Gaussian into three types: light source, window, and others, and to construct radiometric functions that conform to the corresponding types, thereby achieving high-precision indoor lighting characterization; other types refer to other indoor scene lighting objects besides light sources and windows, which may include walls, floors, and tabletops, etc.

[0035] For three-dimensional Gaussian radiance functions belonging to other categories The main focus is on describing the high-frequency details of the incident light, therefore spherical harmonic functions are proposed. It can be represented as: (5) In the formula, It is a spherical harmonic parameter. These are orthogonal basis functions on a sphere. It is the spherical harmonic order.

[0036] The spherical harmonic order will be determined based on the actual application scenario; in this example... ; Indoor light sources are typically directional; the following aperture function is proposed. The formula is: (6) In the formula, It refers to the direction of the opening; Control the aperture size; based on this, the three-dimensional Gaussian radiance function belonging to the light source category... It can be defined as follows: (7) in, It refers to the brightness of the light source.

[0037] This invention constructs a dedicated three-dimensional Gaussian radiance function for different types of light sources, which can accurately describe the bright areas near the light source and their spatial attenuation process, thereby significantly improving the accuracy and realism of indoor lighting modeling and estimation.

[0038] Furthermore, while interior windows are not actively emitting light sources, daytime indoor lighting primarily relies on sunlight provided by them. Therefore, to accurately characterize the impact of sunlight on indoor lighting, a three-dimensional Gaussian radiance function belonging to the window category is used. We propose to superimpose a term representing solar illumination onto the spherical harmonic function representation, with the following formula: (8) in, ; It is the direction and The cosine of the angle formed; This is the directional diffusion width, which can be set to 0.05 in this embodiment; It is the brightness of the sun; It is the visibility term of the sun; The invention is designed The function is used to describe the concentration of sunlight over a small area, i.e., the direction of the sun. .

[0039] This invention constructs a dedicated three-dimensional Gaussian radiance function for window-type lighting, making the radiance related to the direction of solar incidence, and independently models the high-intensity directional radiation components entering the room through the window. This allows for a more accurate characterization of the directional lighting effect formed by natural light entering through the window, while effectively reducing the impact of bright areas on the stability of the overall lighting modeling and estimation process. Compared with the modeling method using a unified spherical harmonic function, this invention can effectively reduce the systematic bias generated in the window lighting modeling process.

[0040] In summary, indoor lighting can be further represented as the set of all three-dimensional Gaussian mixtures. ,in .

[0041] Furthermore, due to the very wide dynamic range of indoor lighting, the data distribution of three-dimensional Gaussian radiance is extremely uneven. Extreme values ​​in the radiance may cause gradient explosion or vanishing problems during subsequent neural network training, affecting the network's convergence speed and final performance. Therefore, a pruning function based on an exponential function is proposed. The radiance of a three-dimensional Gaussian Compressed to low dynamic range : (9) in, It is a hyperparameter that controls the transformation of the function from linear to exponential decay, and can be appropriately selected according to the actual application scenario.

[0042] Finally, the set of all the above three-dimensional Gaussian mixtures They jointly defined the opacity function and radiance function of the indoor light radiation field: (10) in, Indoor lighting Opacity at the location; It is the light shining on Along Radiance in a direction.

[0043] Based on the indoor light radiation field, a high dynamic range lighting map can be rendered at any location indoors according to the classical radiation-absorption equation.

[0044] High dynamic range lighting maps can be used in augmented reality, such as product visualization and home makeup / fitting trials; they can also be used for industrial maintenance and guidance, such as overlaying virtual operating instructions, data tags, or pipelines onto real equipment; and they can also be used in games and entertainment, such as developing AR games based on real space, where virtual characters and effects can interact with indoor lighting in real time, greatly enhancing the sense of immersion.

[0045] High dynamic range lighting maps can also be used in virtual reality and metaverse, film and game production, computer vision and robotics, etc.

[0046] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for continuous indoor illumination modeling and estimation based on three-dimensional Gaussian sputtering, characterized in that, Includes the following steps: Step 1: Acquire single-frame images of the indoor environment and generate an indoor illumination map using the single-frame images; Step 2: Extract the depth map using the indoor illumination map, and input the depth map and indoor illumination map into a 3D Gaussian mixture illumination parameter estimation network to output the estimated values ​​of illumination parameters for position, shape, and radiance.

2. The method for continuous indoor illumination modeling and estimation based on three-dimensional Gaussian sputtering according to claim 1, characterized in that, The 3D Gaussian mixture illumination parameter estimation network includes: Using encoder From indoor lighting diagram and depth map Extracting depth features from; Three Gaussian parameter decoders were used respectively. For each pixel predict The group includes Gaussian parameters such as position, shape, and radiance. Characterization of indoor lighting output ;in, It is the three-dimensional Gaussian opacity; and These are the mean and covariance matrices, respectively. It depends on the direction of light incidence. The radiance function.

3. The method for continuous indoor illumination modeling and estimation based on three-dimensional Gaussian sputtering according to claim 2, characterized in that, mean The formula is: in, It is the camera calibration matrix; In pixels; For depth; For displacement; ; For depth map The depth value at that location; This is for depth offset.

4. The method for continuous indoor illumination modeling and estimation based on three-dimensional Gaussian sputtering according to claim 2, characterized in that, Radiance includes: Gaussian radiance of the light source, Gaussian radiance of the window, and other radiance.

5. The method for continuous indoor illumination modeling and estimation based on three-dimensional Gaussian sputtering according to claim 4, characterized in that, The formula for the Gaussian radiance of a light source is: in, It refers to the direction of the opening; Control the size of the opening; It refers to the brightness of the light source; It is a spherical harmonic function.

6. The method for continuous indoor illumination modeling and estimation based on three-dimensional Gaussian sputtering according to claim 4, characterized in that, The formula for Gaussian radiance of a window: in, ; It is the direction and The cosine of the angle formed; It is the directional diffusion width; It is the brightness of the sun; It is the visibility term of the sun.

7. The method for continuous indoor illumination modeling and estimation based on three-dimensional Gaussian sputtering according to claim 4, characterized in that, Using clipping functions radiance Compress to a low dynamic range.

8. The method for continuous indoor illumination modeling and estimation based on three-dimensional Gaussian sputtering according to claim 7, characterized in that, The formula for the clipping function is: in, It's a hyperparameter.

9. The method for continuous indoor illumination modeling and estimation based on three-dimensional Gaussian sputtering according to claim 2, characterized in that, Indoor illumination is characterized by the opacity function and radiance function of the indoor illumination radiation field.

10. A continuous indoor illumination modeling and estimation system based on three-dimensional Gaussian sputtering, characterized in that, include: Memory is used to store instructions that can be executed by the processor; A processor for executing instructions to implement the continuous indoor illumination modeling and estimation method based on three-dimensional Gaussian sputtering as described in any one of claims 1-9.

Images