A method for indoor scene illumination estimation based on local to global completion strategy

Abstract

The present invention discloses a method for indoor scene illumination estimation based on a local-to-global completion strategy. This method aims to estimate the illumination conditions at any position in a single restricted-view indoor scene image and decomposes the indoor scene illumination estimation problem into three subtasks: depth-based image warping, low dynamic range (LDR) panorama completion, and high dynamic range (HDR) reconstruction. Based on the second subtask, this method proposes a local-to-global panorama completion strategy. This strategy first performs depth-guided local completion on the sparse panorama output after the first stage of warping to fill small and dense holes. Then, a reasonable global completion is performed using a cubemap projection method and a transformer network (called PanoTransformer) capable of fitting long-range dependencies. Finally, a physically reasonable panorama with texture details can be restored at any position in the input image to capture the spatially varying indoor illumination.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and computer graphics, and in particular relates to an indoor scene illumination estimation method based on a local to global completion strategy. Background Art

[0002] Estimating high dynamic range (HDR) indoor lighting from a single-view low dynamic range (LDR) image is a fundamental problem in computer vision and graphics. It has a huge impact on many applications, including augmented reality and mixed reality (AR / MR), scene understanding, and relighting. In AR / MR applications, to create a believable and immersive experience, the reflections and other lighting effects of inserted virtual objects must match the real-world background to ensure highly realistic lighting consistency between virtual and real objects. However, since the pixel intensity in the observation image is a complex function of the scene geometry, material properties, and lighting distribution, especially for an indoor scene image with a limited field-of-view (FOV), different combinations of lighting and surface reflections will produce the same pixel intensity, which increases the difficulty of the problem.

[0003] With the advent of deep learning and large-scale datasets, convolutional neural networks (CNNs) are now the de facto architecture for solving this problem. For example, Song and Funkhouser et al. proposed Neural Illumination (SONG S, FUNKHOUSER T. Neural illumination: Lighting prediction for indoor environments[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 6918-6926). They decompose the indoor illumination estimation problem into three subtasks: depth-based image warping, LDR panorama completion, and HDR panorama reconstruction. The success of this pipeline lies in the second subtask, which recovers missing content from incomplete panoramas. However, in the process of implementing the present invention, the inventors found that it is unreasonable to directly apply the existing CNN-based image completion network to incomplete panoramas because (1) the hole area of ​​the deformed image is very large (generally greater than 60%), which increases the difficulty of completion; (2) the panorama has serious spatial distortion, which will mislead the CNN-based feature extraction module; (3) CNN has a linear receptive field, which makes it difficult to capture the long-distance dependencies that are prevalent in panoramas. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide an indoor scene lighting estimation method based on a local to global completion strategy in response to the shortcomings of the existing technology.

[0005] Technical Solution: To solve the above technical problems, the present invention discloses a method for indoor scene illumination estimation based on a local-to-global completion strategy. The method is used to estimate the illumination conditions at a specified location from a single indoor scene image, and includes the following steps:

[0006] Step 1: Based on the LDR image I with limited viewing angle and the specified position R in the LDR image, a sparse panoramic image centered at position R is obtained.

[0007] Step 2: the sparse panorama centered at position R Perform local to global completion to obtain a complete LDR panorama P with reasonable structure and texture details G ;

[0008] Step 3: GPerform HDR highlight reconstruction to obtain the HDR panoramic environment map P corresponding to the R position HDR , complete the indoor scene lighting estimation.

[0009] Furthermore, step 1 includes:

[0010] Step 11: Use DPT to predict the pixel-by-pixel depth of the LDR image I with restricted viewing angle to obtain the corresponding depth map D;

[0011] Step 12: Back-project the points on the LDR image I from the pixel space to the three-dimensional world coordinate space according to the depth map D;

[0012] Step 13: Take the three-dimensional coordinate corresponding to position R as the spherical center, calculate the depth and spherical pitch angle values ​​of the remaining three-dimensional coordinates relative to the coordinate, and perform a spherical projection onto the panorama to obtain a sparse depth panorama centered at position R. and LDR panorama

[0013] Furthermore, step 2 is for sparse LDR panorama The process of completing from local to global is formulated as follows:

[0014]

[0015] Among them, P G represents the LDR panorama after local to global completion, and M represents the LDR panorama that distinguishes sparse A binary mask indicating whether a pixel is visible, ⊙ represents pixel-by-pixel multiplication, L and G represent local completion and global completion, respectively.

[0016] Furthermore, step 2 includes the following steps:

[0017] Step 21: Based on the partially completed panorama P L The mask image M is generated from the hole area of ​​​​the image, and paired with the corresponding complete panorama ground truth to form a set of indoor scene panorama completion task datasets;

[0018] Step 22: Based on the sparse depth panorama For sparse panoramas Perform depth-guided local completion to obtain a local completed panorama P L ;

[0019] Step 23, according to step 22, the mask map in the data set in step 21 is updated and trained to obtain PanoTransformer, and then the partially completed panorama P is trained. L Perform global completion to obtain a complete LDR panorama P with reasonable structure and texture detailsG Furthermore, step 21 includes the following steps:

[0020] Step 211, collecting indoor panoramic image datasets from three datasets: Matterport3D, SUN360, and Laval;

[0021] Step 212: deform the image based on the depth information provided by the Matterport3D dataset and the position points provided by Neural Illumination in the dataset to obtain a sparse panoramic depth map, perform local completion on the depth map, and generate a mask map M based on the hole areas of the locally completed panoramic depth map.

[0022] In step 213 , the mask image M is randomly paired with the indoor panoramic image obtained in step 211 to obtain a set of indoor panoramic image completion datasets.

[0023] Furthermore, step 22 includes the following steps:

[0024] Step 221: Fill the sparse depth panorama using morphological operations and bilateral filtering The dense and small hole areas in the image are filled to obtain the depth panorama D′;

[0025] Step 222: Setting the initial partially completed panorama The threshold is t(0 <t<1);

[0026] Step 223: Select a padded pixel d in the padded depth panorama D′ and calculate the pixel coordinate c of d. p ; Then c p Back-projection to the world coordinate system to obtain its world coordinate c w ; Then c w Project back to the pixel coordinate system of the depth map D to obtain the pixel coordinate c;

[0027] Step 224: compare the depth value of the depth map D at coordinate c with the error of d; if the absolute value of the error is less than t, then P L The RGB value at coordinate c is equal to the RGB value of image I at coordinate c, that is, P L [c]=I[c];

[0028] Step 225: traverse the filled pixels in the filled depth panorama D′ and repeat steps 223 to 224 to obtain the partially completed panorama P. L Furthermore, step 23 updates the mask map in the data set of step 21 according to step 22, and then trains the data set of step 22 through the designed PanoTransformer network, and performs the local completion on the panorama P LPerform global structure completion and then back-project onto the spherical surface to generate a complete LDR panorama P G , the PanoTransformer network design is as follows:

[0029] PanoTransformer includes 12 transformer blocks as encoders and 6 residual blocks as decoders. PanoTransformer takes a cubemap as input. The cubemap has 6 sides and 4 channels. The 3-channel indoor panorama and 1-channel mask map in the dataset produced in step 22 are concatenated in the channel dimension and projected onto the cube. The input is represented as x∈R 6×H×W×4 , where H = 128 and W = 128 are the height and width of each face in the cubemap. In the encoder, the input x is first reorganized into a series of unfolded 2D blocks, represented as Where p=16 represents the resolution of each block, Represents the number of blocks; each block is then mapped into a one-dimensional vector through a trainable linear layer, represented by z∈R d , where d = 1024, and the position information is retained by position embedding; these vectors are then input into the transformer block, each transformer block contains a multi-head self-attention mechanism MHSA layer, a normalization layer LN and a multi-layer perceptron MLP; the input of the lth transformer block is represented as z l , 1≤l≤12, the output is expressed as:

[0030] w l =MHSA(LN(z l ))+z l

[0031] z l+1 =MLP(LN(w l ))+w l

[0032] where w l Represents the intermediate features after the lth layer MHSA. After passing through the encoder, the output result is remapped back to the cube map z∈R 6×H×W×4 After 6 residual blocks, the global completed cube map is obtained, and then the complete LDR panorama P is obtained through back projection. G .

[0033] Furthermore, step 3 includes: reconstructing the complete LDR panorama P after local to global completion by using a deep learning-based HDR reconstruction method. G Perform HDR lighting reconstruction to obtain the final HDR panoramic environment map P corresponding to position R HDR .

[0034] Furthermore, the method further includes step 4, wherein the HDR panoramic environment map P corresponding to the R position is used. HDR Render the virtual object at position R.

[0035] Beneficial effects:

[0036] The present invention adopts a local-to-global completion strategy, combined with a transformer-structured network and a collected seamless, artifact-free panoramic image dataset to estimate indoor scene lighting. This solves the large-area (>60%) completion problem that is difficult to solve due to the limited receptive field of conventional CNNs, as well as the distortion and consistency issues during panoramic image completion. It enables the prediction of a panorama with reasonable structure and texture details centered at any position in the indoor input image, ultimately achieving a consistent coloring rendering effect for virtual object insertion. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0038] Figure 1 Schematic diagram of the overall process of the method of the present invention.

[0039] Figure 2 Schematic diagram of the image deformation process in the present invention.

[0040] Figure 3 These are the attention maps and attention score maps of the sparse and dense panoramas in this invention after passing through the 9th transfromer block of PanoTransfromer respectively.

[0041] Figure 4 Schematic diagram of the Neural Illumination dataset.

[0042] Figure 5 Schematic diagram of the data set in the present invention.

[0043] Figure 6 This is a comparison chart of the illumination prediction effects of the present invention and various methods.

[0044] Figure 7 A comparison chart of the lighting rendering effects of different objects predicted by the present invention and various methods.

[0045] Figure 8 This is a user study comparison chart of the lighting prediction effects of the present invention and various methods.

[0046] Figure 9 The figure is a comparison chart of the completion effects of the present invention and various methods.

[0047] Figure 10 2 is a comparison chart of the ablation experimental effects of the present invention.

[0048] Figure 11 This is a comparison chart of the training effects of the present invention and the Neural Illumination dataset.

[0049] Figure 12 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0050] The embodiments of the present invention will be described below with reference to the accompanying drawings.

[0051] For a given indoor scene image I taken from a single viewpoint and a specified position R in the scene image, this embodiment completes the illumination estimation of the specified position in the single restricted view indoor scene image through the following steps. The flowchart is as follows: Figure 1 and Figure 12 As shown:

[0052] Step 1: Based on the LDR image I with limited viewing angle and the specified position R in the LDR image, a sparse panoramic image centered at position R is obtained. The specific steps are as follows:

[0053] Step 11: Use DPT to predict the pixel-by-pixel depth of the input image I to obtain the corresponding depth map D.

[0054] Step 12: Back-project the points on the LDR image I from the pixel space to the three-dimensional world coordinate space according to the depth map D. In this embodiment, the camera coordinate system is set as the world coordinate system, and the camera intrinsic parameter f is set as x 、f y is the width and height of image I, c x 、c y The point on the LDR image I is converted from the pixel space to the image space, and then the point in the image space is back-projected into the 3D world coordinate space according to the depth map D.

[0055] Step 13: Take the three-dimensional coordinate corresponding to the selected position R as the spherical center, calculate the depth and spherical pitch angle values ​​of the remaining three-dimensional coordinates relative to the center coordinate, and project the remaining three-dimensional coordinates (including relative depth, spherical pitch angle and RGB information) from the spherical coordinate system to the panoramic pixel coordinate system to obtain a sparse depth panoramic image centered at the R position. and LDR panorama

[0056] Step 2: the sparse panorama centered at position R Perform local to global completion to obtain a complete LDR panorama P with reasonable structure and texture details G;

[0057] Due to the limited viewing angle of the input image I, There are large areas of holes in the image; the pixel stretching caused by the deformation process in step 1 is Another factor of sparseness. For sparse LDR panoramas The process of completing from local to global can be formulated as:

[0058]

[0059] Among them, P G represents the LDR panorama after local to global completion, and M represents the LDR panorama that distinguishes sparse The binary mask image of whether the pixel is visible, ⊙ represents pixel-by-pixel multiplication, L and G represent local completion and global completion respectively. The specific steps are as follows:

[0060] Step 21: Based on the deformed panorama The mask map M is generated from the hole area of ​​​​the image, and paired with the complete panorama to form a set of indoor panorama completion task datasets.

[0061] Step 22, first based on the sparse depth panorama For the deformed sparse panorama Perform local completion to fill the dense small holes caused by pixel stretching during the deformation process, alleviate sparsity, and obtain P L .Will

[0062] Step 23, update the mask map in the above dataset according to step 22, and train the PanoTransformer, and then perform the local completion on the panorama P L Perform global completion to obtain a complete LDR panorama P with reasonable structure and texture details G .

[0063] In step 21, the data set is prepared as follows:

[0064] Step 211, collecting indoor panoramic image datasets from three datasets: Matterport3D, SUN360, and Laval;

[0065] Step 212: deform the image based on the depth information provided by the Matterport3D dataset and the position points provided by Neural Illumination in the dataset to obtain a sparse panoramic depth map, perform local completion on the depth map, and generate a mask map M based on the hole areas of the locally completed panoramic depth map.

[0066] In step 213 , the mask image M is randomly paired with the indoor panoramic image obtained in step 211 to obtain a set of indoor panoramic image completion datasets.

[0067] In step 22, the specific steps of partial completion are as follows:

[0068] Step 221: Fill the sparse depth panorama using morphological operations and bilateral filtering The dense and small hole areas in the image are filled to obtain the depth panorama D′;

[0069] Step 222: Setting the initial partially completed panorama The threshold is t(0 <t<1);

[0070] Step 223: Select a padded pixel d in the padded depth panorama D′ and calculate the pixel coordinate c of d. p ; Then c p Back-projection to the world coordinate system to obtain its world coordinate c w ; Then c w Project back to the pixel coordinate system of the depth map D to obtain the pixel coordinate c;

[0071] Step 224: compare the depth value of the depth map D at coordinate c with the error of d; if the absolute value of the error is less than t, then P L The RGB value at coordinate c is equal to the image, the RGB value at coordinate c, that is, P L [c]=I[c];

[0072] Step 225: traverse the filled pixels in the filled depth panorama D′ and repeat steps 223 to 224 to obtain the partially completed panorama P. L Step 23, update the mask map in the data set of step 21 through step 22, and train the designed PanoTransformer network on the data set, and perform local completion on the panorama P L Perform global structure completion and then back-project onto the spherical surface to generate a complete LDR panorama P G , the PanoTransformer network design is as follows:

[0073] PanoTransformer consists of 12 transformer blocks as encoders and 6 residual blocks as decoders. It takes a cubemap as input. The cubemap has 6 sides and 4 channels. The 3-channel indoor panorama and 1-channel mask map from the dataset produced in step 22 are concatenated in the channel dimension and projected onto the cube. The input is represented as x∈R 6×H×W×4, where H = 128 and W = 128 are the height and width of each face in the cubemap. In the encoder, the input x is first reorganized into a series of unfolded 2D blocks, represented as Where p=16 represents the side length of each square, Represents the number of blocks. Each block is then mapped into a one-dimensional vector through a trainable linear layer, represented by z∈R d , where d = 1024, and the position information is preserved through position embedding. These vectors are then fed into the transformer block, each of which contains a multi-head self-attention mechanism MHSA layer, a normalization layer LN, and a multi-layer perceptron MLP. The input to the lth transformer block is represented as z l , the output can be expressed as:

[0074] w l =MHSA(LN(z l ))+z l

[0075] z l+1 =MLP(LN(w l ))+w l

[0076] where w l Represents the intermediate features after the lth layer MHSA. After passing through the encoder, the output result is remapped back to the cube map z∈R 6×H×W×4 After 6 residual blocks, the global completed cube map is obtained, and then the complete LDR panorama P is obtained through back projection. G The training loss function of PanoTransformer is Huber and Patch-GAN based adversarial loss.

[0077] Step 3: G Perform HDR highlight reconstruction to obtain the HDR panoramic environment map P corresponding to the R position HDR , complete the indoor scene lighting estimation.

[0078] The complete LDR panorama P after local to global completion is reconstructed using a deep learning-based HDR reconstruction method. G Perform HDR lighting reconstruction to obtain the final HDR panoramic environment map P corresponding to the R position HDR .

[0079] More specifically, this embodiment first predicts the scene depth and deforms a single indoor scene image I input by the user to obtain a sparse panoramic image centered at the specified position; then completes the sparse panoramic image from local to global; and finally obtains an HDR panoramic image through HDR reconstruction.

[0080] Example

[0081] This embodiment aims to estimate a complete HDR environment map P from a position R of a single-view indoor scene image I. HDR This embodiment follows the framework of Neural Illumination and decomposes the problem into three subtasks: depth-based image warping, LDR panorama completion, and HDR panorama reconstruction. The following describes the various steps of the present invention based on the embodiment.

[0082] Step 1: Depth-based image deformation.

[0083] like Figure 1 , given an indoor scene image I taken from a single viewpoint and a specified position R in the image, the existing method is used to estimate the depth map D of I. Then D and I are geometrically deformed into a sparse 360° LDR spherical panorama centered at the selected position R, represented as and

[0084] Step 11: Use the deep learning-based method DPT to predict the pixel-by-pixel depth of the input image I to obtain the corresponding depth map D. For details, see the reference: Ranftl R, Bochkovskiy A, Koltun V. Vision transformers for dense prediction [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 12179-12188.

[0085] Step (12): Set the camera coordinate system to the world coordinate system, and the camera internal parameter f x 、f y is the width and height of image I, c x 、c y The point on the LDR image I is converted from the pixel space to the image space, and then the point in the image space is back-projected into the three-dimensional world coordinate space according to the depth map D; the three-dimensional coordinate point and the corresponding RGB information are obtained.

[0086] Step (13): Take the three-dimensional coordinate corresponding to position R as the spherical center, calculate the depth and spherical pitch angle of the remaining three-dimensional coordinates relative to the center coordinate, and project the remaining coordinates from the spherical coordinate system to the panoramic image pixel coordinate system to obtain a sparse depth panoramic image centered at position R. and LDR panorama in and The size is 256×512.

[0087] Step (2), LDR panorama completion.

[0088] like Figure 2 As shown in Figure 1, due to the limited viewing angle of the input image I and the pixel stretching involved in the deformation operation, the sparsity problem of the deformed panorama is unavoidable. The missing areas caused by the deformation operation are classified as pixel stretched areas and out-of-view areas. For panorama completion, most previous methods use fully convolutional networks. However, CNN-based models have achieved suboptimal performance due to the large areas of sparse missing content in the deformed panorama and some inherent limitations of the convolutional layer. CNN is good at preserving local structures and modeling rich textures, but cannot complete large areas of void areas. Therefore, previous works have found it difficult to obtain sufficiently broad context and important high-level representations from sparse panoramic images. At the same time, the distortion of spherical panoramas will further hinder the performance of large-area completion. Compared with CNN models with limited receptive fields, transformers are designed to support long-range interactions through self-attention modules. The self-attention mechanism can directly calculate the interactions between any pair of blocks, naturally capture long-range dependencies, and have a global receptive field at each stage. However, transformers work poorly directly on sparse inputs. Figure 3 This illustrates the point. Figure 3 The attention matrices and attention maps of the 9th transformer block for blocks selected from sparse and dense panoramas are compared in

[15] . Figure 3 As shown in the figure, given a dense input, the query block affects some areas (such as the ground, marked in the attention map) and contains sufficient lighting information. However, the transformer block structure has difficulty recovering the global structure from scattered pixels, and the lighting cannot correctly transfer information to the invisible blocks, resulting in smoothing of the attention map. Therefore, the present invention first fills the pixel stretching area based on the adjacent pixels to alleviate the sparsity, and then fills other large holes based on the global understanding of the entire scene. To this end, a novel local-to-global completion strategy is proposed, which is expressed as follows:

[0089]

[0090] Among them, P G represents the LDR panorama after local to global completion, and M represents the LDR panorama that distinguishes sparse A binary mask indicating whether the pixel is visible, ⊙ represents pixel-by-pixel multiplication, L and G represent the local completion module and the global completion module, respectively, which are applied to the deformed panorama in sequence. The local completion method of this embodiment aims to fill the dense holes in the pixel stretching area based on depth information. The local completion module adopts an improved bilateral filtering-based method to remove the dense and small holes in the pixel stretching area. Afterwards, a network model of global structure completion based on a new transformer is designed to extract reliable global features from the visible area and then fill the large holes in the area outside the field of view. The transformer architecture designed in this embodiment uses cube map projection as input to solve the spatial distortion problem in spherical signals, and is named PanoTransformer.

[0091] Step (21): construct a dataset for indoor panorama completion tasks.

[0092] Before completion, the designed global completion network module needs to be trained. Currently, the only dataset that contains paired LDR view images and corresponding HDR panoramas is proposed by Neural Illumination based on Matterport3D (Reference 2, CHANG A, DAI A, FUNKHOUSER T, et al. Matterport3D: Learning from RGB-D Datain Indoor Environments[J]. International Conference on 3D Vision(3DV), 2017.). However, the HDR panoramas reconstructed by this dataset have obvious artifacts (such as Figure 4 As shown, sutures and broken structures). This makes it impossible for the global completion model of this embodiment to infer a complete, globally consistent structure at any position. Taking the above problems into consideration, this method collects a large-scale, high-quality and diverse panoramic image dataset (such as Matterport3D, SUN360 (Reference 3, XIAO J, EHINGER KA, OLIVA A, et al. Recognizing scene viewpoint using panoramic place representation [C] / / 2012 IEEE Conference on Computer Vision and Pattern Recognition. 2012: 2695-2702.) and Laval (Reference 4, GARDNER MA, SUNKAVALLI K, YUMER E, et al. Learning to Predict Indoor Illumination from a Single Image [J]. ACM Trans. Graph., 2017, 36 (6).) Figure 5 In addition to panoramas, training PanoTransformer also requires masks to generate sparse inputs Since the invisible areas are mainly at the top of the panorama, this method generates masks from the Neural Illumination dataset instead of randomly generating them. These sparse masks are obtained by geometric deformation to adapt to the real-world data distribution. These masks are locally completed before being fed into the PanoTransformer. The main difference between our dataset and the Neural Illumination dataset is that the panoramas and masks are unpaired, so this method can randomly apply a variety of irregular masks to a panorama to generate a variety of inputs. Since this method focuses on the completion task, it does not require the mask and panorama to be physically related. Ultimately, this method collected 38,929 high-quality panoramas with randomly selected masks for training and 5,368 for evaluation.

[0093] Step (22), LDR panorama partial completion.

[0094] The purpose of local completion is to alleviate the sparse problem caused by pixel stretching. Since the depth value of the pixel stretching area changes smoothly, the panorama There may be rich textures, so morphological operations and bilateral filtering are first used to fill the depth of the deformation as much as possible. The holes in the ; then traverse the filled The effective depth value of the pixel in I is filled with the re-projected pixel value in I The hole area in the image is used to obtain the partially completed panorama P. L .

[0095] Step (23), global completion of the LDR panorama.

[0096] Based on the aforementioned dataset, this method designs and trains a transformer-based network, named PanoTransformer, to “hallucinate” content in out-of-view areas. The PanoTransformer can be logically separated into an encoder and a decoder, where the encoder captures undistorted representations of long-range information, while the decoder gradually recovers spatial information to produce accurate pixel-level predictions.

[0097] PanoTransformer consists of 12 transformer blocks as encoders and 6 residual blocks as decoders. It takes a cubemap as input. The cubemap has 6 sides and 4 channels. The 3-channel indoor panorama and the 1-channel mask map in the dataset produced in step 22 are concatenated in the channel dimension and projected onto the cube. The input is represented as x∈R6×H×W×4 , where H = 128 and W = 128 are the height and width of each face in the cubemap. In the encoder, the input x is first reorganized into a series of unfolded 2D blocks, represented as Where p=16 represents the side length of each square, Represents the number of blocks. Each block is then mapped into a one-dimensional vector through a trainable linear layer, represented by z∈R d , where d = 1024, and the position information is preserved through position embedding. These vectors are then fed into the transformer block, each of which contains a multi-head self-attention mechanism MHSA layer, a normalization layer LN, and a multi-layer perceptron MLP. The input to the lth transformer block is represented as z l , the output can be expressed as:

[0098] w l =MHSA(LN(z l ))+z l

[0099] z l+1 =MLP(LN(w l ))+w l

[0100] where w l Represents the intermediate features after the lth layer MHSA. After passing through the encoder, the output result is remapped back to the cube map z∈R 6×H×W×4 Due to the powerful representation ability and global receptive field of the transformer, these reconstructed prior features z contain sufficient clues of global structure and rough texture. z can be regarded as six images projected by the cube map. Finally, these image-like features will be fed back to the decoder to supplement the texture details.

[0101] Step (3): HDR illumination reconstruction is performed on the complete LDR panorama after local to global completion using the method of Santos et al. to obtain the final HDR panoramic environment map corresponding to position R.

[0102] The present invention is verified below:

[0103] Loss function and training details

[0104] The present invention optimizes PanoTransformer by minimizing the pixel-wise Huber loss between the predicted and reference panoramas. The Huber loss takes advantage of the pixel-wise l-norm loss function and the 2-norm loss function as shown below:

[0105]

[0106] Where y is the reference panorama, is the predicted value. In the experiments of the present invention, the threshold T is set to 0.2. In order to produce more realistic details, an additional adversarial loss is added during the training process. The discriminator of this method adopts the same architecture as Patch-GAN. The present invention uses the PyTorch framework, document 12Paszke A, Gross S, Massa F, et al. Pytorch: An imperative style, high-performance deep learning library [J]. Advances in neural information processing systems, 2019, 32, to implement our PanoTransformer. The Adam optimizer is used, document 13Kingma DP, Ba J. Adam: A method for stochastic optimization [J]. arXiv preprint arXiv: 1412.6980, 2014, with default parameters β1 = 0.9 and β2 = 0.999, and an initial learning rate of 0.0001. PanoTransformer was trained for 100 epochs on the dataset of the present invention. Training is performed on two NVIDIA RTX 3090 GPUs with a batch size of 8.

[0107] Comparison of experimental results

[0108] This method evaluates the performance of indoor lighting prediction and compares it with the most advanced methods. In order to further verify the effectiveness of the completion method from local to global, this method also calculates the completed result P G Compared with the results of Neural Illumination, the specific evaluation is as follows:

[0109] Lighting prediction evaluation

[0110] Qualitative comparison Figure 6As shown, experiments demonstrate the predicted lightmaps and corresponding rendering results of several state-of-the-art methods and our method. Gardner et al. regressed HDR lighting from images with limited viewpoints without making assumptions about scene geometry, material properties, or illumination. Because they predicted illumination for the entire scene, this method cannot model spatial variations. EMLight ignores complex scene geometry and simplifies the scene's illumination distribution using a Gaussian map. Therefore, it cannot handle occlusions. In contrast, our method estimates depth to better understand the scene, resulting in more accurate lighting estimation. Similar to our method, Neural Illumination also decomposes the task into subtasks. However, due to its limited receptive field, it has difficulty inferring lighting positions, especially when the input is very sparse. Because Neural Illumination does not share code and model weights, our method's dataset was used to implement and train their network in the experiments. As can be seen, in the local-to-global completion module, our method produces a detailed, panoramic environment map with accurate lighting and perceptually plausible features, ensuring realistic shading.

[0111] The experiment further conducted user research on 20 scenarios of Garon et al. and another 10 scenarios on the Internet. Figure 7 As shown, multiple virtual objects are inserted into these scenes. For the scenes from Garon et al., we re-render the rabbit model with diffuse material using reference lighting and lighting predicted by methods from Neural Illumination, Li et al., and the present invention. Li et al. achieve state-of-the-art performance by leveraging a deep inverse rendering framework to obtain complete scene reconstruction, estimating shape, spatially varying lighting, and surface reflectivity from a single RGB image. To reflect the details of the predictions, we render a specular sphere with the estimated lightmap in the remaining scenes and compare them through a user study. The user study was conducted by asking 84 users to choose which rendering method was more realistic between pairs of rendered images, and the results are shown in. Figure 8 For the scenes with the rabbit model inserted, Li et al. and our method both beat each other in half of the scenes, indicating that they are comparable in predicting the lighting distribution. However, Li et al. used a spherical Gaussian to model the lighting, causing the mirror ball to appear diffuse. For scenes 21-30, our method outperformed the other methods on the mirror ball, indicating that our method produces credible details that are consistent with the environment.

[0112] Quantitative comparison In order to evaluate the performance of the present method in lighting estimation, the experiment used the predicted panoramic environment map and the reference environment map to render spheres of different materials, namely diffuse D, matte silver S and mirror M, and compared the rendering results. This experiment uses some standard indicators, including root mean square error (RMSE) and mean absolute error (MAE) for evaluation. In order to evaluate the accuracy of the light source, this experiment uses the average angular error A between the reference light and the predicted light as the final angular error between the two HDR panoramas. All these indicators are widely used to evaluate lighting prediction. For evaluation, this experiment used 2000 pairs of input LDR images from the Laval dataset and the reference HDR panoramic environment map taken on the camera. The following table shows the quantitative comparison of the method of the present invention with Gardner et al., Neural Illumination and EMLight:

[0113] Table 1 Comparison of rendering indices of light predicted by the present invention and other methods on diffuse reflection, matte silver and mirror material balls

[0114]

[0115] As shown in Table 1, the effect of this method is better than other methods in comparison under different evaluation indicators and materials.

[0116] Panoramic image completion evaluation

[0117] Qualitative comparison In order to prove the effectiveness of the transformer-based network of the present invention for global completion, this experiment is conducted on Figure 9 The completion results of Neural Illumination and the method of the present invention are compared in

[15] . In particular, Neural Illumination is able to roughly generate coarse structures. However, the limited receptive field of the CNN limits its understanding of the global structure in the panorama. Furthermore, Neural Illumination does not use masks to preserve visible areas, so these areas may change after prediction. In contrast, the structure completed by the present invention has only minor inconsistencies and artifacts, outperforming Neural Illumination in global completion.

[0118] Quantitative comparative completion evaluation was conducted on our test set, which contains over 5,000 pairs of masked input and reference. Table 2 shows a quantitative comparison of our method with Neural Illumination. The average PSNR, SSIM, and FID values ​​are listed for the colorized LDR panoramas. Clearly, our method achieves superior results compared to Neural Illumination across all metrics.

[0119] Table 2 Comparison results of the present invention and Neural Illumination in completion indicators

[0120] method SSIM↑ PSNR↑ FID↓ Neural Illumination 0.30 14.85 255.60 Ours 0.68 21.44 39.36 -Cubemap 0.62 20.51 70.09 -Local 0.60 19.46 112.32 -GAN 0.68 21.50 76.07

[0121] Ablation experiments

[0122] In order to evaluate the effectiveness of the design in the method of the present invention, three model variants were developed in the experiment, which are denoted as the GAN-free model, the Cubemap-free model, and the Local-free model. The quantitative results are reported in Table 2. Figure 10 The performance of these model variants is qualitatively evaluated in

[15] .

[0123] -GAN represents the PanoTransformer of the present invention trained without GAN loss. Figure 10 As can be seen in the top row of the dataset, the model trained without GAN loss produces an overly smooth texture that is close to the average intensity of the surrounding area. With the help of GAN loss, the complete model of the present invention is able to produce high-frequency signals and conjure realistic details.

[0124] The goal is to remove distortion from panoramas using cubemap projection as input. To demonstrate the effectiveness of cubemap projection, the PanoTransformer is adapted to a cubemap model, which takes equirectangular projection as input and directly outputs an LDR panorama. Figure 10 As can be seen from the middle row of

[15] , the -Cubemap model has the problem of structural distortion. The complete model of the present invention is significantly better than the -Cubemap model, proving the superiority of cubemap projection in processing spherical signals.

[0125] In order to verify the importance of the local completion module of the present invention, this module was removed in the experiment and the sparse panoramic image was used directly. Train PanoTransformer. Figure 10 The bottom row in

[15] shows that the -Local model introduces artifacts to the predictions, which explains Figure 3 Attention maps for sparse inputs in

[15] . With the local completion module, our strategy produces more realistic and clearer results, indicating that the local completion module promotes the performance of PanoTransformer.

[0126] Dataset evaluation

[0127] like Figure 11, the model trained using the dataset of the present invention produces much better results, while the model trained using the dataset of NeuralIllumination is prone to produce messy structures due to the artificial traces in the dataset.

[0128] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium is capable of storing a computer program that, when executed by the data processing unit, can execute the invention content of the method for indoor scene illumination estimation based on a local-to-global completion strategy provided by the present invention and some or all of the steps in each embodiment. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0129] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a computer program, i.e., a software product. The computer program software product can be stored in a storage medium and includes a number of instructions for enabling a device including a data processing unit (which can be a personal computer, server, single-chip microcomputer, MUU or network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0130] The present invention provides a method for indoor scene illumination estimation based on a local-to-global completion strategy. There are numerous methods and approaches for implementing this technical solution. The foregoing description is merely a specific embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A method for indoor scene illumination estimation based on a local-to-global completion strategy, characterized in that: The following steps are involved: Step 1: Based on the LDR image I with limited viewing angle and the specified position R in the LDR image, a sparse panoramic image centered at position R is obtained. Step 2: the sparse panorama centered at position R Perform local to global completion to obtain a complete LDR panorama P with reasonable structure and texture details G ; Step 3: G Perform HDR highlight reconstruction to obtain the HDR panoramic environment map P corresponding to the R position HDR , complete the indoor scene lighting estimation; Step 2 For sparse LDR panoramas The process of completing from local to global is formulated as follows: Among them, P G represents the LDR panorama after local to global completion, and M represents the LDR panorama that distinguishes sparse A binary mask indicating whether the pixel is visible, ⊙ represents pixel-by-pixel multiplication, L and M represent local completion and global completion respectively; Step 2 includes the following steps: Step 21: Based on the deformed panorama The hole area of ​​​​generates a mask map M, which is paired with the complete panorama to form a set of indoor panorama completion task datasets; Step 22: Based on the sparse depth panorama For sparse panoramas Perform depth-guided local completion to obtain a local completed panorama P L ; Step 23: Update the mask map in the dataset according to step 22 and train it to obtain PanoTransformer, and then perform the training on the partially completed panorama P. L Perform global completion to obtain a complete LDR panorama P with reasonable structure and texture details G .

2. The indoor scene illumination estimation method based on the local-to-global completion strategy according to claim 1, characterized in that: Step 1 includes: Step 11: Use DPT to predict the pixel-by-pixel depth of the LDR image I with restricted viewing angle to obtain the corresponding depth map D; Step 12: Back-project the points on the LDR image I from the pixel space to the three-dimensional world coordinate space according to the depth map D; Step 13: Take the three-dimensional coordinate corresponding to position R as the spherical center, calculate the depth and spherical pitch angle values ​​of the remaining three-dimensional coordinates relative to the coordinate, and perform a spherical projection onto the panorama to obtain a sparse depth panorama centered at position R. and LDR panorama 3. The indoor scene illumination estimation method based on the local-to-global completion strategy according to claim 2, characterized in that: Step 21 includes the following steps: Step 211, collecting indoor panoramic image datasets from three datasets: Matterport3D, SUN360, and Laval; Step 212: deform the image based on the depth information provided by the Matterport3D dataset and the position points provided by Neural Illumination in the dataset to obtain a sparse panoramic depth map, perform local completion on the depth map, and generate a mask map M based on the hole areas of the locally completed panoramic depth map. In step 213 , the mask image M is randomly paired with the indoor panoramic image obtained in step 211 to obtain a set of indoor panoramic image completion datasets.

4. The indoor scene illumination estimation method based on the local-to-global completion strategy according to claim 3, characterized in that: Step 22 includes the following steps: Step 221: Fill the sparse depth panorama using morphological operations and bilateral filtering The dense and small hole areas in the image are filled to obtain the depth panorama D. ′ ; Step 222: Setting the initial partially completed panorama The threshold is t,0 <t<1; Step 223, the depth panorama D after filling ′ Select a padded pixel d and calculate the pixel coordinate c of d p ; Then c p Back-projection to the world coordinate system to obtain its world coordinate c w ; Then c w Project back to the pixel coordinate system of the depth map D to obtain the pixel coordinate c; Step 224: compare the depth value of the depth map D at coordinate c with the error of d; if the absolute value of the error is less than t, then P L The RGB value at coordinate c is equal to the RGB value of image I at coordinate c, that is, P L [c]=I[c]; Step 225, traverse the filled depth panorama D ′ Repeat steps 223 to 224 to obtain the partially completed panorama P L .

5. The indoor scene illumination estimation method based on the local to global completion strategy according to claim 4, characterized in that: Step 23 updates the mask map in the data set in step 21 according to step 22, trains the data set in step 21 through the designed PanoTransformer network, and performs local completion on the panorama P L Perform global structure completion and then back-project onto the spherical surface to generate a complete LDR panorama P G ; The PanoTransformer network design is as follows: PanoTransformer includes 12 transformer blocks as encoders and 6 residual blocks as decoders. PanoTransformer takes a cubemap as input. The cubemap has 6 sides and 4 channels. The 3-channel indoor panorama and 1-channel mask map in the dataset produced in step 22 are concatenated in the channel dimension and projected onto the cube. The input is represented as x∈R 6×H×W×4 , where H = 128 and W = 128 are the height and width of each face in the cubemap. In the encoder, the input x is first reorganized into a series of unfolded 2D blocks, represented as Where p=16 represents the side length of each square, Represents the number of blocks; each block is then mapped into a one-dimensional vector through a trainable linear layer, represented by z∈R d , where d = 1024, and the position information is retained by position embedding; these vectors are then input into the transformer block, each transformer block contains a multi-head self-attention mechanism MHSA layer, a normalization layer LN and a multi-layer perceptron MLP; the input of the lth transformer block is represented as z l , 1≤l≤12, the output is expressed as: In l =MHSA(LN(z l ))+z l With l+1 =MLP(LN(in l ))+in l where w l Represents the intermediate features after the lth layer of MHSA; After the encoder, the output is remapped back to the cubemap z∈R 6×H×W×4 After 6 residual blocks, the global completed cube map is obtained, and then the complete LDR panorama P is obtained through back projection. G .

6. The indoor scene illumination estimation method based on the local to global completion strategy according to claim 5, characterized in that: Step 3 includes: reconstructing the complete LDR panorama P after local to global completion by using a deep learning-based HDR reconstruction method. G Perform HDR lighting reconstruction to obtain the final HDR panoramic environment map P corresponding to position R HDR .

7. The indoor scene illumination estimation method based on the local to global completion strategy according to claim 6, characterized in that: Also includes step 4, according to the HDR panoramic environment map P corresponding to the R position HDR Render the virtual object at position R.

Images