Apparatus, data structure and method for predicting light maps

Abstract

An apparatus, data structure and method for predicting an illumination map, wherein the method comprises providing an input image (112) comprising a first illumination and a second illumination; determining a decomposition of the first illumination from the input image (112); determining a decomposition of the second illumination from the input image; and determining (506) an illumination map from the decompositions.

Description

Technical Field

[0001] This invention relates to an apparatus, data structure, and method for predicting illumination maps. Background Technology

[0002] Attention-based illumination decomposition (arXiv:2402.18277v1) is a mechanism for solving multi-source white balance problems by decomposing the mixed illumination map into chromaticity vectors and weight maps for each light source. Summary of the Invention

[0003] A method for predicting a lighting map includes: providing an input image comprising a first lighting and a second lighting; determining a decomposition of the first lighting based on the input image; determining a decomposition of the second lighting based on the input image; and determining a lighting map based on these decompositions. The input image comprises pixels. The method predicts these decompositions based on the pixels of the input image.

[0004] Determining the decomposition may include: determining a first chromaticity vector and a first weight map for a first illumination based on the input image; and determining a second chromaticity vector and a second weight map for a second illumination based on the input image, wherein determining the illumination map includes determining the illumination map based on the weight map and the chromaticity vector. This means that the method predicts the weight map and the chromaticity vector based on these pixels of the input image.

[0005] Determining the decomposition of the first illumination may include: specifically using a first encoder configured to encode an input image into a first embedding, determining the first embedding based on the input image; specifically using a second encoder configured to encode an input image or a resized input image into a second embedding, determining the second embedding based on the input image; and specifically using a decoder to determine a first chroma vector and a first weight map based on these embeddings, wherein the decoder is configured to decode a decoder input dependent on these embeddings into a first chroma vector and a first weight map. These embeddings include features extracted from the input image or the resized input image in the embedding space. The decoder can process this information efficiently.

[0006] Determining the decomposition of the second illumination may include, in particular, using a decoder to determine a second chromaticity vector and a second weight map based on the embedding, wherein the decoder is configured to decode the decoder's input into the second chromaticity vector and the second weight map.

[0007] The method may include: determining a second embedding using a second encoder, wherein the second encoder includes convolutional layers, wherein determining the second embedding includes: obtaining the second embedding from the outputs of different convolutional layers of the second encoder. The second embedding from the plurality of convolutional layers contains spatial information, which is combined with a first embedding to incorporate spatial information.

[0008] To train a decoder to determine a chroma vector and a weight map, the method may include: providing a ground truth for the chroma vector and a ground truth for the weight map; and training the decoder according to a loss function comprising: a first term that depends on the chroma vector and the ground truth for the chroma vector; and a second term that depends on the weight map and the ground truth for the weight map.

[0009] The method may include: decomposing the input image into a first single-light source scene for a first illumination and a second single-light source scene for a second illumination, and predicting the illumination decomposition, such as a weight map and a chromaticity vector, from the single-light source scenes respectively.

[0010] The input image includes pixels, wherein the first weight map associates each pixel of the input image with a weight, wherein the method includes: determining whether the weight associated with the pixel indicates that the pixel is associated with a first illumination; when the weight indicates that the pixel is associated with the first illumination, associating the pixel with the first illumination; otherwise, not associating the pixel with the first illumination.

[0011] The method may include: providing a threshold indicating whether a weight indicates that the pixel is associated with a first illumination; and determining a binary mask for a masked input image that distinguishes weights in a first weight map that exceed the threshold from other weights in the first weight map.

[0012] An apparatus for predicting illumination maps, characterized in that the apparatus includes at least one processor and at least one memory, wherein the at least one memory is configured to store instructions executable by the at least one processor, wherein when the at least one processor executes the instructions, the instructions cause the apparatus to perform the method.

[0013] A computer program for predicting illumination patterns includes computer-readable instructions that, when executed by a computer, cause the computer to perform the method.

[0014] A data structure for predicting a lighting map includes at least one data field for: an input image containing a first lighting and a second lighting, a decomposition of the first lighting determined based on the input image, a decomposition of the second lighting determined based on the input image, and a lighting map determined based on these decompositions. Attached Figure Description

[0015] Further examples are derived from the following description and accompanying figures. In the figures: Figure 1 schematically depicts the first conduit used for predicting illumination patterns. Figure 2 schematically depicts the second pipe used for predicting illumination patterns. Figure 3 schematically depicts the apparatus used for predicting illumination patterns. Figure 4 schematically depicts the data structure used to predict illumination maps. Figure 5 schematically depicts the steps of a method for predicting illumination maps. Detailed Implementation

[0016] Figure 1 schematically depicts a first inference pipeline 100 for predicting a lighting map. The prediction of the lighting map is described using an exemplary image model as an example. This exemplary image model is given below: in Indicates position, I represents the RGB image, and r represents the surface magnetoresistance in the RGB image. It is a scaling factor that includes the intensity of shadows and lighting. Let represent the RGM illumination chromaticity vector, and This represents the dot product.

[0017] Scenes with multiple light sources can be expanded as follows: .

[0018] The corresponding white balance image can be described as: , in This refers to the chromaticity vector of a standard canonical light source. This is because it has been multiplied by a scaling term. , It can be represented as A white balance image can be represented as: in .

[0019] Where N represents the number of light sources. This represents element-wise division. Represents a weighted graph, and Indicates the light source The chromaticity vector.

[0020] According to the example, Indicates inclusion A stacked N-channel weighted graph.

[0021] According to the example, Indicates inclusion The stacked N-channel chroma.

[0022] The first inference pipeline 100 is configured to predict the decomposition of illumination, i.e., the weighted graph, from a single illumination source. and chromaticity vector .

[0023] After prediction, the final illumination map can be determined, for example... .

[0024] The first inference pipeline 100 includes, for example, a first encoder 102, a second encoder 104, and a decoder 106.

[0025] For example, the first encoder 102 is a SAM-based image encoder. In this example, the first encoder 102 can be adapted via an adapter 108. For example, the adapter 108 is configured as a low-rank adapter (LoRA).

[0026] For example, SAM is described in "Segment anything, published in 2023 by Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, AC; Lo, W.-Y. et al., ICCV, 4015–4026".

[0027] SAM utilizes a pre-trained visual transformer (ViT). An example of ViT is described in “An image is worth 16x16 words: Transformers for image recognition at scale”, published in 2020 by Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S. et al., arXivpreprint arXiv:2010.11929.

[0028] ViT can be adapted to handle high-resolution inputs for segmentation tasks. An example of a ViT adapted for segmentation tasks is described in “Exploring plain vision transformer backbones for object detection” by Li, Y.; Mao, H.; Girshick, R.; and He, K., 2022, ECCV, 280–296, Springer.

[0029] The multi-source estimation problem can be viewed as a task that includes pixel-wise illumination weight map segmentation and region-wise chromaticity vector prediction.

[0030] For example, the first encoder 102, specifically the image encoder of SAM, is configured to use LoRA to handle this cross-domain task.

[0031] For example, LoRA is described in “Lora: Low-rank adaptation of large language models”, published in 2021 by Hu, EJ; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W., arXiv preprint arXiv:2106.09685.

[0032] For example, the first encoder 102 is configured to process a continuous weight map for object segmentation.

[0033] For example, the first encoder 102 is an image encoder based on SAM. For example, the SAM image encoder is modified to process continuous weight maps instead of the binary mask used for object segmentation in SAM. For example, the encoder used for sparse cueing in the SAM image encoder is omitted in the first encoder 102.

[0034] The first encoder 102, the second encoder 104, and the decoder 106 can be arranged as part of the model 110.

[0035] Model 110 is configured to map the input image 112 to a weight map 114 associated with the input image 112. Model 110 is also configured to map the input image 112 to a chroma vector 116 associated with the input image 112.

[0036] For example, model 110 is configured to generate predictions of constant size, each prediction containing a weight map 114 and an associated chromaticity vector 116.

[0037] For example, model 110 is configured to use a first encoder 102 to determine a first embedding 118 based on the input image 112.

[0038] For example, the first encoder 102 is configured to encode the input image 112 into a first embedding 118. This means, for example, that the model 110 is configured to process the input image 112 using the first encoder 102 to extract the first embedding 118.

[0039] For example, model 110 is configured to use a second encoder 104 to determine a second embedding 120.

[0040] According to the example, model 110 is configured to process input image 112 using second encoder 104 to extract second embedding 120. Second embedding 120 is used to establish spatial correspondence.

[0041] According to the example, model 110 is configured to resize input image 112 to the resized input image and process the resized input image using a second encoder 104 to extract a second embedding 120. The embedded resized input image preserves the spatial context.

[0042] The second encoder 104 may include multiple convolutional layers. Model 110 may be configured to obtain a second embedding 120 from the outputs of the different convolutional layers of the second encoder 104. This incorporates spatial information.

[0043] Model 110 can be configured to add, concatenate, or multiply the first embedding 118 and the second embedding 120 to form input 122 for decoder 106.

[0044] Model 110 can be configured to use decoder 106 to map input 122 to decoder 106 to a weight map 114 associated with input image 112 and to chroma vector 116 associated with input image 112.

[0045] Model 110 can be configured to determine the illumination map based on the weight map 114 and the chromaticity vector 116. .

[0046] Decoder 106 can be learnable. For example, decoder 106 contains a stacked multilayer perceptron (MLP) network, instead of a single MLP used in SAM, for example. This makes it possible to process the decomposition of illumination more efficiently.

[0047] For example, decoder 106 is configured to process the decomposition of illumination in the dimension of chroma vector 116.

[0048] During training, the dimension of the chroma vector 116 can be used for mask matching.

[0049] The chromaticity vector 116 for the RGB image I has three dimensions.

[0050] For example, decoder 106 includes nearest neighbor interpolation, followed by two-dimensional (2D) convolution, instead of transposed convolutional layers used in SAM, for example.

[0051] For example, in “Deconvolution and Checkerboard Artifacts”, published in Distill in 2016 by Odena, A.; Dumoulin, V.; and Olah, C., two-dimensional convolution after nearest neighbor interpolation is described.

[0052] Performing 2D convolution after nearest neighbor interpolation helps reduce checkerboard artifacts in the output.

[0053] During training, padding illuminations can be generated and added. The number of padding illuminations can be determined for each scene. The actual amount of light present in the environment is used to determine this.

[0054] For example, a weighted graph To provide fill lighting. This means: there is no lightmap for fill lighting, for example... It generates expert influence.

[0055] Therefore, the color of the light source can change, making the chromaticity vector... Unrestrained.

[0056] Based on the example, an angle loss function is used to supervise the chroma vector. The angle loss function is, for example, as follows: in and This represents a vector that represents different objects.

[0057] An angular loss function can be used for training. The Hungarian algorithm can be used to establish a binary match between the baseline ground truth and the predicted decomposition. This technique ensures effective training regardless of the prediction order.

[0058] For example, the Hungarian algorithm is described in "The Hungarian method for the assignment problem, published by Kuhn, H. W in 1955, in Naval Research Logistics Quarterly, 2(1-2): 83–97".

[0059] A specific loss function can be defined as follows. Conduct training.

[0060] First, the set of matched indices that minimizes the L1 cost between the decomposition and the baseline truth. It can be calculated as: Where N is the predefined number of light sources. It is one of the combinations of N elements in the set 1, …, N, and where This represents the predicted chromaticity vector. Represents the true value of the chromaticity vector reference. This represents the weighted graph for prediction, and This represents the baseline truth value of the weighted graph.

[0061] This approach enables adaptive adjustment of the decomposition order, thereby facilitating efficient model training.

[0062] Based on permutation The rearranged benchmark truth weights are stacked again and represented as .

[0063] Complete loss Including supervised chromaticity vectors and supervised weight graph : in It is aimed at The weighted terms, and It is aimed at The weighting terms.

[0064] At once In this case, training might involve calculating the mean absolute error (MAE) pixel by pixel, that is, not calculating it across the entire vector of the weight map: Where M is the number of pixels in the weighted map. This represents the pixel value at position x in the stacked weight map.

[0065] At once In this regard, training can include calculating a weighted MAE by incorporating the average of the ground truth values ​​of the weighted graph as a weight term: in This represents the average value of the baseline true value of the weighted graph after adjustment.

[0066] Therefore, stronger illumination is beneficial to the term The contribution of artificial light is greater, while artificially added lighting has no impact on training.

[0067] The training may not directly supervise the final lighting map throughout the training process. This strategy ensures the integrity of the decomposition without additional supervision.

[0068] Figure 2 depicts the second inference pipeline 200.

[0069] The second inference pipeline 200 is configured to decompose the multi-light source scene in the input image 112 into multiple single-light source scenes 202. This decomposition enables existing single-light source methods to be used in multi-light source estimation applications without additional training.

[0070] The second inference pipeline 200 includes model 110. Model 110 is configured to decompose a multi-light source scene into multiple single-light source scenes 202.

[0071] According to the example, model 110 is configured to decompose a multi-light source scene into multiple, for example, N, single-light source scenes, and predict the illumination decomposition 202, i.e., the weight graph, from each of these single-light source scenes. and chromaticity vector The decomposition 202 includes, for example, weight maps 204-1, …, 204-N and masked input images 206-1, …, 206-N, wherein weight map 204-i is associated with masked input image 206-i.

[0072] Model 110 can be configured to predict the decomposition of illumination for single-light source scenes, i.e., weighted graphs. and chromaticity vector .

[0073] Model 110 can be configured to map the weights of each prediction. Normalization to, in particular, range Inside. Model 110 can be configured to weight each prediction in the following way. Normalize: .

[0074] Model 110 can be configured, in particular, by adjusting the normalized weight graph. A binary mask is generated by thresholding, particularly using a predefined threshold. This process isolates the pixels corresponding to each light source, thus allowing the scene to be easily broken down into individual light source components.

[0075] The second pipeline 200 may include a model 208 based on a single-source method. This single-source method-based model 208 can be configured to predict for a given source. Local illumination estimate The second pipe 200 can be configured to predict local illumination estimates using existing methods. The second pipe 200 is configurable for generating output 210, which includes... and A defined lighting pattern, for example .

[0076] Training can include training the model on the LSMI dataset.110 For example, the LSMI dataset is described in “Large scale multi-illuminant (lsmi) dataset for developing white balance algorithm under mixed illumination”, published by Kim, D.; Kim, J.; Nam, S.; Lee, D.; Lee, Y.; Kang, N.; Lee, H.-E.; Yoo, B.; Han, J.-J.; and Kim, SJ in 2021, ICCV, 2410–2419.

[0077] The trained model 110 is able to perform zero-shot generalization.

[0078] For zero-shot generalization, the method may include: training the model using the Sony split of the LSMI dataset

[110] .

[0079] This method may include testing the zero-shot generalization ability of Model 110 using Sony segmentation of the NUS-8 dataset. For example, the NUS-8 dataset is described in “Illuminant estimation for color constancy: why spatial-domain methods work and the role of the color distribution” by Cheng, D.; Prasad, DK; and Brown, MS, 2014, JOSA A, 31(5): 1049–1058.

[0080] Figure 3 schematically depicts a device 300 for predicting illumination patterns.

[0081] The device 300 includes at least one processor 302 and at least one memory 304.

[0082] The at least one memory 304 is configured to store instructions that can be executed by at least one processor 302.

[0083] When at least one processor 302 executes these instructions, these instructions cause the device 300 to perform a method for predicting illumination patterns.

[0084] A computer program for predicting illumination patterns may be provided. This computer program contains computer-readable instructions that, when executed by a computer, cause the computer to perform the method described above.

[0085] Figure 4 schematically depicts the data structure 400 used for predicting illumination maps.

[0086] The data structure 400 includes at least one data field 402 for input image 112, for decomposition of illumination determined based on input image 112, and for illumination map determined based on these decompositions.

[0087] Figure 5 schematically depicts the steps of a method for predicting illumination maps.

[0088] The method includes step 502.

[0089] Step 502 includes providing an input image 112.

[0090] The input image 112 includes pixels.

[0091] The method is described using input image 112 as an example, where the input image contains a first illumination i=1 and a second illumination i=2.

[0092] This method is not limited to two lighting conditions. It can handle multiple lighting conditions.

[0093] The method includes step 504.

[0094] Step 504 includes: determining the decomposition for the illumination based on the input image 112.

[0095] Step 504 includes, for example, determining the decomposition of the first illumination based on the input image 112, for example... This means that determining the decomposition includes, for example, determining the first chromaticity vector for the first illumination. And the first weighted graph .

[0096] Step 504 includes, for example, determining the decomposition of the second illumination based on the input image 112, for example... .

[0097] This means that determining the decomposition includes, for example, determining the second chromaticity vector for the second illumination. Second weighted graph .

[0098] For different lighting conditions, the method includes: determining a decomposition, such as a weighted graph, for different lighting conditions i. and chromaticity vector .

[0099] Weighted graph Each pixel of the input image 112 is associated with a weight.

[0100] Determining the decomposition of the first illumination may include determining a first embedding 118 based on the input image 112. For example, the first embedding 118 may be determined using a first encoder 102.

[0101] Determining the decomposition of the first illumination may include: determining the second embedding 120 based on the input image 112.

[0102] For example, the second encoder 104 is used to determine the second embedding 120.

[0103] The second embedding 120 can be determined by encoding the input image 112 or the resized input image into the second embedding 120.

[0104] The first chromaticity vector can be determined based on the embeddings 118 and 120. And the first weighted graph .

[0105] The first chromaticity vector can be determined using decoder 106. And the first weighted graph .

[0106] The second chromaticity vector can be determined based on the embeddings 118 and 120. Second weighted graph .

[0107] The second chromaticity vector can be determined using decoder 106. Second weighted graph .

[0108] The second embedding 120 can be determined using a second encoder 104 that includes convolutional layers. For example, determining the second embedding 120 includes obtaining the second embedding 120 from the outputs of different convolutional layers of the second encoder 104.

[0109] The method includes step 506.

[0110] Step 506 includes determining the illumination map based on the decomposition.

[0111] For these two decompositions, lighting diagrams, for example It is determined. This means that the lighting map is, for example, based on the weight map ( ) and chromaticity vector ( It has been determined.

[0112] For multiple illuminations, such as N illuminations, the method may include: determining a illumination map based on decompositions for different illuminations, for example... .

[0113] The method may include step 508.

[0114] Step 508 includes training the decoder 106.

[0115] Step 508 may include: training decoder 106 to determine chroma vectors using decoder 106, for example... And weighted graphs, for example .

[0116] This method may include providing a chromaticity vector, for example The baseline truth value, and provides for weighted graphs, such as The baseline truth value.

[0117] Decoder 106 can be based on the loss function Being trained.

[0118] loss function Includes factors that depend on the chroma vector, such as item , and the baseline truth value for the chromaticity vector.

[0119] loss function Includes weighted graphs, for example item , and the baseline truth for the weighted graph.

[0120] The input image 112 may be part of the lighting used to train the decoder 106. The decoder 106 may be trained based on the lighting containing the input image 112. The lighting used to train the decoder 106 may include additional padding lighting.

[0121] The method may include: decomposing the input image 112 into a first single-light source scene for a first illumination and a second single-light source scene for a second illumination, and predicting the illumination decomposition based on these single-light source scenes respectively. This means predicting weight maps based on the single-light source scenes respectively. and chromaticity vector .

[0122] The method may include: based on the weight graph Determine whether the weights associated with a pixel indicate that the pixel is associated with illumination i.

[0123] The method may include: associating the pixel with illumination i when the weight associated with the pixel indicates that the pixel is associated with illumination i; otherwise, not associating the pixel with illumination i.

[0124] The method may include providing a threshold that indicates whether the weight indicates that the pixel is associated with illumination i.

[0125] The method may include: determining a binary mask for a masked input image, which will weight the image... Weights exceeding the threshold and the first weight graph The other weights are distinguished.

[0126] For example, the method includes determining weight maps 204-1, ..., 204-N and binary masks for masked input images 206-1, ..., 206-N.

Claims

1. A method for predicting illumination maps, characterized in that, The method includes: providing (502) an input image (112) comprising a first illumination and a second illumination; determining (504) a decomposition of the first illumination based on the input image (112); determining (504) a decomposition of the second illumination based on the input image; and determining (506) the illumination map based on the decomposition.

2. The method according to claim 1, characterized in that, Determining (504) the decomposition includes: determining a first chromaticity vector and a first weight map for the first illumination based on the input image (112); and determining a second chromaticity vector and a second weight map for the second illumination based on the input image (112); wherein, determining (506) the illumination map includes: determining the illumination map based on the weight map and the chromaticity vector.

3. The method according to claim 2, characterized in that, Determining (504) the decomposition of the first illumination includes: in particular using a first encoder (102) configured to encode the input image (112) into a first embedding (118), determining the first embedding (118) based on the input image (112); in particular using a second encoder (104) configured to encode the input image (112) or a resized input image into a second embedding (120), determining the second embedding (120) based on the input image (112); and in particular using a decoder (106) to determine a first chroma vector and a first weight map based on the embeddings (118, 120), wherein the decoder is configured to decode the input of the decoder (106) depending on the embedding into the first chroma vector and the first weight map.

4. The method according to claim 3, characterized in that, Determining (504) the decomposition of the second illumination includes, in particular, using a decoder (106) to determine the second chromaticity vector and the second weight map based on the embedding (118, 120), wherein the decoder (106) is configured to decode the input of the decoder (106) into the second chromaticity vector and the second weight map.

5. The method according to any one of claims 2 to 4, characterized in that, The second embedding (120) is determined using the second encoder (104), wherein the second encoder (104) includes convolutional layers, and determining the second embedding (120) includes obtaining the second embedding (120) from the outputs of different convolutional layers of the second encoder (104).

6. The method according to any one of claims 2 to 5, characterized in that, The decoder (106) is used to determine the chroma vector and the weight map, wherein the method includes: providing a ground truth value for the chroma vector and a ground truth value for the weight map, and training (508) the decoder (106) according to a loss function, wherein the loss function includes: a first term depending on the chroma vector and the ground truth value for the chroma vector, and a second term depending on the weight map and the ground truth value for the weight map.

7. The method according to any one of the preceding claims, characterized in that, The method includes: decomposing the input image (112) into a first single-light source scene for the first illumination and a second single-light source scene for the second illumination, and predicting the illumination decomposition, such as the weight map and the chromaticity vector, from the single-light source scene respectively.

8. The method according to any one of the preceding claims, characterized in that, The input image (112) includes pixels, wherein the first weight map associates each pixel of the input image (112) with a weight, wherein the method includes: determining whether the weight associated with the pixel indicates that the pixel is associated with the first illumination, associating the pixel with the first illumination when the weight indicates that the pixel is associated with the first illumination, and otherwise not associating the pixel with the first illumination.

9. The method according to claim 8, characterized in that, The method includes providing a threshold indicating whether a weight indicates that the pixel is associated with the first illumination, and determining a binary mask for a masked input image (208-1) that distinguishes weights in the first weight map that exceed the threshold from other weights in the first weight map.

10. An apparatus (300) for predicting illumination patterns, characterized in that, The apparatus (300) includes at least one processor (302) and at least one memory (304), wherein the at least one memory (304) is configured to store instructions executable by the at least one processor (302), wherein when the at least one processor (302) executes the instructions, the instructions cause the apparatus (300) to perform the method according to any one of claims 1 to 9.

11. A computer program for predicting illumination maps, characterized in that, The computer program contains computer-readable instructions that, when executed by the computer, cause the computer to perform the method of any one of claims 1 to 9.

12. A data structure (400) for predicting illumination maps, characterized in that, The data structure includes at least one data field (402) for: an input image (112) containing a first illumination and a second illumination, a decomposition of the first illumination determined according to the input image (112), a decomposition of the second illumination determined according to the input image (112), and an illumination map determined according to the decomposition.

Images