Systems and methods for image decomposition

Abstract

Systems and methods for performing image composition to determine an estimated shading and estimated albedo. An input image is processed by a first neural network to generate at least one shading image by minimizing loss based on minimizing differences between a relative shading order of the pixels in the at least one first shading image and a relative shading order of pixels in the shading of the input image. Shading can be estimated using inverse shading values. The at least one shading image can comprise lower and high resolution shading images to be processed by a second neural network to generate a second shading image such that an albedo image can also be generated. Multi-illumination photographs can be used to generate real world training data. A diffuse albedo image, diffuse shading image, and at least one residual image can be generated using a third, fourth, and fifth neural network.

Description

SYSTEMS AND METHODS FOR IMAGE DECOMPOSITIONCCROSS-REFERENCE TO OTHER APPLICATIONS

[0001] This application claims priority to United States Provisional Patent Application No. 63 / 515,684, filed on July 26, 2023, the entire contents of which is incorporated herein by reference for all purposes.TECHNICAL FIELD

[0002] The present disclosure relates to image decomposition and in particular to the decomposition of photographs into at least an albedo image and a shading image.BACKGROUND

[0003] Intrinsic decomposition is a fundamental mid-level vision problem that plays a crucial role in various inverse rendering and computational photography pipelines. Generating highly accurate intrinsic decompositions is an inherently underconstrained task that requires precisely estimating continuous-valued shading and albedo.

[0004] Since intrinsic decomposition separates the illumination-invariant scene properties from the illumination-dependent lighting effects, it is a critical component fora wide range of computational photography pipelines such as relighting, recoloring, and compositing. Realistic image editing through intrinsic decomposition requires accurate decompositions at high resolutions for in-the-wild photographs. Prior data- driven methods have not been able to live up to these requirements and as a result, intrinsic computational photography methods have not yet been widely adopted by digital artists.

[0005] Accordingly, systems and methods that enable decomposition of real world images remain highly desirable.SUMMARY

[0006] In accordance with one aspect of the present disclosure, a method of image decomposition is disclosed, the method comprising: processing an input imagewith a first neural network trained to determine shading representing an illumination effect on content depicted in the input image; and generating, with the first neural network, at least one first shading image from the input image, the at least one first shading image depicting the shading of the input image, the input image comprising a plurality of pixels represented using color channel values and the at least one first shading image comprising a plurality of pixels represented using numerical shading values; and the first neural network trained to determine the shading by minimizing loss based on minimizing differences between a relative shading order of the pixels in the at least one first shading image and a relative shading order of pixels in the shading of the input image.

[0007] In accordance with another aspect of the present disclosure, a method of image decomposition is disclosed, the method comprising: processing an input image with a first neural network trained to determine shading representing an illumination effect on content depicted in the input image; and generating, with the first neural network, at least one first shading image from the input image, the at least one first shading image depicting the shading of the input image, the input image comprising a plurality of pixels represented using color channel values and the at least one first shading image comprising a plurality of pixels represented using numerical shading values; and the first neural network is trained to derive the shading values by mapping the shading values to a first domain of pre-defined range.

[0008] In some aspects, the first neural network is trained to determine the shading by minimizing loss based on minimizing differences between a relative shading order of the pixels in the at least one first shading image and a relative shading order of pixels in the shading of the input image.

[0009] In some aspects, the shading values are derived by the first neural network using mapped shading values, each mapped shading value being mapped from a shading value associated with each pixel that is representative of the illumination effect.

[0100] In some aspects, the shading values are mapped to a first domain of pre-defined range.

[0101] In some aspects, the mapped shading values are calculated as D=1 / (S+1), where S is a shading value and D is a corresponding mapped shading value.

[0010] In some aspects, the first domain has a range of 0 to 1 .

[0011] In some aspects, the at least one first shading image comprises: a low resolution shading image with a resolution that is within a receptive field of the first neural network, the low resolution shading image providing global context of the shading, and a high resolution shading image with a resolution that that exceeds the receptive field of the first neural network, the high resolution shading image providing local context of the shading.

[0012] In some aspects, the method further comprises: processing the input image, the low resolution shading image, and the high resolution shading image with a second neural network trained to further determine the shading of the input image; generating, with the second neural network, a second shading image from the input image, the low resolution shading image, and the high resolution shading image, the second shading image being a greyscale shading image comprising a plurality of pixels represented using the numerical shading values for depicting the shading of the input image; and generating a first albedo image from the input image and the second shading image, the first albedo image comprising a plurality of pixels represented using the color channel values for depicting reflectance of the content in the input image.

[0013] In some aspects, the first albedo image represents Lambertian reflectance and is generated based on a relationship that the product of the first albedo image and the second shading image is the input image.

[0100] In some aspects, the input image is a 3-channel RGB image comprising a photograph or photorealistic image; each of the at least one first shading image and the second shading image is a single-channel greyscale image; and the first albedo image is a 3-channel RGB image.

[0014] In some aspects, the method further comprises: obtaining a first set of training data comprising synthetic images and at least one training shading image corresponding to each synthetic image comprising: a high resolution training shading image and a low resolution training shading image; training the first neural network to generate the at least one first shading image corresponding to each synthetic image based on the first set of training data; obtaining a second set of training data comprising synthetic images, high resolution training shading images corresponding to the synthetic images, low resolution training shading images corresponding to the synthetic images, and training shading images corresponding to the synthetic images, the low resolution training shading images, and the high resolution shading images; and training the second neural network to generate the second shading image corresponding to each synthetic image based on the second set of training data.

[0100] In some aspects, the method further comprises: obtaining a third set of training data comprising photographs and at least one training shading image corresponding to each photograph comprising: a high resolution training shading image and a low resolution training shading image; training the first neural network to generate the at least one first shading image corresponding to each photograph based on the third set of training data; obtaining a fourth set of training data comprising photographs, high resolution training shading images corresponding to the photographs, low resolution training shading images corresponding to the photographs, and training shading images corresponding to the photographs, the low resolution training shading images, and the high resolution shading images; and training the second neural network to generate the second shading image corresponding to each photograph based on the fourth set of training data.

[0101] In some aspects, the method further comprises: generating training data comprising photographs by: obtaining a plurality of photographs for a scene under different lighting conditions; generating a plurality of albedo images corresponding to the plurality of photographs with the first neural network and the second neural network; generating a predicted albedo image by performing statistical analysis on the plurality of albedo images; generating a plurality of predicted shading images by dividing each of the plurality of photographs by the predicted albedo image; andcompiling the plurality of photographs and the plurality of predicted shading images into the training data.

[0015] In some aspects, the predicted albedo image is generated by taking the per pixel median of the plurality of albedo images.

[0016] In some aspects, training of the first neural network comprises use of a loss function defining a relationship between ground truth and a monotonically increasing or monotonically decreasing function comprising shading generated by the first neural network and in which loss determined by the loss function is indicative of the differences between the relative shading order of the pixels of the at least one first shading image and the relative shading order of the shading of the input image.

[0017] In some aspects, the training of the first neural network further comprises use of a multi-gradient loss function to evaluate estimated shading and the differences between the relative shading order of the pixels of the at least one first shading image and the relative shading order of the shading of the input image.

[0018] In some aspects, training of the second neural network comprises use of the low resolution training shading images for scale.

[0019] In some aspects, the method further comprises: processing the input image, the second shading image and the first albedo image with a third neural network trained to determine a chromaticity adjusted shading representing the illumination effect on the content depicted in the input image; and generating, with the third neural network, a color component image by deriving chromaticity values of pixels in the second shading image, the color component image depicting chromaticity contributions to the shading of the input image; generating a third shading image by augmenting the second shading image with the color component image, the third shading image being a shading image depicting the chromaticity dependent shading of the input image; and generating a second albedo image from the input image and the third shading image, the second albedo image being a chromaticity adjusted albedo image depicting the reflectance of the input image.

[0020] In some aspects, the second albedo image is generated based on a relationship where the product of the second albedo image and the third shading image is the input image.

[0021] In some aspects, the chromaticity values of the pixels are defined as channel variables U and V where U = S^ / S^ and V = S^ / S^.

[0100] In some aspects, the chromaticity values are mapped to a second domain of pre-defined range.

[0022] In some aspects, the second domain has a range of 0 to 1 .

[0023] In some aspects, the method further comprises: processing the input image, the third shading image, and the second albedo image with a fourth neural network trained to determine an adjusted albedo for representing diffuse reflectance of the content depicted in the input image; generating, with the fourth neural network, a third albedo image from the input image, the third shading image, and the second albedo image, the third albedo image being an albedo image depicting the diffuse reflectance of the input image; and generating a fourth shading image from the input image and the third albedo image, the fourth shading image being a shading image depicting a diffuse illumination adjusted shading of the input image.

[0024] In some aspects, the fourth shading image is generated based on a relationship where the product of the third albedo image and the fourth shading image is the input image.

[0025] In some aspects, the method further comprises: processing the input image, the fourth shading image, and the third albedo image with a fifth neural network trained to determine a diffuse shading representing a diffuse illumination effect on the content depicted in the input image; generating, with the fifth neural network, a fifth shading image from the input image, the fourth shading image, and the third albedo image, the fifth shading image being a shading image depicting the diffuse shading of the input image; and generating at least one residual image from the input image, the fourth shading image, and the third albedo image, the at least one residual imagecomprising a plurality of pixels represented using color channel values for depicting an effect of non-diffuse illumination on the content depicted in the input image.

[0026] In some aspects, the fifth shading image is generated based on a relationship where the product of the third albedo image and the fifth shading image is the input image.

[0027] In some aspects, the fifth neural network is trained to derive shading values of the fifth shading image by mapping the shading values of the fifth shading image to a third domain of pre-defined range.

[0028] In some aspects, the at least one residual image is generated based on a relationship that an addition between the at least one residual image and a product of the fifth shading image and the third albedo image is the input image.

[0029] In some aspects, the at least one residual image comprises: a positive residual image comprising pixels of positive values corresponding to the non-diffuse illumination, and a negative residual image comprising pixels of negative values corresponding to overexposure.

[0030] In some aspects, the method further comprises: obtaining a fifth set of train data comprising synthetic images, training shading images corresponding to the synthetic images, training albedo images corresponding to the synthetic images, and training color component images corresponding to the synthetic images; training the third neural network to generate the color component image corresponding to each synthetic image based on the fifth set of training data; obtaining a sixth set of training data comprising photographs, training shading images corresponding to the photographs, training albedo images corresponding to the photographs, and training color component images corresponding to the photographs; and training the third neural network to generate the color component image corresponding to each photograph based on the sixth set of training data.

[0100] In some aspects, the method further comprises: obtaining a seventh set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the syntheticimages; training the fourth neural network to generate the third albedo image corresponding to each synthetic image based on the seventh set of training data; obtaining an eighth set of training data comprising photographs, training shading images corresponding to the photographs, and training albedo images corresponding to the photographs; and training the fourth neural network to generate the third albedo image corresponding to each photograph based on the eighth set of training data.

[0101] In some aspects, the method further comprises: obtaining a ninth set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the synthetic images; and training the fifth neural network to generate the fifth shading image corresponding to each synthetic image based on the ninth set of training data.

[0031] In some aspects, each of the third shading image, the fourth shading image, the fifth shading image, the second albedo image, the third albedo image, and the at least one residual image is a 3-channel RGB image.

[0032] In accordance with another aspect of the present disclosure, a method of image decomposition is disclosed, the method comprising: processing an input image, a first shading image and a first albedo image with a chroma neural network trained to determine a chromaticity adjusted shading representing an illumination effect on content depicted in the input image, each input image comprising a diffuse shading component corresponding to shading that depicts an illumination effect on the content, a diffuse albedo component corresponding to albedo that depicts reflectance on the content, and a residual component corresponding to a non-diffuse illumination effect on the content; and generating, with the chroma neural network, a color component image by deriving chromaticity values of pixels in the shading image, the color component image depicting chromaticity contributions to the shading of the input image; generating a second shading image by augmenting the first shading image with the color component image, the second shading image being a shading image depicting the chromaticity dependent shading of the input image; generating a second albedo image from the input image and the second shading image, the second albedo image being a chromaticity adjusted albedo image depicting the reflectance of the content; processing the input image, the second shading image, and the secondalbedo image with an albedo neural network trained to determine the diffuse albedo component; generating, with the albedo neural network, a third albedo image from the input image, the second shading image, and the second albedo image, the third albedo image being an albedo image depicting the diffuse albedo component; generating a third shading image from the input image and the third albedo image, the third shading image being a shading image depicting a diffuse illumination adjusted shading of the input image; processing the input image, the third shading image, and the third albedo image with a diffuse shading neural network trained to determine the diffuse shading component; generating, with the diffuse shading neural network, a fourth shading image from the input image, the third shading image, and the third albedo image, the fourth shading image being a shading image depicting the diffuse shading component; and generating at least one residual image from the input image, the third shading image, and the third albedo image, the at least one residual image comprising a plurality of pixels represented using color channel values for depicting an effect of non-diffuse illumination on the content depicted in the input image.

[0033] In some aspects, the second albedo image is generated based on a relationship where the product of the second albedo image and the second shading image is the input image.

[0034] In some aspects, the chromaticity values of the pixels are defined as channel variables U and V where U = Sc / S and V = S^ / S^.

[0100] In some aspects, the chromaticity values are mapped to a first domain of pre-defined range.

[0035] In some aspects, the first domain has a range of 0 to 1 .

[0036] In some aspects, the third shading image is generated based on a relationship where the product of the third albedo image and the third shading image is the input image.

[0037] In some aspects, the fourth shading image is generated based on a relationship where the product of the third albedo image and the fourth shading image is the input image.

[0038] In some aspects, the diffuse shading neural network is trained to derive shading values of the fourth shading image using mapped shading values, each mapped shading value being mapped from a shading value associated with each pixel that is representative of the illumination effect.

[0039] In some aspects, the diffuse shading neural network is trained to derive shading values of the fourth shading image by mapping the shading values of the fourth shading image to a second domain of pre-defined range, the shading values representative of the illumination effect.

[0040] In some aspects, the at least one residual image is generated based on a relationship that an addition between the at least one residual image and a product of the fourth shading image and the third albedo image is the input image.

[0041] In some aspects, the at least one residual image comprises: a positive residual image comprising pixels of positive values corresponding to the non-diffuse illumination, and a negative residual image comprising pixels of negative values corresponding to overexposure.

[0100] In some aspects, the method further comprises: obtaining a first set of train data comprising synthetic images, training shading images corresponding to the synthetic images, training albedo images corresponding to the synthetic images, and training color component images corresponding to the synthetic images; training the chroma neural network to generate the color component image corresponding to each synthetic image based on the first set of training data; obtaining a second set of training data comprising photographs, training shading images corresponding to the photographs, training albedo images corresponding to the photographs, and training color component images corresponding to the photographs; and training the chroma neural network to generate the color component image corresponding to each photograph based on the second set of training data.

[0101] In some aspects, the method further comprises: obtaining a third set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the synthetic images; training the albedo neural network to generate the third albedo image corresponding to each synthetic image based on the third set of training data; obtaining a fourth set of training data comprising photographs, training shading images corresponding to the photographs, and training albedo images corresponding to the photographs; and training the albedo neural network to generate the third albedo image corresponding to each photograph based on the fourth set of training data.

[0102] In some aspects, the method further comprises: obtaining a fifth set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the synthetic images; and training the diffuse shading neural network to generate the fourth shading image corresponding to each synthetic image based on the fifth set of training data.

[0103] In some aspects, the method further comprises: generating network training data comprising photographs by: obtaining a plurality of photographs for a scene under different lighting conditions; generating a plurality of albedo images corresponding to the plurality of photographs; generating a predicted albedo image by performing statistical analysis on the plurality of albedo images; generating a plurality of predicted shading images by dividing each of the plurality of photographs by the predicted albedo image; and compiling the plurality of photographs, the predicted albedo image, and the plurality of predicted shading images into the network training data.

[0042] In some aspects, each of the second shading image, the third shading image, the fourth shading image, the second albedo image, the third albedo image, and the at least one residual image is a 3-channel RGB image.

[0043] In accordance with another aspect of the present disclosure: a method for training one or more neural networks for performing image decomposition is disclosed, the method comprising: obtaining a first set of training data comprisingsynthetic images and at least one training shading image corresponding to each synthetic image, the at least one training shading image comprising: a high resolution training shading image and a low resolution training shading image; training a first neural network to generate at least one first shading image corresponding to each synthetic image based on the first set of training data, the at least one first shading image depicting shading representing an illumination effect and comprises a plurality of pixels represented using numerical shading values; training the first neural network to determine the shading by minimizing loss based on minimizing differences between a relative shading order of the pixels in the at least one first shading image and a relative shading order of pixels in the shading of an input image; and training the first neural network to generate the at least one first shading image from the input image.

[0044] In accordance with another aspect of the present disclosure, a method for training one or more neural networks for performing image decomposition is disclosed, the method comprising: obtaining a first set of training data comprising synthetic images and at least one training shading image corresponding to each synthetic image, the at least one training shading image comprising: a high resolution training shading image and a low resolution training shading image; training a first neural network to generate at least one first shading image corresponding to each synthetic image based on the first set of training data, the at least one first shading image depicting shading representing an illumination effect and comprises a plurality of pixels represented using numerical shading values; training the first neural network to derive the shading values by mapping the shading values to a first domain of predefined range.

[0100] In some aspects, the method further comprises: training the first neural network to determine the shading by minimizing loss based on minimizing differences between a relative shading order of the pixels in the at least one first shading image and a relative shading order of pixels in the shading of an input image.

[0101] In some aspects, the method further comprises: training the first neural network to derive the shading values using mapped shading values, each mapped shading value being mapped from a shading value associated with each pixel that is representative of the illumination effect.

[0045] In some aspects, the shading values are mapped to a first domain of pre-defined range.

[0046] In some aspects, the first domain has a range of 0 to 1 .

[0047] In some aspects, the at least one first shading image comprises: a low resolution shading image with a resolution that is within a receptive field of the first neural network, the low resolution shading image providing global context of the shading, and a high resolution shading image with a resolution that that exceeds the receptive field of the first neural network, the high resolution shading image providing local context of the shading.

[0100] In some aspects, the method further comprises: obtaining a second set of training data comprising synthetic images, high resolution training shading images corresponding to the synthetic images, low resolution training shading images corresponding to the synthetic images, and training shading images corresponding to the synthetic images, the low resolution training shading images, and the high resolution shading images; training a second neural network to generate a second shading image corresponding to each synthetic image based on the second set of training data, the second shading image being a greyscale shading image comprising a plurality of pixels represented using the numerical shading values for depicting the shading; and training the second neural network to generate the second shading image from the input image and the at least one first shading image.

[0101] In some aspects, the method further comprises: obtaining a third set of training data comprising photographs and at least one training shading image corresponding to each photograph comprising: a high resolution training shading image and a low resolution training shading image; training the first neural network to generate the at least one first shading image corresponding to each photograph based on the third set of training data; obtaining a fourth set of training data comprising photographs, high resolution training shading images corresponding to the photographs, low resolution training shading images corresponding to the photographs, and training shading images corresponding to the photographs, the low resolution training shading images, and the high resolution shading images; andtraining the second neural network to generate the second shading image corresponding to each photograph based on the fourth set of training data.

[0102] In some aspects, the method further comprises: generating training data comprising photographs by: obtaining a plurality of photographs for a scene under different lighting conditions; generating a plurality of albedo images corresponding to the plurality of photographs with the first neural network and the second neural network; generating a predicted albedo image by performing statistical analysis on the plurality of albedo images; generating a plurality of predicted shading images by dividing each of the plurality of photographs by the predicted albedo image; and compiling the plurality of photographs and the plurality of predicted shading images into the training data.

[0048] In some aspects, the predicted albedo image is generated by taking the per-pixel median of the plurality of albedo images.

[0100] In some aspects, the method further comprises: training the first neural network using a loss function defining a relationship between ground truth and a monotonically increasing or monotonically decreasing function comprising shading generated by the first neural network and in which loss determined by the loss function is indicative of the differences between the relative shading order of the pixels of the at least one first shading image and the relative shading order of the shading of the input image.

[0101] In some aspects, the method further comprises: training the first neural network using a multi-gradient loss function to evaluate estimated shading and the differences between the relative shading order of the pixels of the at least one first shading image and the relative shading order of the shading of the input image.

[0049] In some aspects, training data for the second neural network further comprises training albedo images depicting reflectance.

[0100] In some aspects, the method further comprises: training the second neural network using the low resolution training shading images and the training albedo images as a reference scale.

[0101] In some aspects, the method further comprises: generating a first albedo image for each second shading image based on a relationship where the product of the first albedo image and the second shading image is the input image, the first albedo image comprising a plurality of pixels represented using the color channel values for depicting reflectance; and training the second neural network using a mean-squared error loss function and a multi-scale gradient loss function for evaluating estimated shading and estimated albedo.

[0102] In some aspects, the method further comprises: obtaining a fifth set of train data comprising synthetic images, training shading images corresponding to the synthetic images, training albedo images corresponding to the synthetic images, and training color component images corresponding to the synthetic images; training a third neural network to generate a color component image corresponding to each synthetic image based on the fifth set of training data, the color component image depicting chromaticity contributions to the shading; obtaining a sixth set of training data comprising photographs, training shading images corresponding to the photographs, training albedo images corresponding to the photographs, and training color component images corresponding to the photographs; training the third neural network to generate the color component image corresponding to each photograph based on the sixth set of training data; and training the third neural network to generate the color component image by determining a chromaticity dependent shading representing the illumination effect by deriving chromaticity values of the pixels in the input image.

[0103] In some aspects, the method further comprises: training the third neural network to derive the chromaticity values of the pixels by defining the chromaticity values as channel variables U and V where U = Sc / S and V = S^ / S^ , and the chromaticity values are mapped to a second domain of pre-defined range.

[0104] In some aspects, the color component image is for generating a third shading image, the third shading image being a shading image depicting a chromaticity adjusted shading of the input image and generated by augmenting the second shading image with the color component image; and the third shading image is for generating a second albedo image, the second albedo image being achromaticity adjusted albedo image depicting the reflectance and generated from the input image and the third shading image.

[0105] In some aspects, the method further comprises: obtaining a seventh set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the synthetic images; training a fourth neural network to generate a third albedo image corresponding to each synthetic image based on the seventh set of training data, the third albedo image being an albedo image depicting a diffuse reflectance; obtaining an eighth set of training data comprising photographs, training shading images corresponding to the photographs, and training albedo images corresponding to the photographs; training the fourth neural network to generate the third albedo image corresponding to each photograph based on the eighth set of training data; and training the fourth neural network to generate the third albedo image from the input image, the third shading image, and the second albedo image by determining an adjusted albedo for representing the diffuse reflectance.

[0050] In some aspects, the third albedo image is for generating a fourth shading image, the fourth shading image being a shading image depicting a diffuse illumination adjusted shading and generated from the input image and the third albedo image.

[0100] In some aspects, the method further comprises: obtaining a ninth set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the synthetic images; training a fifth neural network to generate a fifth shading image corresponding to each synthetic image based on the ninth set of training data, the fifth shading image being a shading image depicting a diffuse shading; and training the fifth neural network to generate the fifth shading image from the input image, the fourth shading image, and the third albedo image by determining the diffuse shading representing a diffuse illumination effect.

[0051] In some aspects, the fifth shading image is for generating at least one residual image comprising a plurality of pixels represented using color channel valuesfor depicting an effect of non-diffuse illumination, the at least one residual image generated from the input image, the fourth shading image, and the third albedo image.

[0100] In some aspects, the method further comprises: training the fifth neural network to derive shading values of the fifth shading image by mapping the shading values of the fifth shading image to a third domain of pre-defined range.

[0052] In some aspects, the third domain has a range of 0 to 1 .

[0053] In accordance with another aspect of the present disclosure, a method for training one or more neural networks for performing image decomposition, is disclosed, the method comprising: obtaining a first set of train data comprising training input images, training shading images corresponding to the training input images, training albedo images corresponding to the training input images, and training color component images corresponding to the training input images; training a chroma neural network to generate a color component image corresponding to each training input image based on the first set of training data, the color component image depicting chromaticity contributions to the shading; training the chroma neural network to generate the color component image by determining a chromaticity adjusted shading representing an illumination effect on content depicted in an input image, each input image comprising a diffuse shading component corresponding to shading that depicts an illumination effect on content depicted in the input image, a diffuse albedo component corresponding to albedo that depicts reflectance on the content, and a residual component corresponding to a non-diffuse illumination effect on the content; obtaining a second set of training data comprising training input images, training shading images corresponding to the training input images, and training albedo images corresponding to the training input images; training an albedo neural network to generate a third albedo image corresponding to each training input image based on the second set of training data, the third albedo image being an albedo image depicting the diffuse albedo component; training the albedo neural network to generate the third albedo image by determining the diffuse albedo component; obtaining a third set of training data comprising training input images, training shading images corresponding to the training input images, and training albedo images corresponding to the training input images; training a diffuse shadingneural network to generate a fourth shading image corresponding to each training input image based on the third set of training data, the fourth shading image being a shading image depicting the diffuse shading component; and training the fifth neural network to generate the fourth shading image by determining the diffuse shading component.

[0054] In some aspects, the method further comprises: obtaining a fourth set of training data comprising photographs, training shading images corresponding to the photographs, training albedo images corresponding to the photographs, and training color component images corresponding to the photographs; training the chroma neural network to generate the color component image corresponding to each photograph based on the fourth set of training data; obtaining a fifth set of training data comprising photographs, training shading images corresponding to the photographs, and training albedo images corresponding to the photographs; and training the albedo neural network to generate the third albedo image corresponding to each photograph based on the fifth set of training data.

[0100] In some aspects, the method further comprises: training the chroma neural network to derive the chromaticity values of the pixels by defining the chromaticity values as channel variables U and V where U = Sc / S and V = S / S , and the chromaticity values are mapped to a first domain of pre-defined range.

[0055] In some aspects, the first domain has a range of 0 to 1 .

[0100] In some aspects, the color component image is for generating a second shading image, the second shading image being a shading image depicting the chromaticity dependent shading of the input image and generated by augmenting the first shading image with the color component image; the second shading image is for generating a second albedo image, the second albedo image being a chromaticity adjusted albedo image depicting the reflectance of the content and generated from the input image and the second shading image; and the third albedo image is for generating a third shading image, the third shading image being a shading image depicting a diffuse illumination adjusted shading of the input image and generated from the input image and the third albedo image.

[0056] In some aspects, the method further comprises: training the albedo neural network to generate the third albedo image from the input image, the third shading image, and the second albedo image.

[0100] In some aspects, the method further comprises: training the diffuse shading neural network to generate the fourth shading image from the input image, the third shading image, and the third albedo image.

[0057] In some aspects, the fourth shading image is for generating at least one residual image comprising a plurality of pixels represented using color channel values for depicting an effect of non-diffuse illumination on the content depicted in the input image, the at least one residual image generated from the input image, the fourth shading image, and the third albedo image.

[0100] In some aspects, the method further comprises: training the diffuse shading neural network to derive shading values of the fourth shading image using mapped shading values, each mapped shading value being mapped from a shading value associated with each pixel that is representative of the illumination effect.

[0101] In some aspects, the method further comprises: training the diffuse shading neural network to derive shading values of the fourth shading image representative of the illumination effect by mapping the shading values to a second domain of pre-defined range.

[0102] In some aspects, the method further comprises: generating network training data comprising photographs by: obtaining a plurality of photographs for a scene under different lighting conditions; generating a plurality of albedo images corresponding to the plurality of photographs; generating a predicted albedo image by performing statistical analysis on the plurality of albedo images; generating a plurality of predicted shading images by dividing each of the plurality of photographs by the predicted albedo image; and compiling the plurality of photographs, the predicted albedo image, and the plurality of predicted shading images into the network training data.

[0058] In some aspects, the statistical analysis is taking the per-pixel median of the plurality of albedo images.

[0059] In accordance with another aspect of the present disclosure, a method of generating training images comprising photographs for training neural networks to perform image decomposition is disclosed, the method comprising: obtaining a plurality of photographs for a scene under different lighting conditions, each photograph comprising a component corresponding to shading that depicts an illumination effect on the scene and a component corresponding to albedo that depicts reflectance on the scene; generating a plurality of albedo images corresponding to the plurality of photographs, each albedo image comprising pixels depicting an estimated albedo of a corresponding photograph; generating a predicted albedo image by performing statistical analysis on the plurality of albedo images; generating a plurality of predicted shading images by dividing each of the plurality of photographs by the predicted albedo image, each predicted shading image comprising pixels depicting an estimated shading of a corresponding photograph; and compiling the plurality of photographs, the predicted albedo image, and the plurality of predicted shading images into network training data.

[0060] In some aspects, the predicted shading images are for use as ground truth shading and the predicted albedo image is for use as ground truth albedo.

[0061] In some aspects, the plurality of albedo images are generated by one or more neural networks trained to perform image decomposition.

[0062] In some aspects, the plurality of albedo images are generated by a first neural network and a second neural network by: processing the plurality of photographs with the first neural network trained to determine shading representing an illumination effect on content depicted in an input image; and generating, with the first neural network from the plurality of photographs, a plurality of first shading images depicting the shading comprising a plurality of low resolution shading images with a resolution that is within a receptive field of the first neural network and a plurality of high resolution shading images with a resolution that that exceeds the receptive field of the first neural network, the shading represented using numerical shading values;and the first neural network is trained to determine the shading by minimizing loss based on minimizing differences between a relative shading order of the pixels in the each of the shading images and a relative shading order of pixels in the shading of the input image, processing the plurality of photographs and the plurality of first shading images with a second neural network trained to further determine the shading of the input image; generating, with the second neural network, a plurality of second shading images depicting the shading from the plurality of photographs and the plurality of first shading images; and generating the plurality of albedo images from the plurality of photographs and the plurality of second shading images by dividing each photograph with a corresponding second shading image.

[0063] In some aspects, training of the first neural network comprises use of a loss function defining a relationship between ground truth and a monotonically increasing or monotonically decreasing function comprising shading generated by the first neural network and in which loss determined by the loss function is indicative of the differences between the relative shading order of the pixels of the at least one first shading image and the relative shading order of the shading of the input image.

[0064] In some aspects, the training of the first neural network further comprises use of a multi-gradient loss function to evaluate estimated shading and the differences between the relative shading order of the pixels of the at least one first shading image and the relative shading order of the shading of the input image.

[0065] In some aspects, training of the second neural network comprises use of the low resolution training shading images for scale.

[0066] In some aspects, the statistical analysis comprises taking the per-pixel median of the plurality of albedo images.

[0100] In some aspects, the method further comprises: training at least one neural network to perform image decomposition using the network training data.

[0067] in accordance with anther aspect of the present disclosure, a method of image decomposition is disclosed, the method comprising: processing an input image with a first neural network to generate a first shading estimate, the input imagecomprising a shading component corresponding to shading that depicts a diffuse illumination effect on the content, an albedo component corresponding to an albedo that depicts diffuse reflectance on the content, and a residual component corresponding to a residual that depicts non-diffuse illumination effect on the content; processing the first shading with a second neural network to generate a second shading estimate; generating a first albedo estimate based on the second shading estimate; processing the second shading estimate and the first albedo estimate with a third neural network to generate a color estimate; generating a third shading estimate based on the color estimate; generating a second albedo estimate based on the third shading estimate; processing the third shading estimate and the second albedo estimate with a fourth neural network to generate a third albedo estimate; generating a fourth shading estimate based on the third albedo estimate; processing the fourth shading estimate and the third albedo estimate with a fifth neural network to generate a fifth shading estimate; generating a residual estimate based on the fifth shading estimate and the third albedo estimate, the first neural network trained to estimate the shading based on a relative shading order of pixels of the shading; generation of shading estimates by one or more neural networks comprises representing the shading by mapping shading values to a domain of pre-defined range, each shading value representing shading of a pixel; and the fifth shading estimate corresponding to the shading component, the third albedo estimate corresponds to the albedo component, and the residual estimate corresponds to the residual component.

[0068] In accordance with another aspect of the present disclosure, a system for comprising one or more processing units configured to perform the method of any one of the above aspects is disclosed.

[0069] In accordance with another aspect of the present disclosure, a non- transitory computer-readable medium having computer readable instructions stored thereon is disclosed, which, when executed by one or more processing units, causes the one or more processing units to perform the method of any one of the above aspects.BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Further features and advantages of the present disclosure will become apparent from the following detailed description, taken in combination with the appended drawings, in which:FIG. 1 depicts a system for performing image decomposition, according to an example embodiment.FIGS. 2A and 2B depict a process for performing image decomposition utilized by the system of FIG. 1 , in particular the deconstruction of an image into a Lambertian albedo component and a greyscale shading component, according to an example embodiment.FIGS. 3 and 4 depict examples of albedo images, shading images, and reconstructed images obtained by the process of FIGS. 2A and 2B in comparison to other methods.FIGS. 5 and 6 depict examples of shading images obtained by the process of FIGS. 2A and 2B using inverse shading showing distribution of shading values in comparison to other methods.FIG. 7 depicts an example of lower resolution shading images and higher resolution shading images obtained by the process of FIGS. 2A and 2B.FIG. 8 depicts an example of an albedo image and a shading image obtained by the process of FIGS. 2A and 2B in which loss in shading and albedo images are used in network training in comparison to another method in which only loss in shading is used for network training.FIG. 9 depicts examples of original images, pseudo ground truth albedo images and corresponding shading images obtained and used by the process of FIGS. 2A and 2B.FIG. 10 depicts examples of images of a multi-illumination dataset, the corresponding albedo images obtained by the process of FIGS. 2A and 2B, and the median albedo image used for network training.FIG. 11 depicts examples of albedo images and shading images obtained by the process of FIGS. 2A and 2B in which synthetic images and real-world images are used for network training in comparison to another method in which only synthetic images are used for network training.FIGS. 12-15 depict examples of albedo images and shading images obtained by the process of FIGS. 2A and 2B from images of various objects and scenes in comparison to other methods.FIG. 16 depicts an example of an original image and a reconstructed image generated using the albedo and shading obtained by the process of FIGS. 2A and 2B in comparison to reconstructed images generated using albedo and shading obtained by other methods.FIG. 17 depicts example performance results of the process of FIGS. 2A and 2B in comparison to other methods.FIG. 18 depicts an example of a shading image obtained by the process of FIGS. 2A and 2B in comparison to shading images obtained using other approaches for shading processing.FIG. 19 depicts an example of an albedo image and a shading image obtained by the process of FIGS. 2A and 2B in comparison to albedo images and shading images obtained using other network inputs.FIG. 20 depicts example performance results of the process of FIGS. 2A and 2B for use in relighting in comparison with other methods.FIG. 21 depicts example performance results of the process of FIGS. 2A and 2B for use in recoloring in comparison with other methods.FIG. 22 depicts example performance results of the process of FIGS. 2A and 2B for use in editing material properties.FIG. 23 depicts example effects of Lambertian shading assumption.FIG. 24 depicts a process for performing further image decomposition utilized by the system of FIG. 1 , in particular to deconstruct an image into a diffuse albedo component, a diffuse shading component, and a residual component, which can be applied after the process of FIGS. 2A and 2B, according to an example embodiment.FIG. 25 depicts an example of a diffuse albedo image, diffuse shading image, and residual image obtained by the process of FIG. 24.FIG. 26 depicts an example of various albedo images obtained during the process of FIG. 24.FIG. 27 depicts an example of a chroma image, various shading images, and residual images obtained during the process of FIG. 24.FIG. 28 depicts an example of albedo correction.FIG. 29 depicts example albedo images obtained by the process of FIG. 24 in comparison to other methods.FIG. 30 depicts an example of an albedo image and a shading image obtained by the process of FIG. 24 in comparison to other methods.FIG. 31 depicts examples of highlights recovered using the process of FIG. 24.FIG. 32 depicts examples of applications for the process of FIG. 24 in white balancing and specularity removal.FIG. 33 depicts examples of specularity removal using the process of FIG. 24.

[0071] It will be noted that throughout the appended drawings, like features are identified by like reference numerals.DETAILED DESCRIPTION

[0072] With the advancement in artificial intelligence (Al), there is an increased interest in applying Al models such as neural networks in image processing. For example, convolutional neural networks are particularly suitable for image analysis and processing. However, shading is a continuous-valued map that represents thecomplex interactions between the light sources, the 3D geometry, and the material properties present in the scene. This makes intrinsic decomposition a high-level problem wherein neural networks have to rely on contextual information in the scene. These challenges are coupled with the lack of dense ground-truth data on real-world images. Due to the complexity of the problem, state-of-the-art intrinsic decomposition models fail to produce accurate results at high resolutions in the wild.

[0073] The present disclosure relates to systems and methods for high- resolution intrinsic decomposition using an Al model comprising a plurality of neural networks, which can be achieved by breaking the technical problem of image decomposition into two parts. First, a first neural network can employ a dense ordinal shading formulation using a shift- and scale-invariant loss to estimate ordinal shading cues without restricting the predictions to obey the intrinsic model, which defines an image as the product of the albedo and shading. The first neural network can output low- and high-resolution ordinal estimations, which can be combined using a second network to generate a shading estimate with both global coherency and local details. The neural networks can be configured to learn accurate decomposition by computing losses on the estimated shading as well as the albedo implied by the intrinsic model. Methods for generating dense pseudo ground truth are also disclosed herein, which can be generated using the Al model’s predictions with multi-illumination data, which can enable generalization to in-the-wild imagery. Further, the systems and methods of the present disclosure may have numerous real-world applications including performing otherwise difficult editing tasks such as recoloring and relighting.

[0074] Intrinsic image decomposition is a fundamental mid-level vision technical problem that aims to represent an image as the product of the reflectance of the materials and the effect of illumination in the scene represented by Equation 1 below:I = A * S, (Equation 1) where I, A, and S represent the input image, the albedo, particularly albedo in the Lambertian model, and the shading, respectively. The intrinsic model described by Equation 1 can be inherently under-constrained, as well as scale-invariant, whichmakes computer-implemented intrinsic image decomposition technically difficult. For example, for a given A and S, 1 / c A and cS also satisfy the intrinsic model for all c>0.

[0075] As described further herein, it is possible to achieve high-resolution intrinsic decomposition by breaking the technical problem of intrinsic decomposition into two. In particular, a first step may involve the use of a scale- and shift-invariant (SSI) dense shading estimation formulation using a neural network, as disclosed herein, which may be referred to as ordinal shading. Accordingly, this neural network may be referred to as an “ordinal network”. In this formulation, the constraints may be relaxed to satisfy the intrinsic model while enforcing the estimation of reliable ordinal relationships between pixels in the dense output. By utilizing this formulation of simplified ordinal problem definition, it is possible to estimate shading smoothness and discontinuities at high resolutions.

[0076] In a second step, two dense ordinal shading maps may be generated at two different resolutions. For example, a low-resolution ordinal shading can be generated at the receptive field resolution of the ordinal network to provide global ordinal constraints. The low-resolution estimation can provide globally coherent ordinal constraints but may lack high-resolution details. The second ordinal shading can be generated at a higher or much higher resolution to provide detailed shading discontinuities as local constraints but may have inconsistencies across distant image regions. These two ordinal maps may be used as input together with the original image for a second neural network that is trained to enforce the intrinsic model through losses determined in respect of both albedo and shading. With high resolution ordinal constraints readily available to the network, it is possible to generate highly detailed intrinsic decompositions that can be used in image editing tasks. In particular, it is possible to generate a globally consistent shading with high-resolution details and sharp shading discontinuities. The corresponding albedo may be then determined using the input image and the estimated shading using the intrinsic equation / model.

[0077] Intrinsic decomposition networks are typically trained on synthetic data with some real-world data with sparse ground-truth annotations. The disclosed networks may be trained with high-resolution real-world dense ground truth data, which may be derived from multiple illumination dataset(s). In particular, since albedois illumination-invariant, and hence a constant across changing illumination, it is possible to formulate a robust pseudo-ground-truth generation method using a plurality of images per scene with different lighting conditions provided in a multiple illumination dataset by estimating the constant albedo. Accordingly, it is possible to generalize a high-resolution intrinsic decomposition method to in-the-wild examples using dense training on real photographs from the multiple illuminations dataset.

[0078] As physically-based rendering techniques have improved, it has become more feasible to train intrinsic decomposition networks using rendered datasets. For example, it is possible to train neural networks using direct supervision on small-scale datasets. Further, there exists datasets of rendered scenes with ground-truth intrinsics. Although many approaches have been proposed that utilize the rendered datasets, these approaches typically depend on scale-invariant losses to directly regress the shading and albedo. Due to the complexity of the intrinsic decomposition problem and the limited capacity of neural networks, these methods can fail to generate accurate and high-resolution estimations. By first making use of the simpler ordinal definition of the problem followed by regression of full decomposition with ordinal constraints as input, as disclosed herein, it may be possible to generate high-resolution decompositions with sharp shading discontinuities and globally coherent sparse albedo maps.

[0079] Some data-driven models may utilize architectures that estimate shading and albedo separately. These methods can enforce constraints on each intrinsic component and incorporate a reconstruction loss that favors outputs that reproduce the input image when multiplied. However, these methods may not generate a faithful reconstruction of the input image for a novel scene, which can limit their use in image editing applications. The disclosed systems and method can derive the albedo using estimated shading and Equation 1. Since this process is differentiable, it is possible to optimize for both albedo and shading using dedicated losses to each while using a single network and to ensure image reconstruction.

[0080] Although a number of synthetic datasets have been developed with ground-truth intrinsic components, it is still a difficult task to train models that generalize to real-world imagery. Rendered datasets typically depict homogeneousindoor scenes. Certain datasets also contain outdoor scenes rendered using a video game engine. However, there is a domain gap between the training data, which would usually be synthetic images, and in-the-wild photographs. To address this shortcoming, many methods supplement their training procedure with sparse annotations for shading smoothness based on real-world data. However, as the sparse annotations are only able to provide a weak supervision, current datasets for use in generalizing to in-the-wild photographs are limited. It is possible to leverage image sequences of stationary scenes under varying illuminations where similar loss functions are used that encourage reflectance consistency across multiple illuminations, but can require complicated priors and do not guide the network toward a single ground-truth. The present disclosure provides systems and methods for generating pseudo-ground-truth intrinsic components using multiple illuminations dataset(s). This can enable the use of a unified dense loss formulation on both synthetic and real data, which can be a key step to bridging the intrinsic decomposition generalization gap.

[0081] The disclosed systems and methods can generate smooth shading results on heavily textured surfaces such as the cloth while reconstructing the input image faithfully. The disclosed formulation can also be robust against challenging regions in the image such as specularities. It is also possible to apply the disclosed formulation to cases not represented in the training data set such as human faces. As described further herein, the disclosed systems and methods can provide improvements in performance in terms of albedo sparsity, shading smoothness, and sharpness of shading discontinuities in a variety of scenarios. Further, the disclosed intrinsic decomposition systems and methods can enable realistic image editing applications in the wild such as recoloring and relighting.

[0082] As noted above, the disclosed systems and methods can separate the surface reflectance and the effects from the illumination given a single photograph, for example, by assuming a single-color illumination and a Lambertian world. The disclosed systems and methods can also be further adapted to separate an input image into its diffuse albedo, colorful diffuse shading, and specular residual components. This further adaption can be achieved by gradually removing first thesingle-color illumination and then the Lambertian-world assumptions. In particular, by dividing the technical problem of image decomposition with diffuse shading into easier sub-problems, in-the-wild colorful diffuse shading estimation can be achieved despite the limited ground-truth datasets. Further, this extended intrinsic model can be used for illumination-aware image editing applications such as specularity removal and per- pixel white balancing.

[0083] Specifically, the intrinsic model defined by Equation 1 may be a grayscale intrinsic diffuse model, where I is the input image in linear RGB, A is the 3- channel albedo, and S is the single-channel grayscale shading. As such, this intrinsic model may be based on two assumptions. The first assumption is the Lambertian world assumption that allows for the two-component multiplicative representation of the image by modeling all surfaces as diffuse. However, by ignoring specular surfaces, this model may not allow for separate editing of diffuse and non-diffuse illumination effects. The second assumption is the single-color shading that can limit the model’s representation of colorful illumination effects that are common in real scenes such as multiple light sources and inter-reflections. Therefore, these color effects may be embedded in the albedo layer and can limit the effectiveness color editing applications.

[0084] To further decompose illumination into diffuse shading and a non-diffuse residual, it is possible to define an intrinsic residual model, represented by Equation 2 below:1 = A * S + R (Equation 2).

[0085] Equation 2 can extend the intrinsic diffuse model with an additive component R that represents non-diffuse lighting effects such as specularities and visible light sources and defines S as an RGB map that reflects the color of illumination. This enhanced capability to model real-world scenes comes at the cost of complexity, for example, by increasing the number of unknown variables from 4 (e.g. 3 channel in A and 1 in S) to 9 per pixel (3 channels in each of A, S, and R), which can exacerbate the under-constrained nature of the problem, made worse by the lack of diverse ground truth.

[0086] In accordance with the present disclosure, systems and methods that can generate decompositions under the intrinsic residual model for in-the-wild photographs are disclosed. In particular, a decomposition that uses the intrinsic diffuse model such as ones disclosed herein may be used, and by gradually removing the single-color shading and the Lambertian world assumptions, it is possible to estimate the diffuse albedo and the colorful diffuse shading at high resolutions. As disclosed herein, it is possible to first estimate the chroma of the shading using the global context present in the scene that can be used to create a sparse diffuse albedo. Given the diffuse albedo, it is possible to decompose the shading into diffuse and specular components. Accordingly, by breaking this highly under-constrained task into multiple conceptually simpler sub-problems, the disclosed systems and methods can generalize to complex in-the-wild scenes. Further, several illumination-aware image editing applications including per-pixel white-balancing and specularity removal that can be made possible by utilizing the intrinsic residual model.

[0087] Existing approaches for solving the RGB diffuse model (e.g. where shading is estimated in RGB) can include unsupervised methods for learning intrinsic components via time-lapse data which parameterize the shading component as a grayscale map multiplied by a global RGB color cast or by estimating an unconstrained RGB shading component. Others may model a RGB shading layer and further decompose shading into separate light sources. However, these methods rely on low-level assumptions and user input and may only be suitable for simple scenes. Other works can implicitly account for colorful shading effects by directly estimating albedo but may constrain the albedo via an image reconstruction loss using the grayscale diffuse model and therefore can be lacking in ability to accurately model colorful lighting effects. Moreover, an intrinsic diffuse model is also not common given the difficulty of the problem and lack of real-world ground-truth supervision. As such, estimates of specularity can only be performed in specific scenarios, or limited / singular objects, therefore limiting the real-world applicability of these methods.

[0088] Alternatively, inverse rendering methods tackle the broader task of estimating all intrinsic scene parameters necessary to re-render an image. These methods may estimate an albedo component explicitly and render shading via inferredgeometry and an illumination model. Although this is a slightly different task formulation, inverse rendering methods may be generally comparable to intrinsic decomposition methods as they still produce intrinsic components.

[0089] Inverse rendering methods can be limited by the availability of training data and are often restricted to indoor scenes. Some methods leverage diffusionbased image generation models to generate plausible intrinsic components conditioned on a given input image where the problem can be model as probabilistic, stemming from the under-constrained nature of the task. Other methods focus on close-up object images and point to the ambiguity between the albedo and illumination colors or focus on indoor images and point to different rendering engines and 3D models in CGI pipelines that occasionally embed several lighting effects in reflectance. These methods may compensate for the random nature of their outputs by averaging over multiple estimations, which results in a loss of details. Additional approaches can directly estimate high-resolution intrinsic components, but may suffer from being constrained to the latent space of the diffusion model.

[0090] In contrast, the systems and methods disclosed herein can estimate unconstrained RGB shading, both specular and diffuse, in the wild without the need for explicit assumptions or constraints. Despite being trained on indoor scenes, the systems and methods as described further herein can generate accurate estimations for out-of-distribution images. Moreover, the present disclosure can focus on the deterministic nature of real-world image formation and may show that material and color ambiguities can be resolved by data-driven algorithms through context present in the scene.

[0091] Embodiments are described below, by way of example only, with reference to FIGS. 1-33.

[0092] FIG. 1 depicts a system for performing intrinsic image decomposition, according to an example embodiment, shown in FIG. 1 as one or more servers 108. The implementation of the servers 108 is not restrictive and servers 108 may be a physical server, cloud-based server, or a hybrid thereof, for example. A user may interact with the servers 108 via a device 102 over a communications network 102(e.g. the internet). The device 102 may be a computer, as depicted in FIG. 1 , but is not restricted to those expressly shown and may be any suitable device known in the art such as smart phones and tablets. The servers 108 may provide a user interface on the device 102 for ease of communication and operation control by the user. The implementation of the user interface is not restrictive and may be, for example, a mobile / computer application or a web page.

[0093] According to the present disclosure, an image 104 may be provided to the servers 108, for example, from the device 102. The image 104 may be a photograph and in particular a 3-channel RGB photographic image or photorealistic image. The image 104 may be in a standard image format such as JPEG or PNG. The servers 108 may perform intrinsic image decomposition on the image 104 by performing image analysis and processing. That is, the servers 108 may process the image 104 to generate an albedo image 122 representing an albedo component of the image 104 extracted from the image 104. The albedo image 122 may depict a reflectance of content depicted in the image 104. The servers 108 may also process the image 104 to generate a shading image 124 representing a shading component of the image 104 extracted from the image 104. The shading image 124 may depict an illumination effect on the content depicted in the image 104. For example, as described with reference to FIGS. 2A and 2B, the servers 108 may generate an albedo image 122 being a 3-channel RGB Lambertian albedo image representing the albedo component of the image 104 under Lambertian assumption as well as a shading image 124 being a single channel greyscale shading image representing the shading component of the image 104 under a Lambertian (greyscale) assumption. In accordance with a further aspect of the present disclosure as described with reference to FIG. 24, the servers 108 may generate an albedo image 122 being a 3-channel RGB diffuse albedo image representing the albedo component of the image 104 where diffuse and non-diffuse lighting effects are considered; a shading image 124 being a 3-channel RGB diffuse shading image representing the diffuse shading component of the image 104 where diffuse and non-diffuse lighting effects are considered; and at least one residual image 126 being a non-diffuse residual image representing the non-diffuse residual component of the image 104. As depicted inFIG. 1 , the albedo image 122, shading image 124 and / or residual image 126 can be returned and provided to the device 102.

[0094] To perform intrinsic image decomposition and to generate the images 122, 124 and / or 126, the servers 108 may process the image 104 using a plurality of neural networks 120. The neural networks 120 may comprise a first neural network 120a, a second neural network 120b, a third neural network 120c, a fourth neural network 120d, and a fifth neural network 120e, as depicted in FIG. 1. The first neural network 120a may be an ordinal network trained to generate a low resolution shading image and a high resolution shading image from the image 104. In particular, the first neural network 120a may estimate the shading of the image 104 by maintaining a relative shading order of the pixels in generated images based on shading values derived from the shading of the image 104 during training. For example, the numerical shading value in each pixel of a generated image relative to other pixels (e.g. greater or smaller) in the generated image may be maintained. The second neural network 120b may be an intrinsic decomposition network trained to generate a greyscale shading image from the images generated by the first neural network 120a and the image 104. A Lambertian albedo image may be then generated by dividing the greyscale shading image from the image 104. The third neural network 120c may be a chroma network trained to generate a chroma image (e.g. chroma layer or map) or color component image representing the chromaticity contribution to the shading of image 104 from the Lambertian albedo image and the greyscale shading image. The generated chroma (e.g. the color component image) may be a two channel image used to augment the greyscale shading image. For example, the greyscale shading image may be combined with the color component image to generate a chromaticity adjusted shading image (e.g. a RGB shading image / layer). A chromaticity adjusted albedo image may be then generated by dividing the chromaticity adjusted shading image from the image 104. The fourth neural network 120d may be an albedo network trained to generate a diffuse albedo image from the image 104, the chromaticity adjusted albedo image and the chromaticity adjusted shading image. A diffuse illumination adjusted shading image representing shading adjusted by the estimated diffuse albedo may be generated by dividing the chromaticity adjusted albedo image from the image 104. The fifth neural network 120e may be a diffuse shading networktrained to generate a diffuse shading image from the image 104, the diffuse albedo image and the diffuse illumination adjusted shading image representing the contributions of diffuse illumination on the image 104. The at least one residual image 126 may be generated by subtracting the product of the diffuse albedo image and the diffuse shading image from the image 104. The generation of the various images and the neural networks 120 are described in further detail herein.

[0095] In a particular implementation, the servers 108 each comprise a CPU 110, a non-transitory computer-readable memory 112, a non-volatile storage 114, an input / output interface 116, and graphical processing units (“GPU”) 118. The non- transitory computer-readable memory 112 comprises computer-executable instructions stored thereon at runtime which, when executed by the CPU 110, configure the server to perform the above described processes of automatic characteristic value calculation. The non-volatile storage 114 has stored on it computer-executable instructions that are loaded into the non-transitory computer- readable memory 112 at runtime. The input / output interface 116 allows the server to communicate with one or more external devices such the device 102 (e.g. via network 106). The non-transitory computer-readable memory 112 may also have thereon the neural networks 120. The GPU 118 may be used to control a display and may be used process the image 104 and to perform image analysis and decomposition. In some aspects, the neural networks 120 may be stored at one or more separate servers. The CPU 110 and GPU 118 may be one or more processors or microprocessors, which are examples of suitable processing units, which may additional or alternatively comprise an artificial intelligence accelerator, programmable logic controller, a microcontroller (which comprises both a processing unit and a non-transitory computer readable medium), Al accelerator, neural processing unit (NPU), or system-on-a-chip (SoC). As an alternative to an implementation that relies on processor-executed computer program code, a hardware-based implementation may be used. For example, an application-specific integrated circuit (ASIC), field programmable gate array (FPGA), or other suitable type of hardware implementation may be used as an alternative to or to supplement an implementation that relies primarily on a processor executing computer program code stored on a computer medium.

[0096] It should be noted that while FIG. 1 depicts the device 102 and the servers 108 as separate entities coupled over the communication network 106, the device 102 and servers 108 may also be coupled directly / physically using cable(s) for data transfer. In some embodiments, the servers 108 may also be the device 102 or comprise the device 102 (e.g. the servers 108 being implemented as a part of a computer system). In such an embodiment, the servers 108 may directly retrieve the image 104 from local storage or removable local storage. In some embodiments, the image 104 may be captured by an image capturing device such as a camera.

[0097] In accordance with the present disclosure, high-resolution intrinsic decomposition may be achieved by the servers 108 by defining the problem in two steps, as depicted in FIGS. 2A and 2B, which depict the deconstruction of an image into a Lambertian albedo component and a greyscale shading component, according to an example embodiment. For example, input images 202, 216 can be provided to a first neural network 204, where the first neural network may be an ordinal network trained to determine shading representing an illumination effect on content depicted in the input image. Input image 202 is shown with highlighted areas 202a, 202b.

[0098] In the first step, local and global constraints for shading discontinuities and smoothness may be generated. These constraints may be generated through a dense ordinal shading formulation as disclosed herein. Rather than regressing the absolute shading values that satisfy the intrinsic model (e.g. Equation 1), it is possible to define a relaxed loss that enforces the correct ordering of shading values. When a result (e.g. shading image) is generated at the resolution (low resolution shading images 208, 218) of the receptive field, the ordinal shading network can generate a coherent structure for the entire scene, as seen in highlighted areas 208a, 208b of the lower resolution shading images 208 corresponding to highlighted areas 202a, 202b of the input image 202. At higher resolutions (e.g. generated higher resolution shading images 206, 220), the ordinal shading network can lose global coherence due to the receptive field size being smaller than the estimation resolution, as seen in highlighted areas 206a 206b of the higher resolution shading images 206 corresponding to highlighted areas 202a, 202b of the input image 202.

[0099] However, as described herein, the ordinal shading network can predict highly detailed local shading discontinuities, for example, using the higher resolution result, which can be achieved with the relaxed formulation that does not regress continuous shading values but instead promotes the correct ordering of pixels.

[0100] The low-resolution and high-resolution ordinal estimations, as a result, can represent two important sets of clues about shading: the global ordering of the shading values in the entire image, and detailed discontinuities in the local neighborhood of a pixel. The two estimations may be provided together with the original image to a second network 210 (e.g. an intrinsic decomposition network) that generates a full result (e.g. shading image) at high resolutions. The second network may generate consistent results beyond its receptive field as the global structure of the shading may be provided in the form of the low-resolution ordinal input / estimation. The result generated by the second network can comprise detailed shading discontinuities derived from the provided high-resolution ordinal input / estimation.

[0101] For example, the input image 202, the low resolution shading image 208, and the high resolution shading image 206 can be provided to the second neural network 210, which may be trained to further determine the shading of the input image. The second neural network can generate a greyscale shading image 214 (or 224) from the input image 202 (or 216), the low resolution shading image 208 (or 218), and the high resolution shading image 206 (or 220), with the greyscale shading image comprising a plurality of pixels represented using the numerical shading values for depicting the shading of the original image. A corresponding albedo image 212 (or 222) can be generated by using Equation 1 to divide the greyscale shading image 214 (or 224) from the from the input image 202 (or 216), with the albedo image comprising a plurality of pixels represented using the color channel values for depicting reflectance of the content in the input image. As shown in FIG. 2A, the generated albedo image 212 and greyscale shading image 214 may have globally consistent albedo and shading with high-resolution details and sharp shading discontinuities as seen respectively in the highlighted areas 212a, 212b, and 214a, 214b.

[0102] Notably, an increase in the estimation quality may be achieved with the use of the ordinal formulation (e.g. in the first step). It should be noted that dense ordinal maps may be provided to the second network to generate the full decomposition. Further, the second network can perform full intrinsic decomposition that satisfies the intrinsic equation using the ordinal estimations as constraints together with the original image as input. Another benefit of this approach can be seen in FIG. 3, which depicts an input image 302 which can be used by an example embodiment to generate an albedo image 304 and a shading image 306. The generated images may be recombined to produce a reconstructed image 308 using Equation 1 to replicate the input image 302. The difference between the reconstruction image 308 and the input image 302 is shown as difference image 310, which shows that there is no difference between the two images. However, for the reconstruction images 320a, 320b, 320c produced using estimated albedo and shading from other methods, the difference (322a, 322b, 322c) can be more pronounced, especially in highlighted regions 324, 326.

[0103] As such, the generated images may be used for recoloring (shown in recolored image 226) and and / or relighting (shown in relighted image 228 using a normal 228a). Referring to FIG. 3, the generated albedo image 304 and shading image 306 using the systems and method of the present disclosure can be better than similar images produced using other methods, as seen in albedo images 312a, 312b, 312c and shading images 314a, 314b, 314c generated by other methods, particularly in highlighted regions 316, 318. Similarly, as shown in FIG. 4, for input images 402 (first column), the estimated results 404 (second column) of an example embodiment can be superior to the results 406a, 406nb, 406c of other methods, particularly in highlighted regions 408, 410, 412, 414, which depict various challenging areas such as textured wall (408), cloth texture (410), specularities in the lamp post (412) and human faces (414).

[0104] Inverse Shading Representation

[0105] Most natural scenes can contain specular objects and objects with very dark albedos, both of which may result in very large shading values. As a result, the shading values in a scene can span a very wide range of values with long-taileddistributions. This skewed distribution characteristic may decrease the contrast in the shading image by concentrating valid shading values in a small window inside the range. This property can make direct regression of shading challenging especially when coupled with the scale-invariant nature of the problem. Although some methods opt to model the problem in logarithmic domain which can increase the contrast in the distribution, this approach still lacks a well-defined range for shading values.

[0106] In accordance with the present disclosure, an inverse shading domain defined in [0,1] can be used to define shading values in both ordinal and full shading estimation by the neural networks as follows:(Equation 3) where S represents the linear-scale shading (values). The inverse shading domain can create a more uniform distribution in the [0,1] range. This uniform distribution with high contrast may be able to more accurately represent intricate shading variations such as smooth gradients and very high shading values on specular objects. The bounded nature of this representation can be more amenable to deep networks as it allows for a predictable input and output range, thereby addressing a technical problem. It also makes it possible to use a saturating activation function in the neural network such as a sigmoid, which may be more stable during training when compared to other functions such as rectified linear unit (ReLU). It should be noted that despite the ambiguous scale, the inverse shading representation can preserve the ordinal relationships in the shading domain, i.e. Dt< Dj for ) > 5) for all pixel pairs (ij).

[0107] That is, the shading values of the shading image can be derived by mapping the shading values to a domain with defined bounds (e.g. bounded domain). By restricting the shading values to a defined range, the performance of the networks in determining the shading estimate may be improved as the mapped shading values may have more predictable behaviours, for example, with a defined range and normal distribution. For example, the shading values may be tonemapped to be in a predefined range. More specifically, each shading value can be mapped to follow a predefined mapping function such that the mapped shading values are bounded to a domain with a pre-defined range. Further, the neural networks can process the inputimages using inverse shading values (e.g. numerical values representing the shading of each pixel), which can be the inverse of the shading value associated with each pixel that is representative of the illumination effect. The domain of the inverse shading values may be 0 to 1. It should be noted that other domain ranges are also possible. The inverse shading values may be reverted to regular shading values for the generation of the shading estimation. The mapping of the shading values can be performed using any suitable neural network(s) for determining an estimated shading.

[0108] For example, as shown in FIG. 5, for a given input image 502, deriving a shading image 504 using unmodified values can have poor results, which can be a result of the shading values being distributed tightly in a particular range (510). The estimated shading (506) using log shading values can show better results but are still unbounded (512). In contrast, by using inverse shading values / domain, the inverse shading values are well distributed in a well defined range (514), which can produce a better estimation (508).

[0109] Dense Ordinal Shading Estimation

[0110] The shading layer as defined in Equation 1 (e.g. as S) may be a continuous-valued scale-invariant variable that is required to satisfy the core model in every pixel. As such, estimating this under-constrained variable for complex scenes at high resolutions can be challenging. As a first step, instead of estimating the continuous shading values directly, it is possible to utilize an ordinal definition of the problem of intrinsic decomposition.

[0111] The ordinal definition of the problem can be seen as a relaxed reformulation of direct shading estimation. Instead of satisfying the intrinsic model as defined in Equation 1 , the network can instead ensure that the inequalities between the shading of individual pixels are satisfied. As a result, in the ordinal world, (S) may be as valid a result as S for any monotonically increasing / decreasing function (• ), as it preserves the inequality relations in shading.

[0112] It is possible to define dense ordinal loss, in which the output of the neural networks in the inverse shading domain De[0,1], and in which the ordinality inshading is preserved. A relaxed loss function may be formulated for use that still ensures the correct ordering in the estimation as follows:(Equation 4) where O can be the estimated ordinal shading and D* can be the ground-truth mapped shading, and f(-) is a monotonically-increasing (or decreasing) function, such as an affine function defined as: (%) = ax + b, (Equation 5)a > 0. (Equation 6)

[0113] Accordingly, it is possible to not penalize the estimated result by the network(s) if the ordering (e.g. linear ordering) of the estimated shading values is accurate. As such, the estimated ordinal shading 0 can be a mapping of shading values through a monotonically increasing or decreasing function, such as an unknown scale and shift away from the ground truth mapped shading, while satisfying the ground-truth ordinal relationships.

[0114] The network(s) trained with this dense ordinal loss can learn to correctly order the shading values for each pixel, while not necessarily satisfying the intrinsic model (e.g. Equation 1 ). This simplified problem definition can allow the network(s) to generate ordinal results with more high-resolution details when compared to direct shading estimation.

[0115] It should be noted that the scale and shift loss may be applied to promote ordinality. While it can be replaced with any monotonically-increasing (or decreasing) function in our formulation, this particular first-order function (•) may be more stable during training. Further, it is possible to estimate a dense result with the dense ordinal shading formulation, where the result has the same resolution as the input image that maintains the pair-wise ordinality between the shading values of any pixel pair using the monotonically increasing transformation (•)•

[0116] It is possible to define a smoothness loss by using a multi-scale gradient loss £°nsgon the ordinal shading as follows:

[0117] (Equation ?) where V can denote the spatial gradient and I can denote the level of a multi-scale pyramid. This loss defined on the gradients may be useful in generating spatially consistent estimations.

[0118] The overall loss function for ordinal shading estimation can be as follows in Equation 8:(Equation 8) where .°nsgcan be set as 0.5. This loss can be evaluated on the single-channel ordinal shading estimation (i.e. greyscale) generated by the network.

[0119] That is, the ordinal network can process an input image to generate at least one first shading image from the input image, where the at least one first shading image depicts the shading of the input image. For example, the shading image may be a shading estimation representing an illumination effect on content depicted in the input image. Specifically, the input image can comprise a plurality of pixels represented using color channel values and the at least one first shading image can comprise a plurality of pixels represented using numerical shading values. Further, as described above, the first neural network can be trained to determine the shading by minimizing loss based on minimizing differences between a relative shading order of the pixels in the at least one first shading image and a relative shading order of pixels in the shading of the input image. In particular, the shading estimated by the network may not be penalized if the relative shading order in the output matches that of the ground truth. In some embodiments, the shading values are derived by the first neural network for generating the at least one shading image using inverse shading values, where each inverse shading value can be an inverse of a shading value associated with each pixel that is representative of the illumination effect.

[0120] The structure of the ordinal shading network can consist of an encoderdecoder network proposed in with a ResNext101 encoder. Pre-trained weights may be omitted and the network may be trained from scratch. A sigmoid activation may be added to the end of the network to bound the generated predictions to [0,1], The network may be trained using the Adam optimizer with a learning rate of 10'5. Further, the ordinal shading network can be trained with a variety of synthetic datasets as well as the dense real-world dataset that can be derived from multi-illumination data, as described further herein.

[0121] For example, as shown in FIG. 6 by way of an example, for an input image 616, the shading estimation (622) of an example embodiment utilizing both the inverse shading domain and allowing the shift of shading values (while maintaining shading order) may be better than an estimation (620) using only the inverse shading domain, which is better than conventional shading estimation (618), particularly for highlighted areas 624, 626, 628.

[0122] Multi-Resolution Behavior

[0123] The ordinal shading network may be fully-convolutional and therefore able to generate ordinal predictions at resolutions beyond or exceeding the training resolution. For example, the training resolution can be 384 x 384 (pixels). While the network can generate high-resolution outputs, the input resolution may change the characteristics of the estimated ordinal shading.

[0124] With reference to FIG. 7, when the input image (images 702) is resized to (or below) the training resolution for inference, it is possible to generate a consistent shading structure for the entire scene depicted in the input image. For example, since the entire image fits in the receptive field size of the network, it is possible to derive consistent shading structure in the estimation (e.g. low-resolution images 704). This behaviour can be due to how convolutional neural networks operate: since the network can “see / visualize” the entire image at once, it may be able to generate global ordinal shading constraints. However, especially in complex scenes, high-resolution details can be missing in this estimation. This is seen, for example, in highlighted areas 708, 710, 712, 714 (for images 704). The lack of high-resolution details cancome from the limited capacity of the neural network. For example, a network that has to produce a complex shading structure may not be able to also generate high- resolution details.

[0125] When the input image is provided to the network at higher resolutions than its receptive field, it is possible to generate intricate shading variations in high resolution (e.g. high resolution images 708). However, this can come at the cost of global coherency and may be a result of the limited receptive field size where the network cannot produce the correct ordering of pixels that are spatially far away from each other. This is seen, for example, in highlighted areas 708, 710, 712, 714 (for images 706). Nonetheless, as the receptive field can “see” image patches with lower local scene complexity, the neural network can generate high-resolution details much more accurately. This behaviour can indicate that there is a trade-off in ordinal shading estimation quality with increasing input resolution. For example, the low- resolution estimations can create reliable global constraints and the high-resolution estimations can generate detailed shading discontinuities.

[0126] It should be noted that monocular depth estimation and shading estimation can share many similarities especially in the ordinal definition of the problems. Further, the image edge density can be used to determine the resolution at which the network can still produce consistent results. This resolution can be defined as??0, and can be computed as the largest resolution in which every receptive field-sized region in the image contains strong image edges. Additionally, JZ0, which can depend on image content, may be a good upper limit for the ordinal shading estimation to produce high-resolution details while creating spatially coherent results. Limited capacity of the neural networks and lack of very high-resolution training data may require a limit on the size of input images, for example, to have at most 1500 pixels in either width or height. These resolutions for ordinal shading estimation can provide the intrinsic decomposition network with reliable local ordinal constraints with detailed high-frequency shading discontinuities.

[0127] That is, the ordinal network may generate a low resolution shading image with a resolution that is within a receptive field of the first neural network, the low resolution shading image providing global context of the shading, and a highresolutions shading image with a resolution that that exceeds the receptive field of the first neural network, with the high resolutions shading image providing local context of the shading. The shading images may be generated as shading estimates. Further, to train the ordinal network, shading images comprising high resolution shading images and low resolution training shading images representing the actual shading at the input resolutions may be used as ground truth. Training images may be scaled to generate images of appropriate resolution.

[0128] Intrinsic Decomposition with Ordinal Input

[0129] In order to achieve high-resolution intrinsic decomposition, two ordinal shading estimations may be provided as input to the intrinsic decomposition network together with the linear RGB image (e.g. the source / input image). The first ordinal estimation, OL, can be generated at the receptive field size of the ordinal shading network, for example 384 x 384 (pixels). Since the first ordinal estimation can be generated at the receptive field resolution, the ordinal network can generate an accurate ordering of the shading values across the entire image, for example by using all the information present in the scene. OL, in the end, may provide a reliable general structure for the final shading estimation and can provide the global constraints for the entire image. The second ordinal estimation, OH, can be generated at a much higher resolution JZ0(e.g. exceeding the receptive field). As this estimation is generated at a higher resolution than the receptive field, it may lack global coherency. However, this estimation can also contain high-resolution shading discontinuities and can provide intricate local constraints to the second network (e.g. the intrinsic decomposition network). The ordinal inputs OLand OHmay be provided to the second network by concatenating them with the input image across the color channel dimension to create an input to the second network of size (e. g.H x W x 5) after upscaling OLto the resolution of OH.

[0130] This process can provide contextually rich information to the second network to simplify the intrinsic decomposition task. The decomposition network may not be required to conduct the high-level task of inferring the overall structure of the shading, which requires reasoning about global context such as geometry and illumination direction, as the global context may be provided in the form of OL. At thelocal level, similarly, the second network may not be required to determine whether a strong RGB edge comes from a shading discontinuity or a sudden change in the albedo, as the local context can be provided in the form of OH. As a result, the task of our decomposition network can become simply adjusting the overall structure from OLto satisfy the intrinsic model in Equation 1 while integrating the details in OHto the final estimation. This way, it is possible to regress continuous shading values with intricate details at high resolutions.

[0131] Output Formulation

[0132] The intrinsic decomposition network can generate the result (e.g. shading image) in the single-channel inverse shading domain. Shading can be computed the using Equation 3 as well as the albedo using the intrinsic model in Equation 1. For example:> l-D > I > I*D - , A - - ,D S l-D where I and D can represent the input image and the estimated inverse shading, respectively. That is, the intrinsic decomposition network may also map shading values to a domain of pre-defined range, such as from between 0 and 1 .

[0133] Some intrinsic decomposition methods estimate the shading and albedo as two different variables and formulate a reconstruction loss using Equation 1 . This approach can result in the estimated S and A not being able to reconstruct the image I correctly. In contrast, the present systems and methods can compute the albedo from the estimated shading, such that the intrinsic model in Equation 1 holds.

[0134] Inferring the albedo from the estimated inverse shading can also allow for the definition of independent losses on the two components (e.g. shading and albedo). While the components are related through the input image, the albedo and shading losses can have complementary information with respect to each other. In particular, it is possible to define losses in respect of the albedo in addition to the inverse shading and backpropagate them through Equation 8 (shown below). Accordingly, it is possible to generate high-resolution shading estimations that yield accurate albedo components at the same time.

[0135] Scale Ambiguity

[0136] Becasue of the inherent scale ambiguity of the intrinsic decomposition formulation, some approaches may rely on scale-invariant losses to supervise deep networks. These scale-invariant losses can require a least-squares fit between the network estimation and the ground truth during training. At later stages of the training, this scale can be computed effectively using least squares. However, during the initial stages of training, the scale may have to be computed between the ground truth shading and the highly inaccurate estimations from the under-trained network due to the outlier-prone nature of least-squares.

[0137] In the disclosed setup with ordinal inputs, the globally consistent low- resolution ordinal estimation OLcan provide a point of reference to the intrinsic decomposition network. Since the ordinal network may be trained before the intrinsic network, the computed OLcan be derived using a well-trained system, such that the least-squares fit between OLand the ground-truth can yield stable results. Therefore, low-resolution input can be used to set the arbitrary scale in the ground truth as follows in Equation 8: (Equation 8)where A" can represent the ground-truth albedo at an arbitrary scale. This fixed scale can be used to define the ground-truth shading, inverse shading, and albedo as follows in Equation 9:(Equation 9)

[0138] The scale matching can be performed on the albedo component due to its limited range of values, unlike shading with long-tailed distributions, for example. The average scale of the high-resolution ordinal input OHcan be fixed to that of OLto obtain the input and output variables at the same overall scale. By fixing the arbitrary scale, it is possible to define dense losses without the need for scale invariance.

[0139] Dense Losses

[0140] An L1 loss or mean squared error loss can be used together with the multi-scale gradient loss on the estimated inverse shading using Equations 10 and 11 below: tion 10)(Equation 11) where V can denote the spatial gradient, and I can denote the level of a multi-scale pyramid. The definition of the loss on the inverse shading instead of shading can come from the constrained range of inverse shading between [0,1], The very large shading values in S, coming from the long-tailed distribution, can result in skewed values in the loss. The balanced distribution of shading values in [0,1] can also help to better represent shading gradients and intricate details around specular regions.

[0141] The same two losses may be defined on the albedo using Equations 12 and 13 below: tion 12)(Equation 13) which can be backpropagated through Equation 8 to be combined with the losses on the inverse shading.

[0142] By defining the output of the neural network as the inverse shading and deriving the estimated albedo using the intrinsic model, it is possible to define and combine losses on both albedo and shading components. Albedo and shading maps / images can have their own unique characteristics that are helpful for the network to model the underlying statistics. For instance, the smoothness of shading can represent continuous geometric surfaces, while high gradients in the shading can represent geometric discontinuities or shadows. The high-gradient regions on the albedo, however, can represent textured surfaces while a smooth albedo can be derived across shadows. By defining the multi-scale gradient loss on both components, it is possible to signal both of these characteristics to the network at thesame time, which can make both the shading and the albedo estimates more accurate. The joint loss overall can improve the accuracy as well as results in images with dark regions. For example, by making use of the albedo priors when there is not enough signal in the regions with very dark shading.

[0143] For example, as seen in shading image 808 and albedo image 810, which may be generated by an example embodiment from an input image 802 using networks trained with loss on shading and albedo, the estimations can be more accurate in comparison to shading image 804 and albedo image 806, which are generated using networks trained with loss on shading only.

[0144] That is, the intrinsic decomposition network can process the input (original) image, the low resolution shading image, and the high resolution shading image (e.g. generated by the ordinal network) to further determine the shading of the input image. In particular, a second shading image being a greyscale shading image may be generated as a shading estimate comprising a plurality of pixels represented using the numerical shading values for depicting the shading of the input image. Further, an albedo image representing the albedo estimating can be generated from the input image and the second shading image, which can comprise a plurality of pixels represented using the color channel values for depicting reflectance of the content in the input image. For example, the albedo image may represent Lambertian assumption reflectance.

[0145] Network Structure and Training

[0146] The encoder-decoder architecture used for the ordinal network may be used for the intrinsic decomposition network. Given the simplified nature of this task, it is possible to use a smaller and more memory-efficient EfficientNet encoder. A sigmoid activation can be added to the end of the network to bound the inverse shading estimation to [0,1], The network can be trained using the Adam optimizer with a learning rate of 10“5. Further, low-resolution rendered datasets may be omitted. Training data may comprise synthetic images and training data derived from a multiIllumination dataset as described further herein. Further, the ground truth for the training of the intrinsic decomposition network may be shading images representingthe actual shading of the original image. In some embodiments, albedo images representing the actual albedo of the original image may be used as well, for example, for calculating the loss with regard to albedo.

[0147] Training on Rendered and Real-World Data

[0148] It can be hard for intrinsic decomposition networks to generalize to real- world images from synthetic-only training data. It can also be challenging to create dense ground-truth data for real images. As such, the only real-world ground truth datasets available may be in the form of sparse ordinal comparisons. These sparse annotations can be useful as a form of real-world supervision but may not allow the use of any dense loss on shading, albedo, or their gradients, which can limit their usefulness in generalizing to in-the-wild input.

[0149] In accordance with the present disclosure, a dense, high-resolution real- world intrinsic decomposition dataset can be generated by exploiting the redundant information present in photographs of the same scene captured under different illuminations, for example, as depicted in FIGS. 9 and 10. The disclosed networks may be trained initially with a set of synthetic datasets. Then, using the network(s), it is possible to formulate a robust albedo estimator that estimates a single consistent albedo using different illuminations for the same scene in a multi-illumination dataset. As such, it is possible to generate a real-world dense dataset (e.g. based on indoor images) that can allow the disclosed systems and methods to generalize to in-the- wild examples with challenging phenomena such as out-of-focus blur, as well as outdoors photographs and novel subjects not present in the training datasets such as the human face.

[0150] For training using synthetic data / images, training can begin with the ordinal shading network, as the decomposition network may require ordinal estimations as input. Synthetic image sets may be used, which can include scenes with outdoor environment. For example, the ordinal network may be trained for 700,000 iterations with a batch size of 8, sampling images from each image set. It is possible to generate the low-resolution and high-resolution ordinal inputs for images in the training data sets and to train the decomposition network. For example, thedecomposition network may be trained for 200,000 iterations with a batch size of 8 and a resolution of 512x512 (pixels). After this training process, the disclosed systems and method may be effective in scenes that are similar to the training data such as indoor environments (e.g. which may be present in multi-illumination datasets).

[0151] Multiple (multi-) illumination datasets may present a number of constant scenes under different ambient lighting conditions (e.g. the same objects under different lighting conditions), for example as obtained through a rotating motorized flashlight with a static camera (e.g. images 1002 which depict the same scene under different lighting). With shared scene content and varying illumination, a plurality of photographs for each scene can have the same albedo with different shadings. The shared albedo can be estimated (e.g. for use as ground-truth) using the shading predictions generated by the disclosed neural networks for the plurality of photographs for each scene.

[0152] In particular, after the initial training, the disclosed systems and methods can estimate a good albedo for each photograph (e.g. albedo images 1004 can be derived from the original images 1002). However, each individual estimation may not be perfect, especially in challenging regions such as poorly illuminated parts of the scene, shadow boundaries, or specularities. Due to the varying illumination across photographs, these imperfections can appear independently in different parts of each image, for example, as seen in highlighted areas 1008, 1010, 1012, 1014.

[0153] The shared albedo can be estimated robustly by combining the individual estimations (e.g. of the same scene generated from different photographs corresponding to different lighting conditions) of the same shared albedo. It is possible to match the scale of the individual albedo estimations. For example, the first estimated albedo in each set (e.g. corresponding to each group of images for the same scene) can be used to determine the scale of the rest of the estimations as follows in Equation 14:k E {2,3, . . ., N}, (Equation 14) where N can represent the number of images, Akand Akcan represent the initial and scale-adjusted albedo estimations for the kthphotograph, respectively, and thesummation is defined over every pixel i. For example, once all the individual albedo estimations are at the same scale, it is possible to compute the robust shared albedo component (e.g. median albedo 1006) as the per-pixel median across the plurality of estimations as following using Equation 15:,4** = median({Ak}k=1). (Equation 15)

[0154] This median operation, due to the high number of estimations for example, may be able to filter out many of the issues in the individual estimations. Since each sequence contains a plurality of images, this method can generate sparse albedo components and shading even for difficult images that contain hard shadows, flash illumination, and specularity. Accordingly, it is possible to generate high-quality shared albedo estimations (e.g. as ground-truth) for use in training the disclosed neural network(s).

[0155] That is, it is possible to generate photograph / real-world training data by obtaining a plurality of photographs for a particular scene under different lighting conditions and generating a plurality of albedo images corresponding to the plurality of photographs with the above-described neural networks or other image decomposition methods. A pseudo ground truth albedo (predicted albedo) can be generated as an albedo image by taking the per pixel median of the plurality of generated albedo images. It should be noted that other methods to determine the predicted albedo are also possible. Other statistical analysis methods can be performed on the generated albedo images to determine the predicted albedo, for example, by performing per-pixel averaging or using by another suitable averaging algorithm. Further, by using Equation 1 , it is possible to generate a plurality of shading images by dividing each of the corresponding photographs by the pseudo ground truth albedo image. The shading images, the pseudo ground truth albedo image, and the photographs can be used as training data for training neural networks to perform image decomposition using real-world data where the shading images and pseudo ground truth albedo image can be used as ground truth for training.

[0156] Examples of possible pseudo-ground truth albedo and shading pairs generated according to an embodiment of the present disclosure are shown in FIG.9, where images 902a, 902b, 902c respectively correspond to the generated pseudo ground truth albedo (median albedo) images 904a, 904b, 904c, which can be used to generate the respective corresponding pseudo ground truth shading images 906a, 906b, 906c. As shown in highlighted areas, 908, 910, 912, 914, 916, 918, the estimated shading and albedo can be well represented.

[0157] To generalize to real-life photographs, the network(s) may be trained using the generated multi-illumination ground-truth pairs. There can be improvements in model generalization after training with the multi-illumination data as processed above. FIG. 11 depicts example shading images generated by an example embodiment trained using multi-illumination data. For input images 1102, the shading images generated (1106) after training with multi-illumination data is more adaptable for estimating shading when comparing the shading images (1104) generated by a network trained with only synthetic data. For example, the model trained with multiillumination data can make accurate predictions on out-of-distribution scenes such as human faces (e.g. as seen in highlighted areas 1108, 1110). Additionally, there may be an increase in detail on high-frequency content such as the leaves and hair. Further, it may be possible to make reliable predictions even in the presence of difficult camera effects such as out-of-focus blur.

[0158] In particular, the ordinal network and the intrinsic decomposition network may be trained with synthetic training data and the photograph training data (generated as described above). The training data may also be obtained for other sources.

[0159] It should be noted that FIGS. 12-22 depict images generated in accordance to an example embodiment and includes comparisons between the generated images and images generated using other methods.

[0160] Qualitative Evaluation

[0161] Generalization to in-the-wild photographs can be an important challenge that many intrinsic decomposition methods is yet to address. The performances of an example embodiment of the present disclosure are evaluated on a variety of scenes and compared to that of other methods.

[0162] Comparison of estimated albedos

[0163] FIG. 12 shows a plurality of input images 1202 including an outdoor scene and a portrait taken in a studio environment. The albedo images (1204) generated by the example embodiment are compared to albedo images (1206) generated by other methods. When the albedo estimations are analyzed, it can be seen that other methods can have trouble generating a smooth albedo across shadow boundaries. For example, some of the other methods fail to disentangle the shadows in their estimated albedo. Some of the other methods generate albedo maps with little contrast and color content, while others slightly shift the color of the scene overall. Similar shortcomings can be seen in the portrait of the man, where most other methods fail to remove the shadows from the man’s face, on his clothing, and on the ground. The lack of contrast in some of the other methods as well as the color shift in some of the other methods are present in this scene as well. In the outdoor scene, the example embodiment can estimate a sparse albedo that remains consistent across the self-shadows on the building as well as the cast shadow on the ground. Similarly, for the portrait image, the example embodiment can generate a flat color for the face and the jacket, and remove the shadow on the ground from the albedo.

[0164] Comparison of estimated shadings

[0165] As depicted in FIG. 12, the shading images (1208) generated by the example embodiment are compared to shading images (1210) generated by other methods. Some of the other methods struggle with a lack of sharpness and contrast in their results. For example, some of the other methods are generally prone to generating very smooth shading maps that lack sharp shading discontinuities. Further, some of the other methods generate low-frequency artifacts, especially in high-resolution estimations. Some of the other methods also often fail to reflect the shadow boundaries in shading, which are incorrectly represented in the corresponding albedo estimations (e.g. as seen in highlighted areas 1212, 1216). The example embodiment can successfully represent the sharp shading discontinuities across shadow boundaries and generate realistic, high-contrast shading maps. For example, it can be seen on the man’s face (1214) that even though the trainingdatasets may not contain any ground truth for people, it is possible to generate an accurate shading map with a sparse albedo.

[0166] Reconstructing the original image

[0167] Some of the other intrinsic decomposition methods formulate separate albedo and shading estimation modules in their pipelines, using the intrinsic model in Eguation 1 as a loss on the reconstruction of the original image. For example, FIG. 16 depicts an example input image 1602 and an image 1604 reconstructed using albedo and shading estimated from the input image 1602 by the example embodiment compared to that of other methods (1606a, 1606b, 1606c, 1606d, 1606e). As seen in FIG. 16, however, the final results do not necessarily satisfy this eguality. Some of the other methods lose much of the original color content in their reconstruction, some of the other methods lose contrast mainly due to their shading estimate, and some of the other methods shift the original colors in the image mostly due to their albedo estimate. While some of the other methods are able to reconstruct the image more faithfully than others, they may fail to reconstruct bright colors or create large artifacts as seen in the sky in FIG. 16. The example embodiment, which computes the estimated albedo directly from the shading, does not suffer from reconstruction errors as the image is perfectly reconstructed by definition. This can be a beneficial property for intrinsic decomposition methods since this reconstruction informs any down-the- line image editing tasks.

[0168] To examine some of the typical challenges in intrinsic decomposition at a deeper level, FIGS. 13-15 depict a number of albedo and shading images generated by the example embodiment in comparison to another method. FIG. 13 depicts example input images 1302 and the corresponding albedo images generated by the example embodiment (1304) and another method (1308) as well as the corresponding shading images generated by the example embodiment (1306) and another method (1310), with highlighted areas 1312. Similarly, FIG. 14 depicts example input images 1402 and the corresponding albedo images generated by the example embodiment (1404) and another method (1408) as well as the corresponding shading images generated by the example embodiment (1406) and another method (1410), with highlighted areas 1412. Further, FIG. 15 depicts example input images 1502 and thecorresponding albedo images generated by the example embodiment (1504) and another method (1508) as well as the corresponding shading images generated by the example embodiment (1506) and another method (1510), with highlighted areas 1512.

[0169] Shadows on surfaces

[0170] Since shadows are a direct result of illumination, a smooth albedo and a sharp discontinuity in shading across shadow boundaries may be expected. FIG. 13 shows that cast shadows on flat surfaces of the attic image can be well- represented in the example embodiment with a sparse corresponding albedo, while the albedos estimated by another method can have the shadows still present. Similarly, sparse albedo maps can be generated across the strong cast shadow boundaries on the leaves in the peppers shown in FIG. 13 and on the wall and the ground in FIG. 15. The other method, on the other hand, generates low-contrast shading maps in such regions with shadows still visible in their estimated albedo.

[0171] Color shift under mixed illumination

[0172] The single-channel shading model assumes a single-color illumination. In the presence of light sources with varying colors, such as light coming from outdoors and the ambient light present in the room with the desks in FIG. 14, as well as the brick room and kitchen in FIG. 15, the color of the ambient light can be represented by a color shift in the albedo. Similarly, the shadows in indoor scenes are only illuminated with secondary illumination reflected from the surrounding surfaces. The color of this ambient illumination can cause a color shift in the albedo as the attic in FIG. 13 and the chairs in FIG. 14 show. While this colored illumination can be well- represented in the example embodiment, we see that the albedos estimated by the other method show a strong shift in albedo brightness while representing the shadows incorrectly in the estimations.

[0173] Shading smoothness on primitive geometry

[0174] A smooth shading gradient can be expected on regions with simple geometry such as a plane or a cylinder. Despite the dramatic change in the albedo onthe mug in FIG. 13 or on the ground in the kitchen of FIG. 15, the example embodiment can generate a smooth gradient across these surfaces. The geometry of the cup holder in FIG. 13 can also be clearly visible in the shading estimation of the example embodiment. On the contrary, in the shading estimations by another method, a strong shading discontinuity can be seen on the smooth ground in the kitchen of FIG. 15, the pattern from the albedo still visible in the attic of FIG. 13, and the shadows not well represented in the attic of FIG. 13.

[0175] Albedo sparsity

[0176] Single-colored objects may appear with a sparse albedo map free of shading variations in a successful intrinsic decomposition. As the peppers in FIG. 13 under strong sunlight and chairs in FIG. 14 under a soft ambient light show, the example embodiment can generate sparse albedos for such objects under varying conditions. In these examples as well as in desk of FIG. 14 and kitchen of FIG. 15, the albedo maps estimated by the other method incorrectly include the changes in shading, resulting in non-sparse albedos for flat-colored regions.

[0177] Specular surfaces

[0178] Specular surfaces can be challenging for intrinsic decomposition methods due to their very bright shading when compared to surfaces that can be modeled as Lambertian. The disclosed inverse shading representation can allow the example embodiment to represent the wide range of shading values effectively by spreading the distribution of shading values in [0,1], The example embodiment can generate smooth shading maps of specular objects such as the lamp on the desk in FIG. 14, the kitchen hood in FIG. 15, or the metallic surfaces in the chairs in FIG. 14. The example embodiment can also generate high-contrast shading maps for materials like leather as FIG. 15 show. The example embodiment can represent smaller specular regions such as on the glasses in on the desk in FIG. 14 and on the legs of the chair in FIG. 14. As shown, the example embodiment can provide an improvement in representing specular surfaces.

[0179] Quantitative Evaluation

[0180] While qualitative comparison can be a reliable source of evaluation for intrinsic decomposition, quantitative evaluations are also provided below.

[0181] Dataset 1

[0182] In an attempt to quantify the physical accuracy and reconstruction of different approaches, quantitative comparisons on a first data set (dataset 1) are performed in the zero-shot setting where the example embodiment is not trained on any data in dataset 1. The example embodiment is compared to other methods, as described herein.

[0183] Evaluation Dataset

[0184] Dataset 1 consists of approximately 150 rendered scenes varying in size, realism, and subject matter. Although the scenes are rendered, they can allow for concrete quantitative comparisons since the ground-truth intrinsics are provided.

[0185] Evaluation Metrics

[0186] Evaluating intrinsic decomposition approaches on dense ground-truth data can be difficult, and oftentimes metrics do not reflect qualitative observations. Any metrics utilized may be required to be scale-invariant given the inherent scale ambiguity of the intrinsic image formulation. For example, a regular scale-invariant mean squared error (si-MSE) can be unforgiving since the incorrect attribution of a single edge to either shading or albedo can result in errors across large regions of the image. An alternative metric, local mean squared error (LMSE), can compute si-MSE in overlapping windows. LMSE, scale-invariant root mean squared error (RMSE), and structural-similarity (SSIM) are utilized herein. In addition to measuring the accuracy of each intrinsic component, the scale-invariant reconstruction error of each approach is also measured.

[0187] Quantitative Analysis

[0188] The quantitative results of the evaluation are shown in Table 1. The example embodiment achieves the best performance across the board when considering zero-shot methods. Although some of the methods train on many of theimages from dataset 1 , the example embodiment still performs competitively with these methods. The example embodiment can also outperform the other methods in all metrics on shading prediction. Due to the disclosed formulation, the example embodiment also has zero reconstruction error by definition. All other methods (e.g. data driven methods) result in some reconstruction error.

[0189] Table 1 also shows the results of an example embodiment (example 2) when trained with less training data (e.g. without using real-world images from the multi-illumination dataset). In particular, the training data consists of only approximately 20,000 examples of rendered indoor scenes. Despite this, this example embodiment is still able to achieve great results. Similar to the standard example embodiment, the lesser trained example embodiment can still outperform the other methods in shading estimation. This can show that a contributing factor of the performance for the disclosed systems and methods is the design of our two-step shading estimation pipeline. Table 1 : Performance Comparison in Dataset 1

[0190] Dataset 2

[0191] Some of the other methods use the pair-wise reflectance judgments of dataset 2 to evaluate the quality of their inferred albedo components. The dataset is introduced with a metric called weighted human disagreement rate (WHDR). This metric measures the rate at which predicted ordinal relationships match the ground truth annotations, weighted by a confidence computed based on the consistency of human annotations. It has been shown that the WHDR metric of dataset 2 may not reflect the true physical accuracy or editing capability of intrinsic decomposition approaches. An accurate comparison of methods on dataset 2 is complicated given that some of the methods perform training and / or validation on dataset 2. For thoroughness, the example embodiment used for comparison herein is not trained on dataset 2.

[0192] Table 2 (below) shows the results of various other methods on dataset 2. Although WHDR scores have trended downward with the introduction of data- driven methods, reported scores can vary drastically depending on whether or not a method is trained on dataset 2. Naturally, methods that fine tune on dataset 2 achieve significantly lower scores than those that do not. While some of the other methods may not directly train on dataset 2, they may use dataset 2 as validation when early- stopping their training, or to drive the design choices of the other methods, resulting in noticeably lower scores.

[0193] Rather than this change in score reflecting generalization to real-world scenes, there may be easily exploitable aspects of the WHDR metric. For example, by simply scaling the input images into the range [0.55,1], a score of 25.7 can be achieved due to the imbalance of equal and non-equal annotations. Similarly, by simply shifting the albedo estimations of the example embodiment by 0.5, it is possible to achieve better performance. Notably, the shifted albedo values are not constrainedto be in the [0,1] range. Methods trained on dataset 2 may use this imbalance to produce lower scores which can make it difficult to compare approaches. This is further exemplified in the images shown in FIG. 17, which depicts an example input image 1702, an annotated image 1708, along with the albedo image (1704) and shading image (1710) generated forthe input image 1702 compared to albedo images (1706) and shading images (1712) generated by other methods, along with highlighted areas 1714 (shown to the right of the images). In particular, certain methods yield lower scores, despite perceptually worse albedo estimations.Table 2: Performance Comparison in Dataset 2

[0194] Dataset 3

[0195] An example embodiment is also evaluated on dataset 3. Dataset 3 consists of annotations of shading smoothness and discontinuities on images from other datasets. Given a shading estimation, each region can be classified as smooth or non-smooth based on image gradients. An average precision metric can be determined by computing the classification precision of these regions over varying gradient threshold values. An improved challenge metric in which smooth shading regions are weighed proportionally to how difficult they are to predict properly is used. Since many smooth shading regions can occur over smooth regions of the image, this improved metric can compute the average image gradient over each smooth shading region to determine its difficulty.

[0196] Similar to the WHDR metric, this metric may only give a partial evaluation of decomposition quality. Since this metric only measures smooth regions and discontinuities, some of the other methods can achieve high scores with low contrast shading estimations as long as a measurable amount of gradient is predicted at discontinuities. In particular, the metric does not measure whether or not shading discontinuities have the correct magnitude. Nevertheless, the example embodiment performs competitively, outperforming almost all other methods, some of which use dataset 3 as validation to evaluate the performance of their model while training.

[0197] Runtime Analysis

[0198] To effectively utilize intrinsic decomposition for image editing applications, a given algorithm should be efficient in both time and memory. Taking multiple minutes of run-time or requiring a high-end GPU can greatly limit the usability of an intrinsic decomposition approach.

[0199] The disclosed systems and methods comprise the use of two networks, where the ordinal network can be larger than the intrinsic decomposition network. Given the local and global ordinal estimation formulation, the image may be run through the first network twice, once at the training resolution and a second time atthe 7?0resolution. These two estimations can be then provided to the second network, along with the input image to compute the final shading estimation.

[0200] All timing and memory consumption measurements herein are performed using a machine with an RTXTM2060 Super GPU, an Intel™ i5-9600k CPU, and 16 gigabytes of RAM. An example embodiment takes 0.3 seconds in total to process a 768x768 (pixels) image. Other methods take 0.06, 0.08 and 0.005 seconds respectively. This discrepancy in runtime may be due to the multiple forward passes that are required for the example embodiment. Another one of the other methods takes 8 seconds for the same image.

[0201] As for memory efficiency, the example embodiment uses 1.2 gigabytes of GPU memory for an image of size of 768x768 (pixels). One of the other methods requires about 0.7 gigabytes for the same size image. Some of the other methods require more memory than an example embodiment of the present method, using 1.3, 2.5 and 8 gigabytes, respectively.

[0202] Accordingly, a technical effect of the present disclosure can be to increase computational efficiency when performing image decomposition.

[0203] Ablation

[0204] To evaluate the various design implementations of an example embodiment, multiple controlled experiments are performed where zero-shot evaluations are made using dataset 1.

[0205] Analysis of Ordinal Training

[0206] The efficacy of the ordinal training formulation can be evaluated using a controlled experiment. Three strategies for learning ordinal shading estimation are compared: regular shading with a scale-invariant loss, inverse shading with a scaleinvariant loss and inverse shading with a scale- and shift-invariant loss, as described above.

[0207] For the experiment, 5,000 images are sampled from two dataset’s training split and each approach is trained for 100 epochs. The gradient loss described in Equation 8 is included in each experiment.

[0208] To evaluate the generalization capabilities of each approach, zero-shot evaluation on dataset 1 is performed. Existing ordinal metrics from monocular depth estimation are used to compare the effectiveness of each training formulation. The pair-wise ordinal metric (Ord.) measures the correctness of ordering between randomly sampled pixels. The D3R metric similarly measures ordinality across discontinuities determined by the super-pixel segmentation of the ground-truth shading. These parameters can roughly quantify global ordinality and local ordinality, respectively. Two resolutions are evaluated: 384 pixels and JZ0.

[0209] Table 3 (below) shows the resulting ordinal metrics across various training setups. The first row shows the typical intrinsic decomposition training strategy wherein the network can directly regresses values in the original shading space using a scale-invariant loss (si-MSE). The second row shows that by learning the shading in a transformed inverse space, the network may be better at generating globally coherent ordinal estimations, but the D3R metric can indicate worse local performance at high-resolution. The example embodiment instead uses a scale- and shift-invariant loss (ssi-MSE) in the inverse shading space. This approach performs the best in both metrics at both resolutions, which can indicate that the example embodiment is capable of generating accurate global structure as well as local discontinuities. Training of a fourth network using ssi-MSE in the original shading space was attempted, but diverged. This is may be due to a poor distribution of values, combined with the shift- and scale-invariant loss function.

[0210] For example, FIG. 18 depicts example input image 1802, the corresponding ground truth image 1804, as well as the shading images generated by incorporating the use of regular shading values (1806), inverse shading values (1808), and inverse shading values and ordinal shading (1810) (for example by an example embodiment), shown with a highlighted area 1812 (below the images) that can show the improved shading estimation by using inverse shading values and ordinal shading for neural networks.Table 3: Performance Comparison for Different Approaches for Ordinal Training

[0211] Analysis of Joint Albedo and Shading Loss

[0212] The effectiveness of utilizing losses on both estimated shading and the corresponding implied albedo can be shown by performing a controlled ablation study with and without albedo loss.

[0213] Training is performed using a dataset consisting of 10,000 examples. An example embodiment is used to generate input estimations and to train a second network for evaluation. The two variants are both trained for 100 epochs with a learning rate of 10'5.

[0214] Zero-shot evaluation is performed on dataset 1. The final estimations are evaluated using three scale-invariant metrics. Specifically, scale-invariant root mean squared error (RMSE), LMSE, and SSIM are measured on both the predicted shading and albedo components. Each training formulation is evaluated at 1 ,024-pixel resolution.

[0215] Table 4 (below) shows the results of the joint loss ablation experiment. The first row shows the result of only computing losses on the shading component during training. Adding losses on the implied albedo can improve the performance of the model on albedo estimation, as shown. Furthermore, the albedo loss can even slightly improve the accuracy of the shading estimation, suggesting that these losses may provide complementary forms of supervision. Additionally, when the network is only trained using the losses on the shading component, artifacts may be produced in image regions with low shading values. Some examples of this behavior are shown in FIG. 8. The artifacts can be alleviated by the addition of the albedo loss, which can further exemplifying the harmonious nature of these two supervision sources.Table 4: Performance Comparison of Training using Different Loss

[0216] Multi-Resolution Ablation

[0217] The effectiveness of the multi-resolution ordinal estimations can be evaluated by performing a controlled experiment over various possible input configurations of the second network.

[0218] An example embodiment can be used to generate input estimations and to train a second network for evaluation. The four variants are trained for 100,000 iterations with a learning rate of 10'5and a batch size of 8.

[0219] Zero-shot evaluation is performed on dataset 1. The final estimations are evaluated using three scale-invariant metrics. Specifically, scale-invariant root mean squared error (RMSE), LMSE and SSIM are measured on both the predicted shading and albedo components. The evaluation is performed at the above described JZ0resolution.

[0220] Table 5 (below) shows the results of the ablation. The first row shows the result of an example embodiment where the second network is provided with the RGB image, and ordinal estimations at both the base resolution and the full resolution. The two following rows show approaches that exclude one of these two ordinal estimations. By excluding either of the ordinal estimations, the performance of the neural network may degrade, especially on the more difficult globally scale-invariant metrics RMSE and SSIM. Furthermore, the final row shows the performance of a second network trained without any ordinal estimations by providing the network with only the RGB image. This configuration can have decreased performance across all metrics.

[0221] There are also differences in the generated images between each training setup as shown in FIG. 19, using an example input image 1902. Details in the input image and the generated images are shown in the highlighted areas 1920 (bottom right of images). When only provided with the low-resolution ordinal estimation (albedo 1908, shading 1910), the network may be unable to accurately predict sharp details in the shading layer, which can result in inaccurate predictions on small shadows (e.g. the thin shadow). When only provided with the high-resolution ordinal estimation (albedo 19012, shading 1914), the model can generate detailed shading predictions but with global inconsistencies across distance image regions (on the water and on the building). When no ordinal estimation is used (albedo 1916, shading 1918), the model may fail to predict accurate shading. The example embodiment (e.g. all inputs) can generate very detailed estimations (albedo 1904, shading 1906) that are also globally coherent at high resolutions due to the multiresolution approach.Table 5: Performance Comparison of Different Inputs for Intrinsic Decomposition

[0222] Applications

[0223] The accuracy of the decompositions of an example embodiment relative to other intrinsic image editing methods is evaluated. Two types of edits that are difficult to perform without access to intrinsic components, namely, illumination-aware recoloring, and single-image relighting are performed.

[0224] Relighting

[0225] Since albedo is a representation of the scene without any lighting effects, it can be an essential component for relighting. In order to show the usability of the example embodiment, relighting is performed using the generated albedo and a standard surface normal estimator. The relighting results are rendered by loading the albedo and the estimated normal into Blender™ software and using a standard diffuse shader to generate a novel shading layer. Virtual point lights are inserted into the scene to simulate various lighting conditions.

[0226] Example relighting effects using the example embodiment are shown in FIG. 20 and compared against other methods. FIG. 20 depicts input images 2002, the relighted images (2004) generated using the estimations of the example embodiment compared to those of other methods (2006), as well as highlight features 2008, 2010. The example embodiment can generate relit images without residual shading effects from the original illumination. In the top row (church), other methods leave shadows on the building and the albedo of one of the other methods yields a relit image with low contrast. In the bottom left (house) example, it is shown that the example embodiment is able to remove the hard shadows on the sides of the houses and on the street. An example from FIG. 12 (building with blue sky) is also shown where the relit image is generated by placing a light source as if it is coming directly from the camera, therefore each face of the building should have the same brightness. Other methods fail to remove the shadow from the left side of the building, resulting in inaccurate relit images.

[0227] Recoloring

[0228] Simple recoloring edits can be made using intrinsic components. By altering the colors of certain regions in the albedo and reapplying the shading, it is possible to edit surfaces while maintaining lighting effects. In the case of solid color objects, this kind of edit can be trivially achieved by decomposing the image into its chrominance and luminance. When both albedo and shading vary in a given region, the two components may need to be disentangled to perform recoloring.

[0229] To generate the recoloring results, estimated shading and albedo of an example embodiment are determined and compared to other methods. Since theother methods may not enforce image reconstruction, the shading and the input image are used compute the implied albedo using Equation 1. This albedo is used for recoloring so that each method reconstructs the image and can be compared fairly. Example recoloring edits from the example embodiment are shown in FIG. 21 and compared to other methods. FIG. 21 depicts input images 2102, the recolored images (2104) generated using the estimations of the example embodiment compared to those of other methods (2106), as well as highlight features 2108, 2012, 2114. The example embodiment can accurately separate lighting effects even when the albedo varies greatly in the edited region. The yellow chair and red carpet examples can show that other methods either do not faithfully represent lighting effects or exhibit residual albedo patterns leaking into their shading component. In the museum example, the example embodiment, unlike other methods, can represent subtle specular effects that significantly increase the realism of the edit.

[0230] In addition to recoloring surfaces, it is also possible to perform simple material editing by altering the distribution of estimated shading values determined with the example embodiment. FIG. 22 shows examples of this type of edit with the original albedo and recolored albedo. In particular, FIG. 22 depicts images edited to have varying levels of specularity using the original albedo (2202) and an edited albedo (2204). By exponentiating the shading, the values can be compressed or expanded causing the edited surface to appear more, or less specular.

[0231] As noted previously, the gray-scale shading assumption can result in shifted albedo colors in regions with multi-colored illumination, including the hard shadows where the illumination is dominated by secondary reflections. While the example embodiment can represent the specularities in the estimated shading, the presence of specularities may adversely affect image editing results which relies on the Lambertian shading assumption. This assumption can also affect the smoothness of the estimated shading on objects composed of different materials as seen, for example on the t-shirt in FIG. 23, which depicts an input image 2302, the corresponding shading estimate under greyscale assumption (2304) and color edited image using the estimated shading (2306).

[0232] As described and evaluated above, high-quality and high-resolution intrinsic decomposition that allows the editing of illumination and material recoloring can be achieved using the systems and methods of the present disclosure. The introduction of the dense ordinal shading representation that simplifies the task can be used to generate highly detailed shading discontinuities. The ordinal shading may be estimated in two resolutions, which can be used as input to full intrinsic decomposition that may make it possible to regress the result at high resolutions. It is also possible to estimate the shading layer in the inverse shading domain such that it is possible to properly represent the specularities as well as the dark regions in the shading layer within [0,1], Further, real-world training data may be generated using a robust albedo estimator from a multi-illumination dataset. By training on the generated dataset, it may be possible to bridge the domain gap between real-world and synthetic images. Additionally, despite the dataset consisting of indoor scenes, example embodiments may generalize to diverse image content such as humans and outdoor scenes.

[0233] As described above, intrinsic decomposition under non-Lambertian assumption, which considers diffuse and non-diffuse lighting effects, may be more accurate. Accordingly, the present disclosure also provides systems and methods for performing intrinsic image decomposition where diffuse and non-diffuse illumination are considered, for example, under the intrinsic residual model, which is described below. The systems and methods described below (e.g. described with reference to FIGS. 24-33) may be used with or as an extension to the systems and methods described above with reference to FIGS. 2-22, as described with reference to FIG. 1 .

[0234] In accordance with the present disclosure, an image I can be decomposed into its diffuse albedo Adand colorful diffuse shading Sdlayers with a residual layer / ? containing non-diffuse illumination effects using the intrinsic residual image formation model according to Equation 16 below:1 = Ad* Sd+ R. (Equation 16)

[0235] This highly under-constrained problem described by Equation 16 can require a network to reason about high-level contextual cues about scene geometry,global and local illumination conditions, and material properties. The scarce high- resolution ground truth and the lack of real-world datasets for the diffuse shading component can make it challenging for neural networks to statistically model the image formation in-the-wild.

[0236] In order to achieve in-the-wild generalization, the disclosed systems and methods can divide the problem into simpler, physically-motivated sub-problems that are convenient for neural networks to model. For example, it is possible start from an existing intrinsic decomposition of the image that relies on the simplified Lambertian intrinsic model with a grayscale shading component Sgaccording to Equation 17 below:I = Ag* Sg. (Equation 17)

[0237] The systems and methods described above, particularly with reference to FIGS. 2-22 can be used to generate an Ag-Sgpair that provides an initial starting point. It is possible to gradually remove the grayscale shading assumption, and then the Lambertian-world assumption, to arrive at the extended model in Equation 16, as depicted in FIG. 24, which shows a process for performing further image decomposition.

[0238] As shown in FIG. 24, an input image 2402 may be processed by a greyscale intrinsic decomposition network 2404, such as those described above, to generate a greyscale shading image 2406 and a corresponding greyscale albedo image 2408. Further, other methods to generate the greyscale shading image 2406 and the corresponding greyscale albedo image 2408 are possible as well.

[0239] Shading Chroma Estimation

[0240] The grayscale shading assumption may simplify the problem by setting the albedo chromaticity to that of the input image. In some embodiments, it is possible to abandon the grayscale assumption and extend the previously described systems and methods to the RGB intrinsic diffuse model using Equation 18 below:1 = AC* SC, (Equation 18)which may require inferring the per-pixel chromaticity of the shading layer. For this purpose, the input grayscale shading Sg(2406) can be taken as the luminance of Sc, and used to estimate the per-pixel chromaticities in a chroma network (2410) (e.g. a third neural network). For example, it is possible to define the chromaticity as color channel ratios using Equation 19 below:U = Scr / S V = S^ / S . (Equation 19)

[0241] Given that color channel ratios can be unbounded variables, it may be challenging to train neural networks with a direct loss. Therefore, it can be preferable to use a mapping of the values to the [0 - 1] range and to define the 2-channel target variable C using Equation 20 below:(Equation 20)

[0242] That is, the chromaticity values can be mapped to a domain of predefined range (such as between 0 and 1 ) using a mapping function. The mapping of the chromaticity values can be similar to that of the shading values. The chroma network can take the grayscale decomposition (Sg,Ag) and the input image as input, for example as a concatenated 7-channel input and estimate the 2-channel C (2412), which may be a color-component image or chromaticity map. This network may be trained using the standard mean-squared error and the multi-scale gradient loss commonly, for example, as described using Equations 21 and 22 below: 21 )(Equation 22) where C* may be the ground-truth color component image, andmay be the gradient of C at scale I.

[0243] The shading chromaticity estimation can require an understanding of the global context present in the scene. The shading chromaticity estimation can also be a low-frequency variable, which can make a low-resolution estimation viable. It is possible to utilize a convolutional architecture and estimate C (2412) at the receptivefield-size resolution. The low-resolution C (2412) can be combined with its luminance Sgto construct the RGB shading layerSc(2414), which can be a chromaticity adjusted shading image.

[0244] That is, the chroma network can process an input image, a shading image and an albedo image to generate a color component image by deriving chromaticity values of pixels in the second shading image. The color component image can depict chromaticity contributions to the shading of the input image and can be used to determine a chromaticity adjusted shading representing the illumination effect on the content depicted in the input image, for example, by the chroma network. Further, by augmenting the input shading image with the color component image, a shading image depicting the chromaticity dependent shading of the input image can be generated. Additionally, a chromaticity adjusted albedo image depicting the reflectance of the input image can be generated from the input image and the chromaticity dependent shading image by using Equation 1. Initial input albedo and shading images can be derived using the previously described processes or other methods. In particular, the input albedo and shading images may be estimates determined under Lambertian assumption or under greyscale models. The chromaticity values of the pixels in the color component image may be mapped using inverse chromaticity values that are mapped to a domain of 0 to 1 , which can be converted to the non-inverse values to generate the color component image.

[0245] Albedo Estimation

[0246] The albedo channel (estimation), when defined under the grayscale diffuse model, can contain strong color shifts coming from colored illumination. The colorized shading Sc(2414) from the chroma network can be used to compute an approximation to the correct albedo, Ac(2416) (e.g. a chromaticity adjusted albedo image), using the RGB diffuse model in Equation 18. It should be noted that Ac(2416) can still exhibit illumination-related artifacts, for example, due to the low-resolution chroma estimation and the lack of enforcement of sparse albedo values up to this point.

[0247] To estimate a final diffuse albedo layer, it is possible to define an albedo network (2418) (e.g. a fourth neural network) that can take Ac(2416) and Sc(2414), as input together with the input image (2402), for example concatenated to be a 9- channel input, which may be used to output / generate the diffuse albedo Ad(2420), which can be a diffuse illumination dependent albedo image. The global context on illumination color may provided in the input (e.g. from Acand Sc). As such, the albedo network can take advantage of the sparse nature of the albedo and can generate an accurate 3-channel albedo map. Similar to the chroma network, it is possible to use mean-squared error £mse(A') and the multi-scale gradient £msg(A) losses defined on the albedo to train this network. As shown in FIG. 26, this can result in a flat albedo layer (e.g. generated diffuse reflectance albedo image 2608) without illumination- related artifacts. In particular, FIG. 26 depicts an example input image 2602 and the initial greyscale image 2604. A color color-corrected albedo image 2606 and a diffuse albedo image 2608 generated using the input image 2602, the initial greyscale image 2604 according to an example embodiment of the present disclosure are also shown, as generated after processing by the chroma network 2418 and albedo network 2418, as described above. Highlight areas 2610, 2612, 2614, 2616 show various features of the images over the course of processing.

[0248] That is, the input image, and the chromaticity dependent shading image, and the chromaticity adjusted image may be processed by the albedo network to generate an albedo image depicting the diffuse reflectance of the input image. The albedo network may be trained to determine an adjusted albedo for representing diffuse reflectance of the content depicted in the input image. Further, a shading image can be generated from the input image and the diffuse albedo image using Equation 1 , which can be a shading image depicting a diffuse illumination adjusted shading of the input image.

[0249] Intrinsic decomposition methods can be trained with synthetic ground truth. Many synthetic intrinsic datasets readily provide the ground-truth albedo. Furthermore, real-world training data for albedo can be extracted from multiillumination datasets as described previously, and can greatly aid the in-the-wild generalization. The chroma and albedo networks, may be trained using syntheticdatasets. Further training on the chroma and albedo networks can be performed using multi-illumination datasets. These datasets can provide a good variety of images during training, and can allow the albedo estimation to generalize to in-the-wild scenes.

[0250] Diffuse Shading Estimation

[0251] By using the estimated diffuse albedo Ad(2420), it is possible to remove the Lambertian world assumption and estimate the colorful diffuse shading and nondiffuse illumination components in the intrinsic residual model in Equation 16. Diffuse shading may be correlated with the scene geometry; as such, a diffuse shading network (2424) (e.g. fifth neural network) can make use of the geometric cues in the scene to separate the diffuse effects from non-diffuse irradiance such as specularities and visible light sources. This problem can also be seen as the decomposition of Sc= I / Adin the RGB diffuse model in Equation 18 into diffuse and non-diffuse components. That is, Sc(2422) may be calculated from the input image (2402) and Ad(2420), where Scis a shading image for depicting a diffuse illumination adjusted shading.

[0252] The diffuse shading network can take the diffuse albedo Ad(2420), colorized shading from the diffuse model Sc(2422), and the input image (2402) as input, for example as a concatenated 9-channel input to output Sd(2426), which can be a shading image depicting the effect of diffuse illumination. The output may be defined in the inverse shading space, for example as a three-channel variable D = / Sd+ 1). The mean-squared error £mse(D) and the multi-scale gradientlosses can be used during training. It should be noted that the shading values may be mapped to a domain with pre-defined range, as described above.

[0253] Given the estimated diffuse shading Sd(2426) and albedo Ad(2420), it is possible to compute the residual non-diffuse layer (2428) using the intrinsic residual model in Equation 16, rearranged as: R = I - (Ad* Sd).

[0254] It should be noted that the estimated diffuse shading can be unbounded, and therefore the diffuse image (Ad* Sd) can exceed the input’s [0 - 1]range. This high-dynamic range property of the diffuse shading can enable various image enhancement applications as shown in FIG. 27. As a result of this property, the estimated residual (e.g. residual image) can have both negative and positive values. The positive part of the residual can contain non-diffuse illumination effects such as specularities and visible light sources, while the negative residual can show overexposed regions in the input image. For example, two residual images may be generated, one corresponding to the positive component and another to the negative component.

[0255] That is, the diffuse shading network can process the input image, the diffuse illumination adjusted shading image, and the diffuse albedo image to generate a shading image depicting the diffuse shading of the input image. The diffuse shading network may be trained to determine a diffuse shading representing a diffuse illumination effect on the content depicted in the input image. Further, at least one residual image can be generated from the input image, the diffuse shading image, and the diffuse albedo image, for example using Equation 16. For example, the at least one residual image can comprise a plurality of pixels represented using color channel values for depicting an effect of non-diffuse illumination on the content depicted in the input image. Additionally, one of the residual images may be a positive residual image comprising pixels of positive values corresponding to the non-diffuse illumination. One of the residual images may also be a negative residual image comprising pixels of negative values corresponding to overexposure. To generate the shading estimates, the diffuse shading network may be trained to process the images using inverse shading values.

[0256] FIG. 27 depicts an example input image 2702 and the initial greyscale image 2704. A color component image 2706, a chromaticity adjusted shading image 2708, a diffuse shading image 2710, positive residual image 2712 and negative residual image 2714 generated using the input image 2602 are shown. The initial greyscale image 2604 according to an example embodiment of the present disclosure is also shown, as generated after processing by the chroma network 2418, albedo network 2418, and diffuse shading network 2424 as described above. Highlighted areas 2702a 2702b show features of the images over the course of processing.

[0257] High-resolution synthetic datasets may be scarce for diffuse shading and lack diversity, while real-world datasets may be non-existent. As such, many other methods that focus on the residual model may have to limit their application scenario to specific object classes. The systems and methods of the present disclosure can use the diffuse shading network to obtain the albedo and Scas input, which can ease the contextual nature of its task. The diffuse network can be trained on synthetic datasets. However, the present systems and methods may nevertheless generalize to a wide range of in-the-wild images. This can show that by simplifying the task of each network, it is possible to utilize the generalizability of the disclosed albedo estimation pipeline to achieve in-the-wild non-diffuse intrinsic decomposition.

[0258] In some embodiments, an encoder-decoder architecture, which can be useful for various mid-level vision tasks, may be used for the disclosed networks. A sigmoid activation may be used to output quantities strictly in the [0 - 1] range. The networks may be trained using the Adam optimizer with a learning rate of 10“5.

[0259] FIG. 25 depicts an example input image 2502 along with the corresponding (diffuse) albedo image 2504, (diffuse) shading 2506, and non-diffuse residual image (2508) by an example embodiment of the process as described above. Further, FIG. 28 depicts an input image 2802 with a region 2812 that the greyscale albedo image 2804 does not represent correctly. While using the inaccurate albedo estimation can lead to the generation of inaccurate final albedo image 2806, if the initial albedo estimation is corrected in a corrected image 2808, the albedo image 2810 generated by an example embodiment using the corrected image 2808 may be accurate.

[0260] It should be noted that the below evaluation section describes the performance of an example embodiment of the present disclosure in comparison to other methods.

[0261] Evaluation

[0262] Quantitative Evaluation

[0263] The evaluation described herein may be focused on albedo estimation due to the lack of ground truth benchmarks.

[0264] Dataset 4

[0265] Dataset 4 may be used to measure real-world albedo accuracy in terms of intensity and color. The dataset consists of -850 indoor images and measured albedo within specific masked regions in the image. The albedo can be measured using a known gray card placed on areas of homogeneous albedo. The evaluation may use two metrics that measure the accuracy of albedo intensity and chromaticity, respectively. The results are reported in T able 6 (below). As shown by the discrepancy between the intensity and chromaticity scores of the other methods, the grayscale shading assumption can result in large discrepancies in the color of the estimated albedo. The initial shading chroma estimation of the example embodiment may already be able to compensate for these color shifts and scores the second-best in all metrics. The final refined albedo estimation of the example embodiment can further improve the results, and outperforms all other methods in terms of both intensity and chromaticity.Table 6: Performance Comparison of Albedo accuracy in Dataset 4

[0266] Dataset 5

[0267] To quantify the generalization abilities of each method to out-of- distribution scenes, the albedo estimation can be evaluated on dataset 5. The dataset consists of about 50 rendered scenes, from various sources. The dataset can be augmented with scenes from other datasets, modified to removed duplicated images and to ensure that each scene is equally represented in the dataset. Metrics on the albedo are computed, similar to those described above. The results are reported in Table 7 (below), with similar conclusions to Table 6.Table 7: Performance Comparison of Albedo accuracy in Dataset 5

[0268] Qualitative Evaluation

[0269] FIG. 30 show example results by other methods that adopt the grayscale intrinsic diffuse model or the RGB intrinsic model in their unsupervised formulation. In particular, FIG. 30 depicts input image 3002, the albedo image 3006 and the shading image 3010 generated by an example embodiment as well as albedo images (3004) and shading images (3008) generated by other methods. As depicted, when the grayscale model is enforced on the albedo-shading pair, the color of secondary illuminations can create color shifts in the albedo, as some results from the other methods show. This color cast may be removed in some of the other methodsthat adopts the RGB intrinsic model. However, since all of the other methods still work within the grayscale model, their intrinsic components can fail to faithfully reconstruct the image. For example, a strong color cast and residual albedo colors in the shading may be produced by other methods, while the example embodiment can use the intrinsic residual model to estimate a clean albedo with colors of the secondary illuminations represented in the colorful shading.

[0270] In-the-wild comparisons against other methods are also shown in FIG. 29. In particular, FIG. 29 depicts input images 2902, the albedo images 2904 generated by an example embodiment as well as albedo images generated by other methods (2906). FIG. 29 also depicts highlighted areas 2908, 2910, 2912, 2914, 2916 showing various features in the images. Some of the other methods can suffer from low resolution in in-the-wild scenes, while in indoor scenes they may be sometimes susceptible to a string tiling effect due to their high-resolution refinement. Some of the other methods can struggle in out-of-distribution scenes and may generate a low- resolution result due to the averaging of their results.

[0271] The refinement and averaging strategies adopted by some of the other methods can result in > 10 second run times, while the example embodiment can take around a second on average to generate a high-resolution result. Accordingly, a technical effect of the present systems and methods can be the improvement in computational efficiency and speed.

[0272] Further, some of the other methods can generate sharp results but suffers from typical diffusion-based generation artifacts around text and may cause cartoonization of human faces. The example embodiment may employ analytical modeling of the problem and can remain faithful to the input image and generalize to out-of-distribution images effectively.

[0273] Applications

[0274] The intrinsic residual model can allow for several computational photography applications by estimating a color component for the shading and separating diffuse and non-diffuse illumination effects. As demonstrated in FIGS. 32 and 33, specularities in an image can be removed by computing the diffuse imageAd* Sd. Estimating the shading in color can allow for per-pixel multi-illuminant white balancing, as shown in FIG. 32. In particular, FIG. 32 depicts an input image 3202 used for the generation of a diffuse image 3206 by combining the albedo and shading estimations generated by an example embodiment as well as the removed specularity image 3204 with a highlighted area 3208 showing the specularities. FIG. 32 also depicts input images 3210 for use in white-balancing along with the generated white balanced images 3212 and an example illumination chroma 3216. Similarly, FIG. 33 depicts input images 3302 used for the generation of diffuse images 3204 by combining the albedo and shading estimations generated by an example embodiment as well as the removed specularity image 3206 with a highlighted areas 3208, 3310 showing the specularities being removed.

[0275] The unbounded estimation of the diffuse shading may enable the recovery of details that are lost to clipping in the input image, as demonstrated in FIG. 31 , which depicts original images 3102 and recovered highlight images 3104, which may be produced by using of an example embodiment to generate residual image(s) (e.g. positive residual image 3106 and negative residual image 3108).

[0276] It would be appreciated by one of ordinary skill in the art that the system and components shown in the figures may include components not shown in the drawings. For simplicity and clarity of the illustration, elements in the figures are not necessarily to scale and are only schematic. It will be apparent to persons skilled in the art that a number of variations and modifications can be made without departing from the scope of the invention as described herein.

[0277] It is contemplated that any part of any aspect or embodiment discussed in this specification can be implemented or combined with any part of any other aspect or embodiment discussed in this specification, so long as such those parts are not mutually exclusive with each other.

[0278] It should be recognized that features and aspects of the various examples provided above can be combined into further examples that also fall within the scope of the present disclosure.

[0279] When used in this specification and claims, the terms "comprises" and "comprising" and variations thereof mean that the specified features, steps or integers are included. The terms are not to be interpreted to exclude the presence of other features, steps or components. Additionally, the term "connect" and variants of it such as "connected", "connects", and "connecting" as used in this description are intended to include indirect and direct connections unless otherwise indicated. For example, if a first device is connected to a second device, that coupling may be through a direct connection or through an indirect connection via other devices and connections. Similarly, if the first device is communicatively connected to the second device, communication may be through a direct connection or through an indirect connection via other devices and connections. Further, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0280] The embodiments have been described above with reference to flow, sequence, and block diagrams of methods, apparatuses, systems, and computer program products. In this regard, the depicted flow, sequence, and block diagrams illustrate the architecture, functionality, and operation of implementations of various embodiments. For instance, each block of the flow and block diagrams and operation in the sequence diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified action(s). In some alternative embodiments, the action(s) noted in that block or operation may occur out of the order noted in those figures. For example, two blocks or operations shown in succession may, in some embodiments, be executed substantially concurrently, or the blocks or operations may sometimes be executed in the reverse order, depending upon the functionality involved. Some specific examples of the foregoing have been noted above butthose noted examples are not necessarily the only examples. Each block of the flow and block diagrams and operation of the sequence diagrams, and combinations of those blocks and operations, may be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0281] Use of language such as "at least one of X, Y, and Z," "at least one of X, Y, or Z," "at least one or more of X, Y, and Z," "at least one or more of X, Y, and / or Z," or "at least one of X, Y, and / or Z," is intended to be inclusive of both a single item (e.g., just X, or just Y, or just Z) and multiple items (e.g., {X and Y}, {X and Z}, {Y and Z}, or {X, Y, and Z}). The phrase "at least one of" and similar phrases are not intended to convey a requirement that each possible item must be present, although each possible item may be present.

[0282] The invention may also broadly consist in the parts, elements, steps, examples and / or features referred to or indicated in the specification individually or collectively in any and all combinations of two or more said parts, elements, steps, examples and / or features. In particular, one or more features in any of the embodiments described herein may be combined with one or more features from any other embodiment(s) described herein.

Claims

CLAIMS:1 . A method of image decomposition comprising: processing an input image with a first neural network trained to determine shading representing an illumination effect on content depicted in the input image; and generating, with the first neural network, at least one first shading image from the input image, the at least one first shading image depicting the shading of the input image, wherein the input image comprises a plurality of pixels represented using color channel values and the at least one first shading image comprises a plurality of pixels represented using numerical shading values; and wherein the first neural network is trained to determine the shading by minimizing loss based on minimizing differences between a relative shading order of the pixels in the at least one first shading image and a relative shading order of pixels in the shading of the input image.

2. The method of claim 1 , wherein the shading values are derived by the first neural network using mapped shading values, each mapped shading value being mapped from a shading value associated with each pixel that is representative of the illumination effect.

3. The method of claim 1 , wherein the shading values are mapped to a first domain of pre-defined range.

4. The method of claim 1 , wherein the at least one first shading image comprises:a low resolution shading image with a resolution that is within a receptive field of the first neural network, the low resolution shading image providing global context of the shading, and a high resolution shading image with a resolution that that exceeds the receptive field of the first neural network, the high resolution shading image providing local context of the shading.

5. The method of claim 4, further comprising: processing the input image, the low resolution shading image, and the high resolution shading image with a second neural network trained to further determine the shading of the input image; generating, with the second neural network, a second shading image from the input image, the low resolution shading image, and the high resolution shading image, the second shading image being a greyscale shading image comprising a plurality of pixels represented using the numerical shading values for depicting the shading of the input image; and generating a first albedo image from the input image and the second shading image, the first albedo image comprising a plurality of pixels represented using the color channel values for depicting reflectance of the content in the input image.

6. The method of claim 5, wherein the first albedo image represents Lambertian reflectance and is generated based on a relationship that the product of the first albedo image and the second shading image is the input image.

7. The method of claim 6, wherein the input image is a 3-channel RGB image comprising a photograph or photorealistic image; wherein each of the at least one first shading image and the second shading image is a single-channel greyscale image; and wherein the first albedo image is a 3-channel RGB image.

8. The method of claim 5, further comprising: obtaining a first set of training data comprising synthetic images and at least one training shading image corresponding to each synthetic image comprising: a high resolution training shading image and a low resolution training shading image; training the first neural network to generate the at least one first shading image corresponding to each synthetic image based on the first set of training data; obtaining a second set of training data comprising synthetic images, high resolution training shading images corresponding to the synthetic images, low resolution training shading images corresponding to the synthetic images, and training shading images corresponding to the synthetic images, the low resolution training shading images, and the high resolution shading images; and training the second neural network to generate the second shading image corresponding to each synthetic image based on the second set of training data.

9. The method of claim 6, further comprising: obtaining a third set of training data comprising photographs and at least one training shading image corresponding to each photograph comprising: a high resolution training shading image and a low resolution training shading image; training the first neural network to generate the at least one first shading image corresponding to each photograph based on the third set of training data; obtaining a fourth set of training data comprising photographs, high resolution training shading images corresponding to the photographs, low resolution training shading images corresponding to the photographs, and training shading images corresponding to the photographs, the lowresolution training shading images, and the high resolution shading images; and training the second neural network to generate the second shading image corresponding to each photograph based on the fourth set of training data.

10. The method of claim 9, further comprising: generating training data comprising photographs by: obtaining a plurality of photographs for a scene under different lighting conditions; generating a plurality of albedo images corresponding to the plurality of photographs with the first neural network and the second neural network; generating a predicted albedo image by performing statistical analysis on the plurality of albedo images; generating a plurality of predicted shading images by dividing each of the plurality of photographs by the predicted albedo image; and compiling the plurality of photographs and the plurality of predicted shading images into the training data.11 . The method of claim 9, wherein training of the first neural network comprises use of a loss function defining a relationship between ground truth and a monotonically increasing or monotonically decreasing function comprising shading generated by the first neural network and in which loss determined by the loss function is indicative of the differences between the relative shading order of the pixels of the at least one first shading image and the relative shading order of the shading of the input image.

12. The method of claim 11 , wherein the training of the first neural network further comprises use of a multi-gradient loss function to evaluate estimated shading and the differences between the relative shading order of the pixels of the atleast one first shading image and the relative shading order of the shading of the input image.

13. The method of claim 9, wherein training of the second neural network comprises use of the low resolution training shading images for scale.

14. The method of claim 13, further comprising: processing the input image, the second shading image and the first albedo image with a third neural network trained to determine a chromaticity adjusted shading representing the illumination effect on the content depicted in the input image; and generating, with the third neural network, a color component image by deriving chromaticity values of pixels in the second shading image, the color component image depicting chromaticity contributions to the shading of the input image; generating a third shading image by augmenting the second shading image with the color component image, the third shading image being a shading image depicting the chromaticity dependent shading of the input image; and generating a second albedo image from the input image and the third shading image, the second albedo image being a chromaticity adjusted albedo image depicting the reflectance of the input image.

15. The method of claim 14, wherein the second albedo image is generated based on a relationship where the product of the second albedo image and the third shading image is the input image.

16. The method of claim 14, wherein the chromaticity values of the pixels are defined as channel variables U and V where U = Sc / S and V = Sc / S .

17. The method of claim 14, wherein the chromaticity values are mapped to a second domain of predefined range.

18. The method of claim 17, further comprising: processing the input image, the third shading image, and the second albedo image with a fourth neural network trained to determine an adjusted albedo for representing diffuse reflectance of the content depicted in the input image; generating, with the fourth neural network, a third albedo image from the input image, the third shading image, and the second albedo image, the third albedo image being an albedo image depicting the diffuse reflectance of the input image; and generating a fourth shading image from the input image and the third albedo image, the fourth shading image being a shading image depicting a diffuse illumination adjusted shading of the input image.

19. The method of claim 18, wherein the fourth shading image is generated based on a relationship where the product of the third albedo image and the fourth shading image is the input image.

20. The method of claim 19, further comprising: processing the input image, the fourth shading image, and the third albedo image with a fifth neural network trained to determine a diffuse shading representing a diffuse illumination effect on the content depicted in the input image; generating, with the fifth neural network, a fifth shading image from the input image, the fourth shading image, and the third albedo image, the fifth shading image being a shading image depicting the diffuse shading of the input image; and generating at least one residual image from the input image, the fourth shading image, and the third albedo image, the at least one residual image comprising a plurality of pixels represented using color channel values for depicting an effect of non-diffuse illumination on the content depicted in the input image.21 . The method of claim 20, wherein the fifth shading image is generated based on a relationship where the product of the third albedo image and the fifth shading image is the input image.

22. The method of claim 20, wherein the fifth neural network is trained to derive shading values of the fifth shading image by mapping the shading values of the fifth shading image to a third domain of pre-defined range.

23. The method of claim 20, wherein the at least one residual image is generated based on a relationship that an addition between the at least one residual image and a product of the fifth shading image and the third albedo image is the input image.

24. The method of claim 18, wherein the at least one residual image comprises: a positive residual image comprising pixels of positive values corresponding to the non-diffuse illumination, and a negative residual image comprising pixels of negative values corresponding to overexposure.

25. The method of claim 21 , further comprising: obtaining a fifth set of train data comprising synthetic images, training shading images corresponding to the synthetic images, training albedo images corresponding to the synthetic images, and training color component images corresponding to the synthetic images; training the third neural network to generate the color component image corresponding to each synthetic image based on the fifth set of training data; obtaining a sixth set of training data comprising photographs, training shading images corresponding to the photographs, training albedo images corresponding to the photographs, and training color component images corresponding to the photographs; andtraining the third neural network to generate the color component image corresponding to each photograph based on the sixth set of training data.

26. The method of claim 22, further comprising: obtaining a seventh set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the synthetic images; training the fourth neural network to generate the third albedo image corresponding to each synthetic image based on the seventh set of training data; obtaining an eighth set of training data comprising photographs, training shading images corresponding to the photographs, and training albedo images corresponding to the photographs; and training the fourth neural network to generate the third albedo image corresponding to each photograph based on the eighth set of training data.

27. The method of claim 23, further comprising: obtaining a ninth set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the synthetic images; and training the fifth neural network to generate the fifth shading image corresponding to each synthetic image based on the ninth set of training data.

28. The method of claim 22, wherein each of the third shading image, the fourth shading image, the fifth shading image, the second albedo image, the third albedo image, and the at least one residual image is a 3-channel RGB image.

29. A method for training one or more neural networks for performing image decomposition, comprising:obtaining a first set of training data comprising synthetic images and at least one training shading image corresponding to each synthetic image, the at least one training shading image comprising: a high resolution training shading image and a low resolution training shading image; training a first neural network to generate at least one first shading image corresponding to each synthetic image based on the first set of training data, the at least one first shading image depicting shading representing an illumination effect and comprises a plurality of pixels represented using numerical shading values; training the first neural network to determine the shading by minimizing loss based on minimizing differences between a relative shading order of the pixels in the at least one first shading image and a relative shading order of pixels in the shading of an input image; and training the first neural network to generate the at least one first shading image from the input image.

30. The method of claim 29, further comprising; training the first neural network to derive the shading values using mapped shading values, each mapped shading value being mapped from a shading value associated with each pixel that is representative of the illumination effect.

31. The method of claim 29, wherein the shading values are mapped to a first domain of pre-defined range.

32. The method of claim 29, wherein the at least one first shading image comprises: a low resolution shading image with a resolution that is within a receptive field of the first neural network, the low resolution shading image providing global context of the shading, anda high resolution shading image with a resolution that that exceeds the receptive field of the first neural network, the high resolution shading image providing local context of the shading.

33. The method of 32, further comprising: obtaining a second set of training data comprising synthetic images, high resolution training shading images corresponding to the synthetic images, low resolution training shading images corresponding to the synthetic images, and training shading images corresponding to the synthetic images, the low resolution training shading images, and the high resolution shading images; training a second neural network to generate a second shading image corresponding to each synthetic image based on the second set of training data, the second shading image being a greyscale shading image comprising a plurality of pixels represented using the numerical shading values for depicting the shading; and training the second neural network to generate the second shading image from the input image and the at least one first shading image.

34. The method of claim 33, further comprising: obtaining a third set of training data comprising photographs and at least one training shading image corresponding to each photograph comprising: a high resolution training shading image and a low resolution training shading image; training the first neural network to generate the at least one first shading image corresponding to each photograph based on the third set of training data; obtaining a fourth set of training data comprising photographs, high resolution training shading images corresponding to the photographs, low resolution training shading images corresponding to the photographs, and training shading images corresponding to the photographs, the lowresolution training shading images, and the high resolution shading images; and training the second neural network to generate the second shading image corresponding to each photograph based on the fourth set of training data.

35. The method of claim 34, further comprising: generating training data comprising photographs by: obtaining a plurality of photographs for a scene under different lighting conditions; generating a plurality of albedo images corresponding to the plurality of photographs with the first neural network and the second neural network; generating a predicted albedo image by performing statistical analysis on the plurality of albedo images; generating a plurality of predicted shading images by dividing each of the plurality of photographs by the predicted albedo image; and compiling the plurality of photographs and the plurality of predicted shading images into the training data.

36. The method of claim 29, further comprising: training the first neural network using a loss function defining a relationship between ground truth and a monotonically increasing or monotonically decreasing function comprising shading generated by the first neural network and in which loss determined by the loss function is indicative of the differences between the relative shading order of the pixels of the at least one first shading image and the relative shading order of the shading of the input image.

37. The method of claim 36, further comprising:training the first neural network using a multi-gradient loss function to evaluate estimated shading and the differences between the relative shading order of the pixels of the at least one first shading image and the relative shading order of the shading of the input image.

38. The method of claim 34, wherein training data for the second neural network further comprises training albedo images depicting reflectance.

39. The method of claim 38, further comprising: training the second neural network using the low resolution training shading images and the training albedo images as a reference scale.

40. The method of claim 39, further comprising: generating a first albedo image for each second shading image based on a relationship where the product of the first albedo image and the second shading image is the input image, the first albedo image comprising a plurality of pixels represented using the color channel values for depicting reflectance; and training the second neural network using a mean-squared error loss function and a multi-scale gradient loss function for evaluating estimated shading and estimated albedo.41 . The method of claim 40, further comprising: obtaining a fifth set of train data comprising synthetic images, training shading images corresponding to the synthetic images, training albedo images corresponding to the synthetic images, and training color component images corresponding to the synthetic images; training a third neural network to generate a color component image corresponding to each synthetic image based on the fifth set of training data, the color component image depicting chromaticity contributions to the shading; obtaining a sixth set of training data comprising photographs, training shading images corresponding to the photographs, training albedo imagescorresponding to the photographs, and training color component images corresponding to the photographs; training the third neural network to generate the color component image corresponding to each photograph based on the sixth set of training data; and training the third neural network to generate the color component image by determining a chromaticity dependent shading representing the illumination effect by deriving chromaticity values of the pixels in the input image.

42. The method of claim 41 , further comprising: training the third neural network to derive the chromaticity values of the pixels by defining the chromaticity values as channel variables U and V where U = Scr / Sl and V = S / Sl , and wherein the chromaticity values are mapped to a second domain of predefined range.

43. The method of claim 42, wherein the color component image is for generating a third shading image, the third shading image being a shading image depicting a chromaticity adjusted shading of the input image and generated by augmenting the second shading image with the color component image; and wherein the third shading image is for generating a second albedo image, the second albedo image being a chromaticity adjusted albedo image depicting the reflectance and generated from the input image and the third shading image.

44. The method of claim 42, further comprising: obtaining a seventh set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the synthetic images;training a fourth neural network to generate a third albedo image corresponding to each synthetic image based on the seventh set of training data, the third albedo image being an albedo image depicting a diffuse reflectance; obtaining an eighth set of training data comprising photographs, training shading images corresponding to the photographs, and training albedo images corresponding to the photographs; training the fourth neural network to generate the third albedo image corresponding to each photograph based on the eighth set of training data; and training the fourth neural network to generate the third albedo image from the input image, the third shading image, and the second albedo image by determining an adjusted albedo for representing the diffuse reflectance.

45. The method of claim 44, wherein the third albedo image is for generating a fourth shading image, the fourth shading image being a shading image depicting a diffuse illumination adjusted shading and generated from the input image and the third albedo image.

46. The method of claim 45, further comprising: obtaining a ninth set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the synthetic images; training a fifth neural network to generate a fifth shading image corresponding to each synthetic image based on the ninth set of training data, the fifth shading image being a shading image depicting a diffuse shading; and training the fifth neural network to generate the fifth shading image from the input image, the fourth shading image, and the third albedo image by determining the diffuse shading representing a diffuse illumination effect.

47. The method of claim 46, wherein the fifth shading image is for generating at least one residual image comprising a plurality of pixels represented using color channel values for depicting an effect of non-diffuse illumination, the at least one residual image generated from the input image, the fourth shading image, and the third albedo image.

48. The method of claim 46, further comprising: training the fifth neural network to derive shading values of the fifth shading image by mapping the shading values of the fifth shading image to a third domain of pre-defined range.

49. A method of image decomposition comprising: processing an input image with a first neural network trained to determine shading representing an illumination effect on content depicted in the input image; and generating, with the first neural network, at least one first shading image from the input image, the at least one first shading image depicting the shading of the input image, wherein the input image comprises a plurality of pixels represented using color channel values and the at least one first shading image comprises a plurality of pixels represented using numerical shading values; and wherein the first neural network is trained to derive the shading values by mapping the shading values to a first domain of pre-defined range.

50. The method of claim 49, wherein the first neural network is trained to derive the shading values by using mapped shading values, each mapped shading value being mapped from a shading value associated with each pixel that is representative of the illumination effect.51 . The method of claim 50,wherein the mapped shading values are calculated as D=1 / (S+1), where S is a shading value and D is a corresponding mapped shading value.

52. The method of claim 49, wherein the first domain has a range of 0 to 1 .

53. The method of claim 50, wherein the first neural network is trained to determine the shading by minimizing loss based on minimizing differences between a relative shading order of the pixels in the at least one first shading image and a relative shading order of pixels in the shading of the input image.

54. A method of generating training images comprising photographs for training neural networks to perform image decomposition, the method comprising: obtaining a plurality of photographs for a scene under different lighting conditions, each photograph comprising a component corresponding to shading that depicts an illumination effect on the scene and a component corresponding to albedo that depicts reflectance on the scene; generating a plurality of albedo images corresponding to the plurality of photographs, each albedo image comprising pixels depicting an estimated albedo of a corresponding photograph; generating a predicted albedo image by performing statistical analysis on the plurality of albedo images; generating a plurality of predicted shading images by dividing each of the plurality of photographs by the predicted albedo image, each predicted shading image comprising pixels depicting an estimated shading of a corresponding photograph; and compiling the plurality of photographs, the predicted albedo image, and the plurality of predicted shading images into network training data.

55. The method of claim 54, wherein the predicted shading images are for use as ground truth shading and the predicted albedo image is for use as ground truth albedo.

56. The method of claim 54, wherein the plurality of albedo images are generated by one or more neural networks trained to perform image decomposition.

57. The method of claim 54, wherein the plurality of albedo images are generated by a first neural network and a second neural network by: processing the plurality of photographs with the first neural network trained to determine shading representing an illumination effect on content depicted in an input image; and generating, with the first neural network from the plurality of photographs, a plurality of first shading images depicting the shading comprising a plurality of low resolution shading images with a resolution that is within a receptive field of the first neural network and a plurality of high resolution shading images with a resolution that that exceeds the receptive field of the first neural network, wherein the shading is represented using numerical shading values; and wherein the first neural network is trained to determine the shading by minimizing loss based on minimizing differences between a relative shading order of the pixels in the each of the shading images and a relative shading order of pixels in the shading of the input image, processing the plurality of photographs and the plurality of first shading images with a second neural network trained to further determine the shading of the input image; generating, with the second neural network, a plurality of second shading images depicting the shading from the plurality of photographs and the plurality of first shading images; and generating the plurality of albedo images from the plurality of photographs and the plurality of second shading images by dividing each photograph with a corresponding second shading image.

58. The method of claim 57, wherein the shading values are derived by the first neural network using mapped shading values, each mapped shading value being mapped from a shading value associated with each pixel that is representative of the illumination effect.

59. The method of claim 57, wherein the shading values are mapped to a domain of pre-defined range.

60. The method of claim 57, further comprising: obtaining a first set of training data comprising synthetic images and at least one training shading image corresponding to each synthetic image comprising: a high resolution training shading image and a low resolution training shading image; training the first neural network to generate at least one first shading image corresponding to each synthetic image based on the first set of training data; obtaining a second set of training data comprising synthetic images, high resolution training shading images corresponding to the synthetic images, low resolution training shading images corresponding to the synthetic images, and training shading images corresponding to the synthetic images, the low resolution training shading images, and the high resolution shading images; training the second neural network to generate a second shading image corresponding to each synthetic image based on the second set of training data; obtaining a third set of training data comprising photographs and at least one training shading image corresponding to each photograph comprising: a high resolution training shading image and a low resolution training shading image; training the first neural network to generate the at least one first shading image corresponding to each photograph based on the third set of training data;obtaining a fourth set of training data comprising photographs, high resolution training shading images corresponding to the photographs, low resolution training shading images corresponding to the photographs, and training shading images corresponding to the photographs, the low resolution training shading images, and the high resolution shading images; and training the second neural network to generate the second shading image corresponding to each photograph based on the fourth set of training data.61 . The method of claim 60, wherein training of the first neural network comprises use of a loss function defining a relationship between ground truth and a monotonically increasing or monotonically decreasing function comprising shading generated by the first neural network and in which loss determined by the loss function is indicative of the differences between the relative shading order of the pixels of the at least one first shading image and the relative shading order of the shading of the input image.

62. The method of claim 61 , wherein the training of the first neural network further comprises use of a multi-gradient loss function to evaluate estimated shading and the differences between the relative shading order of the pixels of the at least one first shading image and the relative shading order of the shading of the input image.

63. The method of claim 60, wherein training of the second neural network comprises use of the low resolution training shading images for scale.

64. The method of claim 54, wherein the statistical analysis comprises taking the per-pixel median of the plurality of albedo images.

65. The method of claim 54, further comprising: training at least one neural network to perform image decomposition using the network training data.

66. A method of image decomposition comprising:processing an input image, a first shading image and a first albedo image with a chroma neural network trained to determine a chromaticity adjusted shading representing an illumination effect on content depicted in the input image, each input image comprising a diffuse shading component corresponding to shading that depicts an illumination effect on the content, a diffuse albedo component corresponding to albedo that depicts reflectance on the content, and a residual component corresponding to a non-diffuse illumination effect on the content; and generating, with the chroma neural network, a color component image by deriving chromaticity values of pixels in the shading image, the color component image depicting chromaticity contributions to the shading of the input image; generating a second shading image by augmenting the first shading image with the color component image, the second shading image being a shading image depicting the chromaticity dependent shading of the input image; generating a second albedo image from the input image and the second shading image, the second albedo image being a chromaticity adjusted albedo image depicting the reflectance of the content; processing the input image, the second shading image, and the second albedo image with an albedo neural network trained to determine the diffuse albedo component; generating, with the albedo neural network, a third albedo image from the input image, the second shading image, and the second albedo image, the third albedo image being an albedo image depicting the diffuse albedo component; generating a third shading image from the input image and the third albedo image, the third shading image being a shading image depicting a diffuse illumination adjusted shading of the input image;processing the input image, the third shading image, and the third albedo image with a diffuse shading neural network trained to determine the diffuse shading component; generating, with the diffuse shading neural network, a fourth shading image from the input image, the third shading image, and the third albedo image, the fourth shading image being a shading image depicting the diffuse shading component; and generating at least one residual image from the input image, the third shading image, and the third albedo image, the at least one residual image comprising a plurality of pixels represented using color channel values for depicting an effect of non-diffuse illumination on the content depicted in the input image.

67. The method of claim 66, wherein the second albedo image is generated based on a relationship where the product of the second albedo image and the second shading image is the input image.

68. The method of claim 66, wherein the chromaticity values of the pixels are defined as channel variables U and V where U = Sc / S and V = Sc / S .

69. The method of claim 66, wherein the chromaticity values are mapped to a first domain of pre-defined range.

70. The method of claim 66, wherein the third shading image is generated based on a relationship where the product of the third albedo image and the third shading image is the input image.71 . The method of claim 66, wherein the fourth shading image is generated based on a relationship where the product of the third albedo image and the fourth shading image is the input image.

72. The method of claim 66, wherein the diffuse shading neural network is trained to derive shading values of the fourth shading image using mapped shadingvalues, each mapped shading value being mapped from a shading value associated with each pixel that is representative of the illumination effect.

73. The method of claim 66, wherein the diffuse shading neural network is trained to derive shading values of the fourth shading image by mapping the shading values of the fourth shading image to a second domain of pre-defined range, the shading values representative of the illumination effect.

74. The method of claim 66, wherein the at least one residual image is generated based on a relationship that an addition between the at least one residual image and a product of the fourth shading image and the third albedo image is the input image.

75. The method of claim 74, wherein the at least one residual image comprises: a positive residual image comprising pixels of positive values corresponding to the non-diffuse illumination, and a negative residual image comprising pixels of negative values corresponding to overexposure.

76. The method of claim 66, further comprising: obtaining a first set of train data comprising synthetic images, training shading images corresponding to the synthetic images, training albedo images corresponding to the synthetic images, and training color component images corresponding to the synthetic images; training the chroma neural network to generate the color component image corresponding to each synthetic image based on the first set of training data; obtaining a second set of training data comprising photographs, training shading images corresponding to the photographs, training albedo images corresponding to the photographs, and training color component images corresponding to the photographs; andtraining the chroma neural network to generate the color component image corresponding to each photograph based on the second set of training data.

77. The method of claim 76, further comprising: obtaining a third set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the synthetic images; training the albedo neural network to generate the third albedo image corresponding to each synthetic image based on the third set of training data; obtaining a fourth set of training data comprising photographs, training shading images corresponding to the photographs, and training albedo images corresponding to the photographs; and training the albedo neural network to generate the third albedo image corresponding to each photograph based on the fourth set of training data.

78. The method of claim 77, further comprising: obtaining a fifth set of training data comprising synthetic images, training shading images corresponding to the synthetic images, and training albedo images corresponding to the synthetic images; and training the diffuse shading neural network to generate the fourth shading image corresponding to each synthetic image based on the fifth set of training data.

79. The method of claim 78, further comprising: generating network training data comprising photographs by: obtaining a plurality of photographs for a scene under different lighting conditions;generating a plurality of albedo images corresponding to the plurality of photographs; generating a predicted albedo image by performing statistical analysis on the plurality of albedo images; generating a plurality of predicted shading images by dividing each of the plurality of photographs by the predicted albedo image; and compiling the plurality of photographs, the predicted albedo image, and the plurality of predicted shading images into the network training data.

80. The method of claim 66, wherein each of the second shading image, the third shading image, the fourth shading image, the second albedo image, the third albedo image, and the at least one residual image is a 3-channel RGB image.

81. A method for training one or more neural networks for performing image decomposition, comprising: obtaining a first set of training data comprising synthetic images and at least one training shading image corresponding to each synthetic image, the at least one training shading image comprising: a high resolution training shading image and a low resolution training shading image; training a first neural network to generate at least one first shading image corresponding to each synthetic image based on the first set of training data, the at least one first shading image depicting shading representing an illumination effect and comprises a plurality of pixels represented using numerical shading values; and training the first neural network to derive the shading values by mapping the shading values to a first domain of pre-defined range.

82. The method of claim 81 , further comprising: training the first neural network to determine the shading by minimizing loss based on minimizing differences between a relative shading order of thepixels in the at least one first shading image and a relative shading order of pixels in the shading of an input image.

83. The method of claim 81 , further comprising: training the first neural network to derive the shading values using mapped shading values, each mapped shading value being mapped from a shading value associated with each pixel that is representative of the illumination effect.

84. The method of claim 81 , wherein the inverse shading values are calculated as D=1 / (S+1), where S is a shading value and D is a corresponding mapped shading value.

85. The method of claim 81 , wherein the first domain of the shading values has a range of 0 to 1 .

86. A method for training one or more neural networks for performing image decomposition, comprising: obtaining a first set of train data comprising training input images, training shading images corresponding to the training input images, training albedo images corresponding to the training input images, and training color component images corresponding to the training input images; training a chroma neural network to generate a color component image corresponding to each training input image based on the first set of training data, the color component image depicting chromaticity contributions to the shading; training the chroma neural network to generate the color component image by determining a chromaticity adjusted shading representing an illumination effect on content depicted in an input image, each input image comprising a diffuse shading component corresponding to shading that depicts an illumination effect on content depicted in the input image, a diffuse albedo component corresponding to albedo thatdepicts reflectance on the content, and a residual component corresponding to a non-diffuse illumination effect on the content; obtaining a second set of training data comprising training input images, training shading images corresponding to the training input images, and training albedo images corresponding to the training input images; training an albedo neural network to generate a third albedo image corresponding to each training input image based on the second set of training data, the third albedo image being an albedo image depicting the diffuse albedo component; training the albedo neural network to generate the third albedo image by determining the diffuse albedo component; obtaining a third set of training data comprising training input images, training shading images corresponding to the training input images, and training albedo images corresponding to the training input images; training a diffuse shading neural network to generate a fourth shading image corresponding to each training input image based on the third set of training data, the fourth shading image being a shading image depicting the diffuse shading component; and training the fifth neural network to generate the fourth shading image by determining the diffuse shading component.

87. The method of claim 86, further comprising: obtaining a fourth set of training data comprising photographs, training shading images corresponding to the photographs, training albedo images corresponding to the photographs, and training color component images corresponding to the photographs; training the chroma neural network to generate the color component image corresponding to each photograph based on the fourth set of training data;obtaining a fifth set of training data comprising photographs, training shading images corresponding to the photographs, and training albedo images corresponding to the photographs; and training the albedo neural network to generate the third albedo image corresponding to each photograph based on the fifth set of training data.

88. The method of claim 86, further comprising: training the chroma neural network to derive the chromaticity values of the pixels by defining the chromaticity values as channel variables U and V where U = S^ / S^ and V = S / S , and wherein the chromaticity values are mapped to a first domain of pre-defined range.

89. The method of claim 86, wherein the color component image is for generating a second shading image, the second shading image being a shading image depicting the chromaticity dependent shading of the input image and generated by augmenting the first shading image with the color component image; wherein the second shading image is for generating a second albedo image, the second albedo image being a chromaticity adjusted albedo image depicting the reflectance of the content and generated from the input image and the second shading image; and wherein the third albedo image is for generating a third shading image, the third shading image being a shading image depicting a diffuse illumination adjusted shading of the input image and generated from the input image and the third albedo image.

90. The method of claim 86, further comprising: training the albedo neural network to generate the third albedo image from the input image, the third shading image, and the second albedo image.Ill91 . The method of claim 86, further comprising: training the diffuse shading neural network to generate the fourth shading image from the input image, the third shading image, and the third albedo image.

92. The method of claim 86, wherein the fourth shading image is for generating at least one residual image comprising a plurality of pixels represented using color channel values for depicting an effect of non-diffuse illumination on the content depicted in the input image, the at least one residual image generated from the input image, the fourth shading image, and the third albedo image.

93. The method of claim 86, further comprising: training the diffuse shading neural network to derive shading values of the fourth shading image using mapped shading values, each mapped shading value being mapped from a shading value associated with each pixel that is representative of the illumination effect.

94. The method of claim 86, further comprising: training the diffuse shading neural network to derive shading values of the fourth shading image representative of the illumination effect by mapping the shading values to a second domain of pre-defined range.

95. The method of claim 91 , further comprising: generating network training data comprising photographs by: obtaining a plurality of photographs for a scene under different lighting conditions; generating a plurality of albedo images corresponding to the plurality of photographs; generating a predicted albedo image by performing statistical analysis on the plurality of albedo images; generating a plurality of predicted shading images by dividing each of the plurality of photographs by the predicted albedo image; andcompiling the plurality of photographs, the predicted albedo image, and the plurality of predicted shading images into the network training data.

96. A method of image decomposition comprising: processing an input image with a first neural network to generate a first shading estimate, the input image comprising a shading component corresponding to shading that depicts a diffuse illumination effect on the content, an albedo component corresponding to an albedo that depicts diffuse reflectance on the content, and a residual component corresponding to a residual that depicts non-diffuse illumination effect on the content; processing the first shading with a second neural network to generate a second shading estimate; generating a first albedo estimate based on the second shading estimate; processing the second shading estimate and the first albedo estimate with a third neural network to generate a color estimate; generating a third shading estimate based on the color estimate; generating a second albedo estimate based on the third shading estimate; processing the third shading estimate and the second albedo estimate with a fourth neural network to generate a third albedo estimate; generating a fourth shading estimate based on the third albedo estimate; processing the fourth shading estimate and the third albedo estimate with a fifth neural network to generate a fifth shading estimate; generating a residual estimate based on the fifth shading estimate and the third albedo estimate, wherein the first neural network is trained to estimate the shading based on a relative shading order of pixels of the shading; wherein generation of shading estimates by one or more neural networks comprises representing the shading by mapping shading values to adomain of pre-defined range, each shading value representing shading of a pixel; and wherein the fifth shading estimate corresponds to the shading component, the third albedo estimate corresponds to the albedo component, and the residual estimate corresponds to the residual component.

97. A system comprising one or more processing units configured to perform the method of any one of claims 1 to 96.

98. A non-transitory computer-readable medium having computer readable instructions stored thereon, which , when executed by one or more processing units, causes the one or more processing units to perform the method of any one of claims 1 to 96.

Images