Method and apparatus for image correction

Abstract

Methods and apparatuses for image correction are provided. A processor-implemented method includes: using a neural network model provided with an input image to generate an illumination map including illumination values that depend on corresponding color shifts caused by one or more light sources individually affecting each pixel of the input image; and generating a white-adjusted image by removing at least a portion of the color shift from the input image using the generated illumination map.

Description

[0001] This application claims the benefit of Korean Patent Application Nos. 10-2021-0032119, filed on Mar. 11, 2021, and 10-2021-0052984, filed on Apr. 23, 2021, in the Korean Intellectual Property Office, the entire disclosures of which are incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to a technology for image correction. Background Art

[0003] For white balance, the color of light or illumination in a scene may be estimated to remove a color cast caused by the light. White balance may mimic the color constancy of the human visual system.

[0004] The inventors have had or obtained the above description in the process of conceiving the present disclosure, and the above description should not be considered as prior art advice known before the filing of the present application. Summary of the Invention

[0005] This Summary of the Invention is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary of the Invention is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter.

[0006] In one general aspect, a processor-implemented method includes: generating a light map including light values using a neural network model provided with an input image, the light values depending on corresponding color casts caused by one or more light sources that individually affect each pixel of the input image; and generating a white-adjusted image by removing at least a portion of the color cast from the input image using the generated light map.

[0007] The generated light map may include a light vector in which a partial light vector corresponding to the chromaticity of one of the light sources and another partial light vector corresponding to the chromaticity of another one of the light sources are mixed for at least one pixel of the input image.

[0008] The light map may include a mixing coefficient of the chromaticity of a light source that affects a pixel of the input image, and the mixing coefficient may be different from another mixing coefficient of the chromaticity of a light source that affects another pixel of the input image.

[0009] For the pixels of the input image, the result of the neural network model may include the first chromaticity information of one of the one or more light sources and the corresponding first mixing coefficient, and for another pixel of the input image, the result of the neural network model may further include the second chromaticity information of the one of the one or more light sources and the corresponding second mixing coefficient.

[0010] The step of generating the illumination map may include: generating a first pixel of the illumination map based on the first chromaticity information and the first mixing coefficient, and generating a second pixel of the illumination map based on the second chromaticity information and the second mixing coefficient.

[0011] The step of generating the white-adjusted image may include: applying the illumination map to the input image through element-wise operations to generate the white-adjusted image.

[0012] The step of applying the illumination map to the input image through element-wise operations may include: dividing the respective pixel values of each pixel at the corresponding pixel position in the input image by the corresponding illumination value at the same pixel position as the corresponding pixel position.

[0013] The white-adjusted image may be a white balance image.

[0014] The method may further include: determining the chromaticity information corresponding to each light source by decomposing the illumination map for each light source; generating a partial white balance image by adjusting the white-adjusted image, the partial white balance image having the retained chromaticity information corresponding to a part of the light source, and the chromaticity information corresponding to the remaining part of the light source being white information corresponding to the remaining part of the light source.

[0015] The method may further include: controlling a display to display the partial white balance image.

[0016] The determination may further include: determining the mixing coefficient corresponding to each light source by decomposing the illumination map for each light source, wherein the step of generating the partial white balance image may include: using the chromaticity information corresponding to the part of the light source and the corresponding mixing coefficient map corresponding to the part of the light source to generate a partial illumination map corresponding to the part of the light source; using the white information corresponding to the remaining part of the light source and the corresponding mixing coefficient map corresponding to the remaining part of the light source to generate a remaining illumination map corresponding to the remaining part of the light source; generating a re-illumination map by adding the partial illumination map and the remaining illumination map together; and multiplying the pixel value of each pixel of the white-adjusted image by the re-illumination value in the re-illumination map corresponding to the pixel position of the corresponding pixel.

[0017] The steps of generating an illumination map may include: extracting the chromaticity information of each light source and the mixing coefficient map of each light source by applying a neural network model to an input image; and generating an illumination map based on the extracted chromaticity information and the extracted mixing coefficient map.

[0018] The steps of generating an illumination map may include: calculating the partial illumination map of each light source by multiplying the chromaticity information of each light source with the mixing coefficient map of each light source respectively; and generating an illumination map by adding the calculated partial illumination maps together.

[0019] The steps of generating a white-adjusted image may include: determining the chromaticity information corresponding to a target light source among the extracted chromaticity information; calculating the partial illumination map of the target light source by multiplying the determined chromaticity information with the corresponding mixing coefficient map among the extracted coefficient maps; generating a re-illuminated map by adding the partial illumination map of the target light source to the remaining illumination map of the remaining part of the light source; and generating a partially white-balanced re-illuminated image by applying the re-illuminated map to the white-adjusted image, wherein at least a part of the color cast corresponding to the target light source is retained in the generated partially white-balanced re-illuminated image.

[0020] The step of determining the chromaticity information corresponding to the target light source may include: changing the chromaticity information among the extracted chromaticity information and corresponding to the remaining part of the light source into white information.

[0021] The method may further include: determining the mixing coefficient map of each light source such that the sum of the mixing coefficients calculated for a pixel with respect to the light sources is a reference value; calculating the partial illumination maps corresponding to each light source respectively based on the chromaticity information of each light source and the mixing coefficient map of each light source; and applying the calculated partial illumination maps to one of the input image and the white-adjusted image.

[0022] The step of determining the mixing coefficient map may include: identifying the total number of light source chromaticities affecting a pixel in the input image; and generating a corresponding mixing coefficient map corresponding to the identified total number of light source chromaticities.

[0023] The step of identifying the total number of light source chromaticities may include: setting the total number of light source chromaticities to three.

[0024] The step of generating a corresponding mixing coefficient map may include: initiating the generation of a corresponding mixing coefficient map in response to the total number of light source chromaticities being identified as two.

[0025] The step of determining the mixing coefficient map may include: when the total number of light sources is two light sources, estimating the mixing coefficient map of one of the two light sources; and calculating the remaining mixing coefficient map of the remaining other light source among the two light sources by subtracting each mixing coefficient in the estimated mixing coefficient map from the reference value.

[0026] In one general aspect, one or more embodiments include a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform any one, any combination, or all of the operations and methods disclosed herein.

[0027] In one general aspect, a device includes: a processor; and a memory storing one or more machine learning models and instructions that, when executed by the processor, configure the processor to: generate a lighting map including lighting values using one of the one or more machine learning models, the lighting values depending on corresponding color casts caused by light sources that individually affect pixels of an input image; and generate a white-adjusted image by adjusting at least a portion of the color cast from the input image using the generated lighting map.

[0028] The processor may further be configured to generate a partial white balance image by adjusting the white-adjusted image, the partial white balance image having retained chromaticity information corresponding to a portion of the light source and different chromaticity information corresponding to the remaining portion of the light source as compared to the corresponding chromaticity information in the lighting map.

[0029] The different chromaticity information may be white information.

[0030] The processor may further be configured to: decompose the lighting map for each light source to obtain chromaticity information of the light source; and generate a partial white balance image by adjusting the white-adjusted image, the partial white balance image having retained chromaticity information corresponding to a portion of the light source and different chromaticity information corresponding to the remaining portion of the light source as compared to the obtained chromaticity information.

[0031] The different chromaticity information may be white information.

[0032] The device may further include: an image sensor configured to capture the input image.

[0033] The instructions may further include instructions that, when executed by the processor, configure the processor to control the operation of the device based on the white-adjusted image.

[0034] The instructions may further include instructions that, when executed by the processor, configure the processor to identify a total number of light sources that affect pixels in the input image, wherein the step of using the one of the one or more machine learning models may include: selecting the one of the one or more machine learning models from among the one or more machine learning models stored in the memory based on the identified total number of light sources, wherein two different machine learning models are selected for two different total numbers of light sources, respectively.

[0035] In one general aspect, an apparatus includes: an image sensor configured to obtain an input image; and a processor configured to: generate an illumination map including illumination values using a neural network model provided with the input image, the illumination values depending on corresponding color casts caused by one or more light sources that individually affect each pixel of the input image; and generate a white balance image by removing at least a portion of the color cast from the input image using the generated illumination map.

[0036] The apparatus may further include a display, wherein the processor may further be configured to cause the display to display the white balance image.

[0037] From the following detailed description, the drawings, and the claims, other features and aspects will be apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 An example of an image correction method is shown.

[0039] Figure 2 An example of a neural network-based white balance operation is shown.

[0040] Figure 3 An example of generating a relit image from a white balance image is shown.

[0041] Figure 4 and Figure 5 An example of a white balance operation performed using chromaticity information, mixing coefficient information, and an illumination map is shown.

[0042] Figure 6 An example of a white balance method is shown.

[0043] Figure 7 An example of retaining an illumination component by a portion of a light source is shown.

[0044] Figure 8 and Figure 9 An example of collecting training data for training a neural network is shown.

[0045] Figure 10 An example of a hardware-implemented image correction device is shown.

[0046] Throughout the drawings and the detailed description, unless otherwise described or provided, the same reference numerals will be understood to represent the same or similar elements, features, and structures. For clarity, illustration, and convenience, the drawings may not be to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. DETAILED DESCRIPTION

[0047] The following specific embodiments are provided to assist the reader in obtaining a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications, and equivalents of the methods, apparatuses, and / or systems described herein will be apparent. For example, the order of operations described herein is merely exemplary, and the order of operations is not limited to those set forth herein, but may be changed as will be apparent after understanding the disclosure of the present application, except for operations that must occur in a specific order. Additionally, descriptions of features known after understanding the disclosure of the present application may be omitted for greater clarity and conciseness.

[0048] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Instead, the examples described herein have been provided only to illustrate some of the many possible ways of implementing the methods, apparatuses, and / or systems described herein that will be apparent after understanding the disclosure of the present application.

[0049] Throughout the specification, when a component is described as "connected to" or "coupled to" another component, it may be directly "connected to" or "coupled to" the other component, or there may be one or more other components intervening therebetween. Conversely, when an element is described as "directly connected to" or "directly coupled to" another element, there may be no other elements intervening therebetween. Similarly, like expressions (e.g., "between" and "immediately between" and "adjacent" and "immediately adjacent") should be interpreted in the same manner. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more thereof.

[0050] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or parts, these members, components, regions, layers, or parts should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or part from another. Thus, the first member, first component, first region, first layer, or first part referred to in the examples described herein may also be referred to as the second member, second component, second region, second layer, or second part without departing from the teachings of the examples.

[0051] The terms used herein are for the purpose of describing various examples only and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. The terms "comprising," "including," and "having" indicate the presence of stated features, quantities, operations, members, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, members, elements, and / or combinations thereof.

[0052] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs and based on the understanding of the disclosure of this application. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) shall be interpreted to have a meaning consistent with their meaning in the context of the relevant art and the disclosure of this application, and shall not be interpreted in an idealized or overly formal sense. The use of the term "may" herein with respect to an example or embodiment (e.g., as to what an example or embodiment may include or implement) means that there is at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto.

[0053] In the description of example embodiments, when it is considered that a detailed description of a structure or operation that is known after understanding the disclosure of this application will result in an ambiguous interpretation of the example embodiments, such a description will be omitted.

[0054] Due to human cognitive characteristics, a camera sensor and / or an image sensor may sense the whiteness of an object with a white chromaticity as another color (e.g., light yellow) due to the illumination or lighting of a light source with chromaticity (e.g., an incandescent lamp). White balance (white balance or white balancing) may represent an operation performed to reproduce the human cognitive process of recognizing an object with a white chromaticity as white in imaging. For example, in one or more embodiments, white balance may remove color cast caused by the chromaticity of a light source from the entire image and correct color distortion. As a non-limiting example, one or more embodiments may provide a method or device for image correction, and the method or device for image correction uses a machine learning model (such as a neural network model) to perform white balance in a multi-lighting environment.

[0055] Figure 1 An example of an image correction method is shown.

[0056] Referring to Figure 1 , in operation 110, the image correction device uses a neural network model to generate an illumination map including an illumination value from the obtained input image, and the illumination value indicates the color cast that the chromaticity of one or more light sources individually affects each pixel of the input image. For example, the input image may include color cast caused by multiple light sources. For example, the image correction device may generate an illumination map having a mixed illumination vector, in which a partial illumination vector corresponding to the chromaticity of one light source among the light sources and a partial illumination vector corresponding to the chromaticity of another light source among the light sources are mixed for at least one pixel of the input image. The light source is not limited to multiple light sources. As a non-limiting example, the image correction device may correspond to Figure 10Image correction device 1000.

[0057] The chromaticity described herein may represent a component indicating a color excluding luminance. In an input image captured in the presence of multiple light sources, different color offsets may exist in each pixel of the captured input image. For example, the ratio of each illumination of multiple light sources acting on one pixel may be different, and the color of the color offset in the pixel may be determined based on the ratio of chromaticities corresponding to the respective illuminations that are mixed in a complex manner. The ratio of the illumination or shadow of one light source affecting the intensity of each pixel may be referred to herein as a mixing ratio or mixing coefficient. The set of mixing coefficients for each pixel may be referred to herein as a mixing coefficient map, which will be described later with reference to, for example, Equation 5. The illumination map may represent a map including illumination values for correcting the color offset of the input image, where the illumination value for a pixel may indicate a value for compensating the color offset caused by the chromaticity of the light sources mixed based on the respective mixing coefficients. Each illumination value may be the sum of partial illumination values corresponding to multiple chromaticities of multiple light sources, and may be represented in the format of a partial illumination vector having components for each color channel defined based on the color channels. For example, the illumination map will be further described later with reference to Equation 6.

[0058] For example, an image captured in a mixed illumination environment may be modeled as represented by, for example, Equation 1 below. For example, in this example, the mixed light sources may include two or more light sources having different chromaticities.

[0059] Equation 1:

[0060]

[0061] In Equation 1, x represents the pixel position of one pixel among the multiple pixels included in the image I, and may indicate the coordinates in the image I (e.g., two-dimensional (2D) coordinates). I(x) represents the pixel value of the pixel at the pixel position x in the image I. r(x) represents the surface reflectance of the pixel position x in the image I. η(x) represents a scaling term including the intensity of the illumination and shadow of the mixed light sources affecting the pixel position x in the image I. η i (x) represents the intensity of the illumination and shadow of the i-th light source among the N light sources affecting the pixel at the pixel position x. l represents the chromaticity information of the mixed light sources. l i represents the chromaticity information of the i-th light source, where i represents an integer greater than or equal to 1 and less than or equal to N. Further, ⊙ represents element-wise multiplication (or product).

[0062] Chromaticity information may include a chromaticity vector that defines the chromaticity of a light source based on a color space. For example, for illustrative purposes of the RGB color space and other color spaces, color vectors defined in the Red, Green, Blue (RGB) color space will be mainly described herein. In this RGB color space example, the primary red chromaticity, the primary green chromaticity, and the primary blue chromaticity may be represented as vectors (1, 0, 0), vector (0, 1, 0), and vector (0, 0, 1), respectively. In addition, black information and white information may be represented as vectors (0, 0, 0) and vector (1, 1, 1), respectively. In one example, the chromaticity vector may be normalized such that the G value becomes 1. For example, for a vector such as (R, G, B) = (0.3, 0.7, 0.3), by dividing the component values of each channel by the G value 0.7, the chromaticity vector may be normalized to (0.43, 1, 0.43). The values of the chromaticity vector are not limited thereto, and for example, in different examples with varying hardware configurations, in other varying examples with the same hardware configuration, or regardless of the hardware configuration, the values of each element in the chromaticity vector or the normalization method may vary. As described above, although chromaticity vectors have been discussed with respect to the RGB color space, as a non-limiting example, chromaticity vectors may also be defined based on other color spaces (such as, for example, the Hue / Saturation / Value (HSV) color space, the International Commission on Illumination (CIE) color space, and / or the Cyan / Magenta / Yellow / Black (CMYK) color space). The total number of dimensions of the chromaticity vector may vary according to the number of components of the color space. Thus, as an example, although the chromaticity vector was described as three dimensions in the foregoing RGB color space example, the chromaticity vector may be defined as four dimensions in a color space defined by four components (such as, in the CMYK color space example).

[0063] In Equation 1 above, N represents an integer greater than or equal to 1. Hereinafter, although there are also examples where N is 1 or N is greater than 2, for purposes of explanation, an example where N = 2, the first light source is light source a, and the second light source is light source b will be described (such as, for example, the example presented in Equation 2 below).

[0064] Equation 2:

[0065] I ab (x) = r(x) ⊙ (η a (x)l a + η b (x)l b )

[0066] In Equation 2, I ab (x) represents the pixel value corresponding to pixel position x in the image I ab in an environment with light sources a and b. η a(x) represents the intensity of light and shadow that light source a affects pixel position x, and l a represents the chromaticity vector of light source a. Similarly, η b (x) represents the intensity of light and shadow that light source b affects pixel position x, and l b represents the chromaticity vector of light source b. As an example, white balance can be interpreted as the correction performed to make all light sources have a standard chromaticity. For example, a white balance image can be an image obtained, for example, by correcting both the chromaticity vector of light source a and the chromaticity vector of light source b (for example, correcting to the value 1) as shown in Equation 3 below.

[0067] Equation 3:

[0068]

[0069] In Equation 3, represents the pixel value at pixel position x of the white balance image . The value 1 in Equation 3 represents the chromaticity information indicating white (hereinafter, white information) (for example, chromaticity vector (1, 1, 1)). For simplicity of description, in Equations 4 to 6 below, the reference to the basic pixel position x of a single pixel is not repeated. Only as a non-limiting example, for illustrative purposes, only image I ab will be described.

[0070] Equation 4:

[0071]

[0072] Equation 5:

[0073]

[0074] Equation 6:

[0075] L ab = αl a + (1 - α)l b

[0076] As shown in Equation 4, the input image I ab can be interpreted as being able to apply the light map L ab to the white balance image The obtained results. α represents the mixing coefficient map of light source a. (1 - α) represents the mixing coefficient map of light source b. For example, in the presence of two light sources, the image correction device can estimate the mixing coefficient map α of one of the two light sources (e.g., light source a). Then, the image correction device can calculate the mixing coefficient map (1 - α) of the remaining light source (e.g., light source b) among the two light sources by subtracting each mixing coefficient of the mixing coefficient map α from the reference value (e.g., 1) for each pixel. Although the mixing coefficient maps of light source a and light source b are described above with reference to Equation 5, as a non-limiting example, the mixing coefficient map of the i-th light source among N light sources can be generalized as

[0077] Therefore, L ab (x) represents the illumination vector corresponding to the pixel position x, and the illumination map L ab can be the set of the corresponding illumination vectors L ab (x). The partial illumination vectors αl ab and (1 - α)l a included in the illumination vector L b can be the products of the mixing coefficient maps α and (1 - α) of the corresponding light sources and the chromaticity vectors l a and l b . Therefore, the illumination map L ab can be interpreted as providing a set of illumination channel maps for each color channel based on the definition of the color space. For example, the partial illumination vector α(x)l a at the pixel position x can be the product of the component corresponding to each color channel (e.g., one channel of RGB) of the chromaticity vector l a and the mixing coefficient α(x) at the pixel position x. Therefore, the partial illumination vector α(x)l a can separately include the partial illumination component corresponding to the R channel, the partial illumination component corresponding to the G channel, and the partial illumination component corresponding to the B channel. As shown in Equation 6 above, the mixing coefficient maps α and (1 - α) of the corresponding light sources and the chromaticity vectors l ab and l a and l b in the illumination map L ab can have spatially different values for each pixel (e.g., since the illumination from each light source falls on the pixel differently, different positioned light sources can be differently represented in the pixel values of the same pixel in the input image). Therefore, the image correction device can use a machine learning model (such as a neural network model) to generate the illumination map L ab . For example, as a non-limiting example, the neural network model can be trained to extract the illumination map L ab from the input image. In one example, the direct extraction of the illumination map L ab will be described in further detail below with reference to Figure 2Such a neural network model. In one example, the neural network model can be trained to extract the chromaticity information of each light source and the mixing coefficient map of each light source from the input image. In such an example, as discussed above for Equation 6, the image correction device can use the extracted chromaticity information and the extracted mixing coefficient map to calculate the illumination map (e.g., illumination map L ab ). Below will refer to Figure 4 For a further more detailed description of an example neural network model that extracts the chromaticity information of each light source and the mixing coefficient map of each light source.

[0078] In operation 120, the image correction device can remove at least a part of the color cast from the input image by using the generated illumination map L ab to generate an image (e.g., pixel-level white balance image). For example, the image correction device can apply the illumination map L ab to the input image through element-wise operations to generate an example pixel-level white balance image. As shown in Equation 7 below, an example image correction device can divide the pixel value of each pixel of the input image I ab by the illumination value in the illumination map L ab corresponding to the pixel position of the corresponding pixel. For example, for each color channel at each pixel position, the image correction device can divide the color channel value by the component corresponding to the color channel in the illumination vector. For example, element-wise operations (e.g., corresponding element-wise multiplication and element-wise division) can be performed between the values of the corresponding color channels at the same corresponding pixel position. Therefore, the image correction device can generate an example image (such as an example pixel-level white balance image) from the input image I ab . As an example, the following example Equation 7 can be obtained from the above Equation 6 and can represent the generation of an example pixel-level white balance image .

[0079] Equation 7:

[0080]

[0081] In Equation 7, represents element-wise division.

[0082] In an example of a scenario where an input image is captured under a single illumination, the entire input image can be processed in gray, or the entire input image can be white balanced by using normalization of the single illumination chromaticity based on regions estimated to be white chromaticity in the input image. In another example, the image capture environment can be a place where potentially multiple light sources with various chromaticities are mixed, and thus, the chromaticities of the corresponding light sources can complexly affect each pixel of the input image captured in the example image capture environment. In one example, an image correction method can compensate for such complex effects of the chromaticities of multiple light sources, and thus, provide a satisfactory white balance result even in a mixed illumination environment. As described above, the image correction operation can apply illuminations of light sources with different chromaticities to each pixel individually or differently, and perform white balance on local regions individually or differently. That is, the chromaticity of the light source that causes a color shift to be compensated in one pixel of the image and the chromaticity of the light source that causes a color shift to be compensated in another pixel of the image can be determined individually or differently.

[0083] For example, an image captured outdoors can differently include color shifts caused by two light sources (e.g., yellow sunlight and blue sky) in each pixel. In contrast, an image captured indoors can differently include color shifts caused by three light sources (e.g., natural light, indoor light, and external flash) in each pixel. In one example, an example image correction method can perform white balance by correcting the image using illumination values determined to be optimized for each corresponding pixel, where the mixing ratio of color shifts caused by light sources with different chromaticities is specific or different for each pixel.

[0084] Figure 2 An example of a neural network-based white balance operation is shown.

[0085] In one example, an image correction device can calculate an illumination map 230 from an input image 201 by using a neural network model 210. As a non-limiting example, the image correction device can correspond to Figure 10 the image correction device 1000.

[0086] The input image 201 can include multiple channel images in a color space. For example, an RGB image can include a red channel image, a green channel image, and a blue channel image. As other examples, each of a YUV image and an HSV image can include corresponding multiple channel images. In one or more examples, the input image 201 can be a full high definition (FHD) image or a 4K resolution image, note that there are examples with a resolution lower than such an FHD image, a resolution between the FHD and 4K images, and a resolution higher than the 4K image.

[0087] In one or more examples, the neural network model 210 can be a machine learning model configured to generate an illumination map 230 with the same resolution as the input image 201, but the examples are not limited thereto. The image correction device can extract the illumination map 230 by inputting the input image 201 into the neural network model 210, and generate the illumination map 230 through corresponding neural network operations of the hidden layers of the neural network model 210. The illumination map 230 can include illumination values (e.g., illumination vectors) for compensating color casts at each pixel position. The machine learning model can be generated through machine learning. As described below for Figure 8 and Figure 9 described, in one or more embodiments, learning can be performed in the image correction device itself (in the image correction device, the trained neural network model 210 can subsequently be executed in an inference operation to generate the illumination map 230), or learning can be performed by a server separated from the image correction device.

[0088] The neural network model described herein can include multiple neural network layers (e.g., one or more input layers, at least one output layer, and multiple hidden layers between the input layer and the output layer). The neural network model can be any one or any combination of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBN), a deep belief network (DBN), a bidirectional recurrent DNN (BRDNN), a deep Q network, a U-net, or a combination of two or more thereof, note that the examples are not limited thereto. As a non-limiting example, the training parameters of the neural network (such as the connection weights between nodes, the connection weights of the same node at previous time and / or subsequent time, and / or kernel elements) can be stored in the memory of the image correction device.

[0089] In the training of the neural network, the learning methods performed can include, for example, supervised learning, unsupervised learning, semi-supervised learning, and / or reinforcement learning, but the examples are not limited thereto.

[0090] Furthermore, in one or more of such example learning methods, the generation of a temporary output (e.g., the temporary illumination map L during training) can be trained based on losses (such as L1, L2) and the structural similarity index measure (SSIM) (or simply, structural similarity). aba neural network model during training to ultimately generate a trained neural network model 210. As a non-limiting example, training can include calculating a loss between a ground truth output (such as label data corresponding to each training input) and a temporary output, and updating the parameters (e.g., connection weights and / or kernel elements) of the neural network model during training to minimize the loss, where the temporary output is calculated by propagating data of the corresponding results of the hidden layers in the neural network during training to the output layer for the corresponding input training data. As a non-limiting example, for the training of the neural network model during training, the training data can include at least one of a ground truth illumination map of an image captured in a multi-illumination environment and a pixel-level white balance image. Reference will be made to Figure 8 and Figure 9 to further describe the training data. Here, with respect to the use of neural network expressions (e.g., "neural network model 210"), compared to, for example, the operation, use, implementation, or execution of a trained neural network model to generate an inference result, a reference to the training of an example neural network model (or neural network) or other training of another neural network model (or other neural network) should be interpreted as performing some of the training of the example neural network during training above (e.g., by a training device or in a training operation of another electronic device according to any training method described herein) to generate the resulting trained neural network model. Similarly, a reference to the execution, use, implementation, or execution of an example neural network model (or neural network) to generate an inference result should be interpreted as the execution, use, implementation, or execution of a trained neural network model (neural network) or a trained other neural network model (or other neural network). Here, the example is not limited to such execution, use, implementation, or execution terms for generating an inference result, and other expressions can also be used for the inference operation of the neural network.

[0091] Thus, as a non-limiting example, the image correction device can, for example, use the machine learning model 210 to estimate an illumination map 230 having an example same size and resolution as the input image 201, and generate a pixel-level white balance image 208 having an example same size and resolution as the input image 201 by applying the illumination map 230 to the input image 201 via element-wise division 280. In one example, as a non-limiting example, the pixel-level white balance image 208 can be stored and / or output, such as via a display, or forwarded to another component or operation of the image correction device.

[0092] Figure 3 An example of generating a relit image from a white balance image is shown.

[0093] In one example, for instance, in addition to the above reference to Figure 2In addition to the operations described, the image correction device may perform a decomposition operation 340. As a non-limiting example, the image correction device may correspond to the Figure 10 image correction device 1000.

[0094] For example, the image correction device may decompose the illumination map 230 for each light source and determine the chromaticity information 351 for each light source and the mixing coefficient map 352 for each light source. In one example, as a non-limiting example, the image correction device may decompose the illumination map 230 for each light source by principal component analysis (PCA). In the illumination map 230, the mixing coefficients of the chromaticities of multiple light sources affecting one pixel may be different from the mixing coefficients of the chromaticities of multiple light sources affecting another pixel.

[0095] For example, the image correction device may generate a partial white balance image 309 by retaining the chromaticity information 351 corresponding to a part of the multiple light sources, changing the chromaticity information corresponding to the remaining part of the light sources, and additionally applying the changed chromaticity information together with the mixing coefficient map 352 to the pixel-level white balance image 208. The partial white balance image 309 may also be referred to as a pixel-level controlled white balance image 309. In this example, the image correction device may use the chromaticity information 351 corresponding to a part of the light source and the mixing coefficient map 352 to generate a target partial illumination map corresponding to that part of the light source. The image correction device may change the chromaticity information corresponding to the remaining part of the light source to white information and use the white information and the mixing coefficient map 352 to generate a remaining illumination map corresponding to the remaining part of the light source. The remaining illumination map may be a partial illumination map in which the chromaticity of the remaining part of the light source is replaced by white, thereby removing the color cast caused by the remaining part of the light source. The image correction device may generate a relit map by adding the target partial illumination map and the remaining illumination map together. The relit map may be a map including relit values that add the color cast effect of the chromaticity of a part of the light source in the white balance image for relighting. The image correction device may apply an element-wise multiplication 390 by multiplying the pixel value of each pixel of the pixel-level white balance image 208 by the relit value corresponding to the pixel position of the corresponding pixel among the relit values in the relit map. Thus, the partial white balance image 309 may be an image obtained by (multiplicatively) adding the color cast caused by the target light source to the white balance image. Thus, the image correction device may generate a partial white balance image 309 that at least retains the color cast caused by a part of the light source. In addition, as a non-limiting example, the partial white balance image 309 may be stored and / or output, such as through a display, or forwarded to another component or operation of the image correction device.

[0096] In one example, in addition to the losses described above with reference to Figure 2 description, Figure 3The machine learning structure for generating the neural network 210 shown may also include the cosine similarity of color channels.

[0097] Figure 4 and Figure 5 illustrates an example of a white balance operation performed using chromaticity information, mixing coefficient information, and a light map.

[0098] In one example, the image correction device may extract the chromaticity information 421 of each light source and the mixing coefficient map 422 of each light source by executing the neural network model 410 for the obtained input image 201. For example, the neural network model 410 may be a model trained to generate the chromaticity information 421 and the mixing coefficient map 422 from the input image 201 (e.g., when there are multiple light sources). For Figure 4 and Figure 5 , as corresponding non-limiting examples, the image correction device may correspond to Figure 10 the image correction device 1000.

[0099] In one example, the image correction device may generate the light map 430 by linear interpolation based on the chromaticity information 421 and the mixing coefficient map 422. For example, the image correction device may generate a partial light map by multiplying the chromaticity information 421 of each of the multiple light sources by the mixing coefficient map 422 of the corresponding light source. The image correction device may generate the light map 430 by adding together the respective partial light maps obtained by multiplying the chromaticity information 421 of each light source by the mixing coefficient map 422. Thus, the image correction device may generate the light map 430 by adding together the generated partial light maps of the light sources. The image correction device may generate the pixel-level white balance image 408 by applying the light map 430 generated without changing the chromaticity information 421 to the input image 201, for example, by element-wise division 480.

[0100] In one example, the image correction device may additionally add a relighting map to the pixel-level white balance image 408 to generate a partially white balance image that retains a color cast and / or an image in which the lighting effect generated by a part of the light source is changed from the input image 201.

[0101] For example, the image correction device may determine the chromaticity information 421 corresponding to the target light source. In this example, the image correction device may maintain the chromaticity information 421 corresponding to the target light source as the original chromaticity vector. The original chromaticity vector of a light source may indicate the chromaticity vector of the light source extracted using the neural network model 410. The image correction device may calculate the partial illumination map of the target light source by multiplying the maintained chromaticity information 421 by the mixing coefficient map 422. For example, the image correction device may calculate the partial illumination map of the target light source through the multiplication between the chromaticity information 421 having the original chromaticity vector of the target light source and the mixing coefficient map 422. The image correction device may generate the relit map by adding the partial illumination map of the target light source and the remaining illumination map of the remaining light sources. The image correction device may generate a relit map that does not include the color cast caused by the remaining light sources by changing the chromaticity information corresponding to the remaining light sources to white information (e.g., the vector (1, 1, 1)). The image correction device may generate a partially white-balanced relit image that retains at least a part of the color cast caused by the target light source by applying the relit map to the pixel-level white-balanced image 408 through element-wise multiplication. Therefore, using the relit map, the image correction device may remove the color cast caused by the remaining light sources from the input image 201 and generate a partially white-balanced relit image that retains the color cast caused by the target light source. However, the generation of the relit map is not limited to the above.

[0102] In one example, the image correction device may change the chromaticity information corresponding to the target light source among the multiple light sources to the target chromaticity vector. For example, the user may select the target light source. Therefore, the target chromaticity vector may be a vector indicating how the user selects to change the chromaticity. The image correction device may calculate the partial illumination map having the target chromaticity of the target light source by multiplying the determined target chromaticity vector by the mixing coefficient map. The image correction device may generate the relit map by adding together the partial illumination map of the target light source and the remaining illumination map of the remaining light sources. For example, while maintaining the chromaticity information of the remaining light sources, the image correction device may generate an image in which only the illumination effect of the target light source is changed to the target chromaticity from the original input image 201 by applying the relit map to the pixel-level white-balanced image 408 based on element-wise multiplication. In one example, the image correction device may generate the relit map by changing the chromaticity information of the remaining light sources to white information and adding together the partial illumination map based on the target chromaticity vector and the remaining illumination map based on the white information. In this example, the image correction device may generate a partially white-balanced image by applying the relit map to the pixel-level white-balanced image 408 through element-wise multiplication, in which only the illumination effect of the target light source having the target chromaticity is added to the pixel-level white-balanced image 408.

[0103] The image correction device can determine the mixing coefficient map 422 of each light source such that the sum of the corresponding mixing coefficients calculated for one pixel according to the light sources becomes a reference value (e.g., 1). For example, the reference value can be the sum of the values corresponding to the same pixel position in the mixing coefficient map of the light sources. Therefore, since the reference value or the sum of the mixing coefficient maps can be the same, there may be no change in brightness during the correction process. Thus, the image correction device can effectively remove only the color cast caused by the chromaticity of the light sources while maintaining the image brightness. The image correction device can calculate the partial illumination map corresponding to each light source based on the chromaticity information 421 of each light source and the mixing coefficient map 422 of each light source. The image correction device can apply the partial illumination map to Figure 3 either the input image 201 or the generated pixel-level white balance image 208.

[0104] In Figure 4 the example, N sets of chromaticity information 421 and N mixing coefficient maps 422 can be extracted from N light sources. In this example, N can be 3 (N = 3). The image correction device can store the corresponding neural network model 410 corresponding to the number of light sources (e.g., when there are three light sources, among the multiple trained neural network models 410 trained separately for various other numbers of light sources, the neural network model 410 specifically trained for three light sources can be trained and stored in the memory of the image correction device). Thus, the image correction device can identify the number of light sources from the input image 201 and load the appropriate neural network model 410 corresponding to the identified number. The image correction device can extract the chromaticity information 421 and the mixing coefficient map 422 corresponding to the identified number of light sources.

[0105] In such an example, the pixel-level white balance image 408, the pixel-level white balance image 408 to which the relighting map has been added, the input image 201 to which the partial illumination map has been applied, and / or the pixel-level white balance image 208 to which the partial illumination map has been applied can be generated and stored and / or output, for example, through a display, or forwarded to other components or operations of the image correction device.

[0106] Figure 5Show example application results. The image correction device can extract first chromaticity information 521a and a first mixing coefficient map 522a from the input image 501 using a correspondingly trained neural network model, and extract second chromaticity information 521b and a second mixing coefficient map 522b. The input image 501 can be an image in which there is external light passing through a window (e.g., light source #1 as shown) and internal light passing through an indoor light source (e.g., light source #2 as shown). The image correction device can obtain an illumination map 530 by linearly interpolating the first chromaticity information 521a, the first mixing coefficient map 522a, the second chromaticity information 521b, and the second mixing coefficient map 522b. Thus, the image correction device can generate a fully white-balanced image 508, in which all illumination effects are compensated by correcting the input image 501 using the illumination map 530.

[0107] Figure 6 Show an example of a white balance method.

[0108] Refer to Figure 6 , in operation 610, the image correction device obtains an image. For example, the image correction device can capture an image through an image sensor. In one example, the image correction device can receive an image from an external server through a communicator. For another example, the image correction device can load an image stored in a memory, or can capture an image from the surrounding environment of the image correction device. As a non-limiting example, the image correction device can correspond to Figure 10 the image correction device 1000.

[0109] In operation 620, the image correction device determines whether there are multiple illuminations in the image. For example, the image correction device can identify the type of chromaticity of the light source that causes a color shift of the pixels affecting the image.

[0110] In operation 621, when there is only a color shift caused by a single light source in the image, the image correction device adjusts the white balance in a first manner. For example, the image correction device can adjust the white balance in such a first manner assuming a single illumination. However, the example is not limited thereto, and even in the case of a single light source, the image correction device can perform white balance for the single light source based on the method or operation described above with reference to Figures 1 to 5 . For example, the image correction device can generate a white balance image under the assumption of a single light source by applying an average map of the corresponding partial illumination map obtained from the light source (e.g., a map including the average value of the illumination values indicating the same pixel position). In the example of a single light source, in addition to being able to generate / output a single light source white balance image in operation 660, the image correction device can omit the operations after operation 620.

[0111] When the total number of identified light sources is two or more, the image correction device may generate a corresponding mixing coefficient map for each of the total number of light sources.

[0112] In operation 630, the image correction device estimates the chromaticity and mixing coefficient map for each illumination. For example, the image correction device may identify the number of light source chromaticities affecting the input image. The image correction device may generate a mixing coefficient map based on the identified number. In one example, in response to the total number of light source chromaticities being identified as two, the image correction device initiates the generation of a corresponding mixing coefficient map, but the example is not limited to the case where the total number of light source chromaticities is two.

[0113] In one example, as discussed above for Figure 3 the image correction device may decompose the illumination map into the identified number of groups of chromaticity information and corresponding mixing coefficient maps. In one example, the image correction device may selectively use a neural network model that has been previously trained for the same total number of training light sources as the total number of identified light sources (e.g., compared to another stored neural network model that has been previously trained for a different total number of training light sources) to extract the identified total number of groups of chromaticity information and the corresponding mixing coefficient maps for the identified total number.

[0114] In one example, the image correction device may set and / or limit the number of light source chromaticities to three. Since the color space is typically formed in three dimensions, it may be effective to calculate chromaticity information and mixing coefficient maps only for three light sources.

[0115] In operation 640, the image correction device estimates the illumination map. For example, the image correction device may calculate the partial illumination map for each light source based on the multiplication between the chromaticity information and the mixing coefficient map, and calculate the illumination map by adding the corresponding partial illumination maps of multiple light sources.

[0116] In operation 650, for example, according to the estimation in operation 640, the image correction device adjusts the chromaticity information based on the light sources selected to be retained or set to be retained. When the chromaticity information is adjusted, an illumination map for retaining a part of the information of the retained light sources may be finally generated. When the chromaticity information is not adjusted, an illumination map for removing the color cast caused by all light sources may be finally generated. Such adjustment of the chromaticity information will be further described with reference to Figure 7 below.

[0117] In operation 660, the image correction device generates an image by applying the generated illumination map to the input image. When the chromaticity information is adjusted, the image correction device generates an image in which a part of the color cast is retained. Thus, the image correction device may perform white balance separately for the illumination of each light source.

[0118] Figure 7An example of retaining a light component through a part of a light source is shown.

[0119] In one example, the image correction device may be an electronic device 700 (e.g., a mobile terminal) as shown in Figure 7 , but the examples are not limited thereto. As a non-limiting example, an example having the electronic device 700 is any one of various devices including an image sensor, a tablet computer, augmented reality (AR) glasses, a vehicle, a television (TV), etc. Additionally or alternatively, as a non-limiting example, the image correction device may correspond to Figure 10 the image correction device 1000.

[0120] In response to a user input 709, the electronic device 700 may output, for example, the white balance image 790 generated as described above to a display or other components or operations of the electronic device. Additionally, the electronic device 700 may provide a user interface that may provide (e.g., display) information for selecting a light source to be retained. For example, in response to a user input for identifying a user selection in the user interface, the electronic device 700 may perform white balance excluding the light effect corresponding to the user preference (e.g., the electronic device 700 may selectively apply white balance only to the remaining light effects). Additionally or alternatively, the user may select or identify those light effects for which the user wants to perform white balance, and the electronic device 700 may determine not to perform white balance on the remaining light effects not selected or identified by the user. Figures 1 to 6

[0121] As an example of selectively applying white balance based on user preference, the electronic device 700 may retain the chromaticity information of a target light source selected by a user input, and change the chromaticity information of the remaining light sources to white information. For example, as described above with reference to Figure 4 , the electronic device 700 may calculate a partial light map of the target light source based on the retained chromaticity information, calculate a remaining light map based on the white information, and generate a relit map by adding the partial light map and the remaining light map together. The electronic device 700 may generate a relit image by applying the relit map to the white balance image 790. In one example, the generated relit image may be output, for example, through a display, and / or the generated relit image may be provided to other components or operations of the electronic device 700.

[0122] In Figure 7 ​In the example, the electronic device 700 can obtain the first relit image 710 by applying the relit map generated by replacing the second chromaticity information of the second light source with white information (e.g., vector (1, 1, 1)) in response to the user input 701 to the white balance image 790. The electronic device 700 can also obtain the second relit image 720 by applying the relit map generated by replacing the first chromaticity information of the first light source with white information in response to the user input 702 to the white balance image 790. Therefore, the electronic device 700 can selectively perform white balance based on the user's preference.

[0123] In another example, the image correction device can change the chromaticity information of each light source in response to the user input. The image correction device can change the chromaticity of the color cast caused by a part of the light source. The electronic device 700 can change the chromaticity of the target light source, so as to change the chromaticity of the lighting effect of a part of the light source to the chromaticity desired by the user. Therefore, the electronic device 700 can selectively apply white balance to a part of multiple light sources while ensuring the vividness of the scene, and generate white balance images in various ways based on the user's selection.

[0124] Figure 8 and Figure 9 illustrates an example of collecting training data for training a neural network.

[0125] In one example, the training data set can be collected, for example, by an image correction device, a training device, and / or a server to train a neural network model (e.g., CNN, HDRnet, U-net, etc.) to generate a lighting map of each light source, or a chromaticity information and a mixing coefficient map. The training data set can be a large-scale multi-illumination (LSMI) data set for multi-illumination white balance. For example, the training data set can include images of various scenes with pixel-level ground truth lighting maps. In one example, the training data set can include training images of various scenes and the ground truth chromaticity information and ground truth mixing coefficient maps of each light source corresponding to each training image.

[0126] As an example, the training device can enhance the input training data by changing the mixing coefficient value of the mixing coefficient map.

[0127] In one or more non-limiting examples, the training device can have Figure 10Configuration of the image correction device 1000 (e.g., the training device includes at least a processor 1020 and / or a memory 1030. For example, one or more of the processors 1020 perform training, and one or more of the memories store the neural network during training and the resulting trained neural network, as well as training data, temporary data generated during the training of the neural network, and instructions that configure one or more processors 1020 to implement training operations when executed by one or more processors 1020). For example, the training device can be configured to perform training without the inference operation of white balancing the input image discussed above, or can be configured to perform training and subsequent inference operations for white balancing the input image. Similarly, the reference to the image correction device herein also refers to such an inference operation of white balancing the input image performed or implemented by the image correction device, and can also refer to the training of the neural network and subsequent use of the trained neural network to perform or implement such an inference operation of white balancing the input image.

[0128] In one example, an image of a scene with a single light source can be captured, and the effect of the single light source present in the scene can be estimated by using a white balance method such as a color checker, a grey card, a spider cube, etc. A virtual illumination map can be randomly generated and applied to the white balance image obtained under the single light source, and a virtual training image with mixed illumination can be obtained.

[0129] In Figure 8 the example, the training image I ab 810 can be an image captured in a state where both light source a and light source b are turned on when the image is captured. In this example, light source a can indicate natural light entering through a window, and light source b can indicate indoor light emitted by a light source or lighting installed indoors. Light source a can be uncontrollable, but light source b can be controllable. Therefore, the image I a 820 can be captured in a state where light source b is turned off. For scene diversity, the training image I ab 810 can be captured at various locations in the real world (such as, for example, an office, a studio, a living room, a bedroom, a dining room, a café, etc.).

[0130] For example, although light source a is not turned off, as represented by Equation 8 below, the training image I ab 810 can be subtracted from the image I a 820, so the image I b 830 under light source b can be obtained.

[0131] Equation 8:

[0132] I b = r⊙(η b l b )

[0133] = r⊙(η a l a + η b l b ) - r⊙(η a l a )

[0134] = I ab - I a .

[0135] The color palettes 821 and 831 (e.g., Macbeth color palette), color charts, gray cards, cube spiders, etc. set in the scenes of each image can be used to separately estimate the ground truth chromaticity of a single light source. For example, the ground truth mixing coefficient map can be estimated as shown in Equation 9 below.

[0136] Equation 9:

[0137]

[0138] Compared with the number of elements that sense the R color or B color, a camera sensor and / or image sensor using the Bayer pattern of the RGB color system can have a larger number of elements that sense the G color in the same area size. Therefore, the intensity value of the green channel can approximate the luminance. Through Equation 9 above, a ground truth mixing coefficient map indicating the ratio of the mixed chromaticity of each light source can be obtained at each pixel level. For example, as Figure 9 shown, the green channel image I a can be obtained from the image I a,G 921, and the green channel image I b can be obtained from the image I b,G 931. The training device can normalize the green values of each of the green channel images I a,G 921 and I b,G 931 to 1 to obtain the mixing coefficient map 922 of light source a and the mixing coefficient map 932 of light source b. Then, the ground truth illumination map 970 can be estimated by adding the product of the chromaticity information of light source b and the mixing coefficient map 932 to the product of the chromaticity information of light source a and the mixing coefficient map 922. As described above with reference to Figure 8 and 9 , the training dataset obtained in the real world can be enhanced by applying random illumination information to the training dataset obtained in the real world.

[0139] As a non - limiting example, the training device can generate neural network models (e.g., neural network models 210 and 410) by using the training data set obtained as described above. For example, the training device can train a temporary neural network model or a neural network model in training by using virtual training images and ground - truth chromaticity information and ground - truth mixing coefficient maps for enhancing the training images as training outputs, and finally generate Figure 4 the neural network model 410. In an additional or alternative example, the training device can train a temporary neural network model or a neural network model in training by using virtual training images and corresponding ground - truth illumination maps as training outputs, and finally generate Figure 3 the neural network model 210. As a non - limiting example, the training device can input the training images into the temporary neural network model or the neural network model in training, calculate the loss between the temporary output of the temporary neural network model or the neural network model in training and the training outputs (e.g., ground - truth illumination map, ground - truth chromaticity information, and ground - truth mixing coefficient map), and repeatedly update the parameters of the temporary neural network model or the neural network model in training until the calculated loss becomes less than a threshold loss.

[0140] As a non - limiting example, the finally trained neural network model can be or include HDRnet and U - net. For example, in Figure 3 and Figure 4 the cases where the neural network models 210 and 410 are each implemented as U - net, improved performance can be achieved.

[0141] Figure 10 An example of a hardware - implemented image correction device is shown.

[0142] Referring to Figure 10 , as a non - limiting example, the image correction device 1000 includes an image acquirer 1010, a processor 1020, a memory 1030, and a display 1040.

[0143] The image acquirer 1010 can acquire an input image. The image acquirer 1010 can include at least one of an image sensor configured to capture the input image and a communicator configured to receive the input image and / or read the input image only from the memory 1030.

[0144] The processor 1020 can generate a white balance image and / or a relit image from an input image using a neural network model stored in the memory 1030. In one example, the processor 1020 can generate a light map including light values indicating color casts from the obtained input image using the neural network model, and the chromaticities of multiple light sources individually affect each pixel of the input image through the color casts. The processor 1020 can generate an image by removing at least a portion of the color cast from the input image using the generated light map, and the image can be output, for example, through the display 1040 and / or provided to other components or operations of the image correction device. Note that the operations of the processor 1020 are not limited to the foregoing. The processor 1020 can perform any one, any combination, or all of the above-described operations and / or methods referred to Figures 1 to 9 in the above description.

[0145] The memory 1030 can store one or more neural network models. The memory 1030 can temporarily or permanently store data required to perform any one, any combination, or all of the image correction operations and / or methods described herein. For example, the memory 1030 can store the input image, the generated image, and / or the relit image (e.g., for output or display by the display 1040 and / or provided to other components or operations for corresponding control of the image correction device 1000 based on the generated image and / or the relit image). Additionally, in one or more examples, with or without performing the inference operation of image correction on the input image, the memory 1030 can temporarily or permanently store data and instructions for performing any one, any combination, or all of the training operations and / or methods described herein. The instructions described herein that can be stored in the memory of the image correction device, the training device, or the electronic device are computer-readable instructions or computer-implementable instructions stored in the memory or other non-transitory media, and when executed by the processor 1020, the instructions configure the processor 1020 to implement any one, any combination, or all of the operations and methods described herein.

[0146] As described above, the display 1040 can display the generated image. The display 1040 can also visualize the input image and / or the relit image. The display can also provide a user interface to the user for which light sources or all light sources to perform white balance correction.

[0147] In one example, the image correction device 1000 can selectively perform white balance on the lighting effects of all light sources in the image by estimating, for example, a lighting map having the same size as the input image. For example, the image correction device 1000 can estimate the chromaticity information and the mixing coefficient map of each light source and selectively correct the mixed lighting effect. The image correction device 1000 can also perform pixel-level white balance, so the chromaticity and amount of the color cast compensated and / or removed from one pixel in the image can be different from the chromaticity and amount of the color cast compensated and / or removed from another pixel in the image.

[0148] The image correction device 1000 can be an image processing device that uses deep image processing, artificial intelligence (AI) computing, or machine learning (such as for generating a lighting map or extracting chromaticity information and coefficient maps), and one or more of the processor 1020 and / or a specially configured machine learning processor (such as a neural processor (NPU) or a hardware accelerator configured to implement such deep image processing, artificial intelligence (AI) computing, or machine learning). The image correction device 1000 can be, for example, a mobile camera, a general camera, an image processing server, a mobile terminal, AR glasses, a vehicle, etc. As a non-limiting example, any one of a mobile camera, a general camera, a mobile terminal, AR glasses, and a vehicle can include one or more image sensors of a corresponding camera configured with an image acquirer 1010 of any one of a mobile camera, a general camera, a mobile terminal, AR glasses, and a vehicle. As a non-limiting example, examples include that the image correction device 1000 is configured to perform recognition and / or tracking of features in a captured image, as well as image classification, object tracking, optical flow estimation, depth estimation, etc. For example, the memory 1030 can store instructions executable by the processor 1020 and / or the corresponding neural network for any one or any combination of such image classification, object tracking, optical flow estimation, depth estimation. Thus, in one or more such examples, as a non-limiting example, compared with performing such object recognition using a non-white balanced captured image, when a white balanced image is used for object recognition, confusion in object recognition due to lighting or illumination can be reduced, and robustness can be correspondingly improved.

[0149] Herein, for Figures 1 to 10The described image correction device, training device, electronic device, processor, NPU, hardware accelerator, image correction device 1000, image acquirer 1010, processor 1020, memory 1030, display 1040, and other devices, apparatuses, units, and components are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtracters, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer can be implemented by one or more processing elements (such as logic gate arrays, controllers, and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve the desired result). In one example, the processor or computer includes or is connected to one or more memories that store instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For the sake of brevity, the singular terms "processor" or "computer" may be used in the description of the examples described in this application, but in other examples, multiple processors or computers may be used, or the processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, can implement a single hardware component, or two or more hardware components. The hardware components can have any one or more of different processing configurations. Examples of different processing configurations include: single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0150] Figures 1 to 10The method of performing the operations described in this application, as shown, is performed by computing hardware (e.g., by one or more processors or computers), which is implemented to execute instructions or software as described above to perform the operations performed by the method described in this application. For example, a single operation, or two or more operations, may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may perform a single operation, or two or more operations.

[0151] Instructions or software for controlling computing hardware (e.g., as a non-limiting example, one or more processors or computers and one or more systolic arrays associated therewith) to implement the hardware components and perform the method as described above may be written as a computer program, code segment, instruction, or any combination thereof, to individually or jointly direct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and method as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by one or more processors or computers. In another example, the instructions or software include high-level code executed by one or more processors or computers using an interpreter. The instructions or software may be written in any programming language based on the block diagrams and flowcharts shown in the figures and the corresponding descriptions used herein, which disclose algorithms for performing the operations performed by the hardware components and method as described above.

[0152] Instructions or software for controlling computing hardware (e.g., one or more processors or computers and one or more systolic arrays associated therewith) to implement hardware components and perform the methods as described above, as well as any associated data, data files, and data structures, can be recorded, stored, or fixed in one or more non-transitory computer-readable storage media, or recorded, stored, or fixed on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDD), solid state drives (SSD), flash memory, cartridge memory (such as, multimedia card micro or card (e.g., secure digital (SD) or extreme digital (XD))), magnetic tape, floppy disk, magneto-optical data storage devices, optical data storage devices, hard disks, solid state disks, and any other device, any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers such that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed on a networked computer system such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed by one or more processors or computers in a distributed manner.

[0153] Although this disclosure includes specific examples, it will be apparent after understanding the disclosure of this application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered only as descriptive and not for purposes of limitation. The description of a feature or aspect in each example should be considered applicable to similar features or aspects in other examples. Appropriate results can be achieved if the described techniques are performed in a different order, and / or if the components in the described system, architecture, device, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

Claims

1. A processor-implemented method for image correction, the method comprising: generating a lighting map including lighting values from an input image using a neural network model, the lighting values depending on corresponding color casts caused by one or more light sources individually affecting each pixel of the input image; generating a white-adjusted image by removing at least a portion of the color cast from the input image using the generated lighting map; and generating a partial white balance image by adjusting the white-adjusted image, the partial white balance image having retained chromaticity information corresponding to a portion of the light source, and chromaticity information corresponding to the remaining portion of the light source being white information corresponding to the remaining portion of the light source.

2. The method according to claim 1, wherein The generated lighting map includes lighting vectors in which a partial lighting vector corresponding to the chromaticity of one light source among the light sources and another partial lighting vector corresponding to the chromaticity of another light source among the light sources are mixed for at least one pixel of the input image.

3. The method according to claim 1, wherein, The lighting map includes a mixing coefficient for the chromaticity of a light source affecting a first pixel of the input image, the mixing coefficient being different from another mixing coefficient for the chromaticity of a light source affecting a second pixel of the input image.

4. The method according to claim 1, wherein For a first pixel of the input image, the result of the neural network model includes first chromaticity information of one light source among the one or more light sources and a corresponding first mixing coefficient, and for a second pixel of the input image, the result of the neural network model further includes second chromaticity information of the one light source among the one or more light sources and a corresponding second mixing coefficient.

5. The method according to claim 4, wherein The step of generating the lighting map includes: generating a first lighting value of the lighting map corresponding to the first pixel based on the first chromaticity information and the first mixing coefficient, and generating a second lighting value of the lighting map corresponding to the second pixel based on the second chromaticity information and the second mixing coefficient.

6. The method according to claim 1, wherein The step of generating the white-adjusted image includes: generating the white-adjusted image by applying the lighting map to the input image through element-wise operations.

7. The method according to claim 6, wherein, The step of applying the lighting map to the input image through element-wise operations includes: dividing the corresponding pixel value of each pixel in the input image by the corresponding lighting value among the lighting values of the lighting map.

8. The method according to any one of claims 1 to 7, wherein, The white-adjusted image is a white balance image.

9. The method according to claim 1, further comprising: determining chromaticity information corresponding to each light source by decomposing the lighting map for each light source.

10. The method according to claim 1, further comprising: Controlling a display to display the partial white balance image.

11. The method according to claim 9, Among them, the determining further includes: determining a mixing coefficient corresponding to each light source by decomposing the lighting map for each light source, wherein the step of generating the partial white balance image includes: generating a partial lighting map corresponding to the portion of the light source using the chromaticity information corresponding to the portion of the light source and a corresponding mixing coefficient map for the portion of the light source; generating a remaining lighting map corresponding to the remaining portion of the light source using the white information corresponding to the remaining portion of the light source and a corresponding mixing coefficient map for the remaining portion of the light source; generating a re-illumination map by adding the partial lighting map and the remaining lighting map together; and Multiply the pixel value of each pixel of the white-adjusted image by the relighting value corresponding to each pixel among the relighting values of the relighting map.

12. The method according to claim 1, Among them, The step of generating a lighting map includes: Extracting the chromaticity information of each light source and the mixing coefficient map of each light source by applying a neural network model to the input image; and Generating a lighting map based on the extracted chromaticity information and the extracted mixing coefficient map.

13. The method according to claim 12, wherein, The step of generating a lighting map includes: Calculating the partial lighting map of each light source by multiplying the chromaticity information of each light source by the mixing coefficient map of each light source respectively; and Generating a lighting map by adding the calculated partial lighting maps together.

14. The method according to claim 12, wherein, The step of generating a partial white balance image includes: Determining the chromaticity information corresponding to the target light source among the extracted chromaticity information; Calculating the partial lighting map of the target light source by multiplying the determined chromaticity information by the corresponding mixing coefficient map corresponding to the target light source among the extracted mixing coefficient maps; Changing the chromaticity information corresponding to the remaining part of the light source in the extracted chromaticity information to white information; Generating the remaining lighting map corresponding to the remaining part of the light source using the white information corresponding to the remaining part of the light source and the corresponding mixing coefficient map corresponding to the remaining part of the light source; Generating a relighting map by adding the partial lighting map of the target light source and the remaining lighting map of the remaining part of the light source; and Generating a partial white balance image by applying the relighting map to the white-adjusted image, wherein at least a part of the color cast corresponding to the target light source is retained in the generated partial white balance image.

15. The method according to claim 14, wherein The step of determining the chromaticity information corresponding to the target light source includes: Changing the chromaticity information corresponding to the target light source to the target chromaticity information.

16. The method according to claim 1, further comprising: Determining the mixing coefficient map of each light source such that the sum of the mixing coefficients calculated for a pixel with respect to the light sources is a reference value; Calculating the partial lighting map corresponding to each light source respectively based on the chromaticity information of each light source and the mixing coefficient map of each light source; And Applying the calculated partial lighting map to one of the input image and the white-adjusted image.

17. The method according to claim 16, wherein, The step of determining the mixing coefficient map includes: Identifying the total number of light source chromaticities affecting the pixels in the input image; and Generating the corresponding mixing coefficient map corresponding to the identified total number of light source chromaticities.

18. The method according to claim 17, wherein, The step of identifying the total number of light source chromaticities includes: Setting the total number of light source chromaticities to three.

19. The method according to claim 17, wherein, The step of generating the corresponding mixing coefficient map includes: Initiating the generation of the corresponding mixing coefficient map in response to the total number of light source chromaticities being identified as two.

20. The method according to claim 16, wherein, The step of determining the mixing coefficient map includes: When the total number of light sources is two light sources, estimating the mixing coefficient map of one of the two light sources; and Calculating the remaining mixing coefficient map of the remaining other light source among the two light sources by subtracting each mixing coefficient in the estimated mixing coefficient map from the reference value for each pixel.

21. A non-transitory computer-readable storage medium storing instructions, which when executed by a processor cause the processor to execute the method for image correction according to any one of claims 1 to 20.

22. An apparatus for image correction, comprising: a processor; and a memory storing one or more machine learning models and instructions which, when executed by the processor, configure the processor to: generate a lighting map including lighting values using one of the one or more machine learning models, the lighting values depending on corresponding color casts caused by light sources that individually affect pixels of an input image; and generate a white-adjusted image by adjusting at least a portion of the color cast from the input image using the generated lighting map, wherein the processor is further configured to generate a partial white balance image by adjusting the white-adjusted image, and the partial white balance image has retained chromaticity information corresponding to a portion of the light source and different chromaticity information corresponding to the remaining portion of the light source compared to corresponding chromaticity information in the lighting map.

23. The apparatus according to claim 22, wherein, The different chromaticity information is white information.

24. The apparatus according to claim 22, wherein, The processor is further configured to: decompose the lighting map for each light source to obtain chromaticity information of the light source.

25. The device according to claim 24, wherein, The different chromaticity information is white information.

26. The apparatus according to any one of claims 22 to 25, further comprising: an image sensor configured to capture the input image.

27. The apparatus according to claim 26, wherein, The instructions further include instructions which, when executed by the processor, configure the processor to control the operation of the apparatus based on the white-adjusted image.

28. The apparatus according to any one of claims 22 to 25, Among them, wherein the instructions further include instructions which, when executed by the processor, configure the processor to identify the total number of light sources affecting pixels in the input image, and wherein the step of using the one of the one or more machine learning models includes: selecting the one machine learning model from among the one or more machine learning models stored in the memory based on the identified total number of light sources, wherein the one machine learning model is trained for the identified total number of light sources.

29. An apparatus for image correction, comprising: an image sensor configured to obtain an input image; and a processor configured to: generate a lighting map including lighting values from the input image using a neural network model, the lighting values depending on corresponding color casts caused by one or more light sources that individually affect each pixel of the input image; generate a white balance image by removing at least a portion of the color cast from the input image using the generated lighting map; and generate a partial white balance image by adjusting the white-adjusted image, and the partial white balance image has retained chromaticity information corresponding to a portion of the light source and different chromaticity information corresponding to the remaining portion of the light source compared to corresponding chromaticity information in the lighting map.

30. The apparatus according to claim 29, further comprising a display, Among them, wherein the processor is further configured to cause the display to display the white balance image.

Images