Electronic device and method performed by electronic device

Abstract

An electronic device and a method performed by the electronic device are disclosed. The method includes: extracting raw light source information representing a raw light source color shift of an input image from the input image, the extracting being performed by applying a light source information extraction model to the input image; generating intermediate light source information representing an intermediate light source color shift of the input image from the original light source information, the step of generating being performed by applying a light source information generation model to the original light source information; determining target light source information based on the extracted original light source information and based on the generated intermediate light source information; and generating an output image by applying the determined target light source information to the input image.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2023-0171076, filed on Nov. 30, 2023, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to image correction technology. Background Art

[0003] Performing white balance (WB) on an image of a scene may require estimating the color of a light source in the scene to eliminate a color-cast caused by the light source from the scene. WB mimics the color constancy of the human visual system and is one of the core elements of an in-camera imaging pipeline for providing visually satisfactory imaging. WB or computational color constancy has long been a topic of discussion in the field of computer vision. Summary of the Invention

[0004] The Summary of the Invention is provided to introduce, in brief form, a selection of concepts that will be further described in the Detailed Description below. The Summary of the Invention is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to assist in determining the scope of the claimed subject matter.

[0005] In one general aspect, a method performed by an electronic device includes: extracting, from an input image, original light source information representing an original light source color-cast of the input image, the extracting step being performed by applying a light source information extraction model to the input image; generating, from the original light source information, intermediate light source information representing an intermediate light source color-cast of the input image, the generating step being performed by applying a light source information generation model to the original light source information; determining target light source information based on the extracted original light source information and based on the generated intermediate light source information; and generating an output image by applying the determined target light source information to the input image.

[0006] The step of determining the target light source information may include: combining at least a part of the original light source information and at least a part of the intermediate light source information.

[0007] The step of generating the output image may include: eliminating the intermediate light source color-cast from the input image and adding a color-cast of the target light source information to the input image.

[0008] The step of determining the target light source information may include: obtaining a user input and controlling a contribution of the original light source information or the intermediate light source information to the target light source information according to the user input.

[0009] The method may further include: outputting an embedding vector generated by a light source information extraction model when extracting original light source information from an input image, wherein intermediate light source information is generated by applying a light source information generation model to the embedding vector and the original light source information of the input image.

[0010] The method may further include: determining whether to apply the light source information extraction model and the light source information generation model to the input image based on the color diversity of the input image.

[0011] The step of determining whether to apply the light source information extraction model and the light source information generation model to the input image based on the color diversity of the input image may include: performing linear regression on the pixel values of the pixels of the input image; and determining a color diversity score based on the difference between the result of the linear regression and the pixel values, the color diversity score indicating the color diversity.

[0012] The step of determining whether to apply the light source information extraction model and the light source information generation model to the input image based on the color diversity of the input image may include: determining a color diversity score of the input image by applying a color diversity determination model to the input image.

[0013] The method may further include: determining whether to apply the light source information extraction model and the light source information generation model to the input image based on the score of the gray pixels corresponding to achromatic objects in the input image.

[0014] The method may further include: determining candidate gray pixels among the pixels of the input image based on the pixel values and the gray scale trajectory mapped to the achromatic object; detecting pseudo-gray pixels corresponding to chromatic objects among the determined candidate gray pixels; and based on the detected pseudo-gray pixels, performing any one of the following steps: generating original light source information, generating intermediate light source information, determining target light source information, and generating an output image.

[0015] The light source information extraction model may include a first convolutional neural network, and the light source information generation model may include a second convolutional neural network.

[0016] In another general aspect, an electronic device includes: one or more processors; and a memory storing instructions configured to cause the one or more processors to: extract original light source information representing the original light source color deviation of the input image from the input image, the extracting step being performed by applying a light source information extraction model to the input image; generate intermediate light source information representing the intermediate color deviation of the input image by applying a light source information generation model to the original light source information; determine target light source information based on the extracted original light source information and based on the generated intermediate light source information; and generate an output image by applying the determined target light source information to the input image.

[0017] The instructions may also be configured to cause the one or more processors to determine target light source information by combining at least a portion of the original light source information and at least a portion of the intermediate light source information.

[0018] The instructions may also be configured to cause the one or more processors to remove intermediate light source color cast from the input image and add the color cast of the target light source information to the input image.

[0019] The instructions may also be configured to cause the one or more processors to obtain user input and control the contribution of the original light source information or the intermediate light source information to the target light source information based on the user input.

[0020] The instructions may also be configured to cause the one or more processors to output an embedding vector generated by a light source information extraction model when extracting the original light source information from the input image; and generate intermediate light source information by applying a light source information generation model to the embedding vector of the input image and the original light source information.

[0021] The instructions may also be configured to cause the one or more processors to determine whether to apply the light source information extraction model and the light source information generation model to the input image based on the color diversity of the input image.

[0022] The instructions may also be configured to cause the one or more processors to perform linear regression on the pixel values of the pixels of the input image; and determine a color diversity score based on the difference between the result of the linear regression and the pixel values, the color diversity score indicating the color diversity.

[0023] The instructions may also be configured to cause the one or more processors to determine whether to apply the light source information extraction model and the light source information generation model to the input image based on the scores of the gray pixels corresponding to achromatic objects in the input image.

[0024] The instructions may also be configured to cause the one or more processors to determine candidate gray pixels among the pixels in the input image based on the pixel values and the gray scale trajectory mapped to the achromatic object; detect pseudo-gray pixels corresponding to colored objects among the determined candidate gray pixels; and perform any one of the following steps based on the detected pseudo-gray pixels: generate the original light source information, generate the intermediate light source information, determine the target light source information, and generate the output image.

[0025] Other features and aspects will be apparent from the following detailed description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 An example image correction method according to one or more example embodiments is shown.

[0027] Figure 2 Illustrates an example of generating an output image by correcting an input image according to one or more example embodiments.

[0028] Figure 3 Illustrates an example of generating an output image based on the color diversity of an input image according to one or more example embodiments.

[0029] Figure 4 Illustrates an example of generating an output image based on the grayscale pixel correlation score of an input image according to one or more example embodiments.

[0030] Figure 5 Illustrates an example of correcting an image using pseudo grayscale pixels according to one or more example embodiments.

[0031] Figure 6 Illustrates an example of correcting an image using multiple white balance (WB) methods according to one or more example embodiments.

[0032] Figure 7 Illustrates an example configuration of an electronic device according to one or more example embodiments.

[0033] Throughout the drawings and the detailed description, unless otherwise described or provided, the same or similar reference numerals will be understood to refer to the same or similar elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated. Detailed Description

[0034] The following detailed description is provided to assist the reader in obtaining a comprehensive understanding of the methods, devices, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be apparent after understanding the disclosure of this application. For example, the order of operations described herein is merely exemplary and is not limited to those set forth herein, but may be changed as will be apparent after understanding the disclosure of this application, except for operations that must occur in a specific order. Additionally, descriptions of features known after understanding the disclosure of this application may be omitted for greater clarity and conciseness.

[0035] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Instead, the examples provided herein are only to illustrate some of the many possible ways of implementing the methods, devices, and / or systems described herein that will be apparent after understanding the disclosure of this application.

[0036] The terms used herein are for describing various examples only and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular is also intended to include the plural. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more of them. As a non-limiting example, the terms "comprising", "including", and "having" indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0037] Throughout the specification, when a component or element is described as "connected to", "coupled to", or "joined to" another component or element, the component or element can be directly "connected to", "coupled to", or "joined to" the other component or element, or one or more other components or elements can reasonably exist in between. When a component or element is described as "directly connected to", "directly coupled to", or "directly joined to" another component or element, no other elements can exist in between. Similarly, expressions such as "between" and "immediately between" and "adjacent to" and "immediately adjacent to" can also be interpreted as described above.

[0038] Although terms such as "first", "second", and "third" or A, B, (a), (b), etc. may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. Each of these terms is not used to define, for example, the nature, order, or sequence of the corresponding component, assembly, region, layer, or part, but is only used to distinguish the corresponding component, assembly, region, layer, or part from other components, assemblies, regions, layers, or parts. Thus, without departing from the teachings of the examples, the first component, first assembly, first region, first layer, or first part mentioned in the examples described herein can also be referred to as the second component, second assembly, second region, second layer, or second part.

[0039] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains based on the understanding of the present application's disclosure. Unless explicitly defined as such herein, terms (such as those defined in common dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present application's disclosure, and should not be interpreted in an idealized or overly formal sense. The use of the term "may" herein with respect to an example or embodiment (e.g., what an example or embodiment may include or implement) indicates that there is at least one example or embodiment that includes or implements such a feature, while all examples are not limited thereto.

[0040] Due to the illumination of a light source (e.g., an incandescent lamp emitting warm light), a white object may be sensed as a color other than white (e.g., light yellow) by a camera sensor and / or an image sensor. In digital image processing, white balance (WB) aims to reproduce in an image the phenomenon that "a person perceives an object having an actual chromaticity corresponding to white in the image as white". WB can correct color distortion in an image by eliminating color shifts occurring due to the chromaticity of the light source throughout the image. As described below, an image correction method according to one or more example embodiments provided for performing WB can convert an input image into a white-balanced output image based on an inference performed on the input image by a neural network model.

[0041] Figure 1 An example of an image correction method according to one or more example embodiments is shown.

[0042] In operation 110, the electronic device may extract original light source information from the input image by applying a light source information extraction model. The term "original" is used here to distinguish it from other light source information (e.g., "intermediate" light source information); this term is not intended to otherwise characterize the original light source information. Similar remarks apply to "intermediate"; such information may also be referred to as "first light source information" and "second light source information". For example, the electronic device may extract original light source information by applying a light source information extraction model to the input image, and the light source information extraction model performs an inference on the input image to generate the original light source information.

[0043] The input image may be an image captured based on light from an original light source (e.g., a lamp, the sun, etc.) reflected from a scene. The input image includes pixels. Each pixel may have multiple values. For example, the pixel values may include an R value corresponding to red, a G value corresponding to green, and a B value corresponding to blue. Incidentally, "original light source information" may represent any light source aspect of the input image and does not necessarily represent the actual light source of the scene captured in the input image.

[0044] Each pixel of the input image may have a pixel value in which the chromaticity of the original light source reflected from the object according to the chromaticity of the object. The different chromaticities of different objects are reflected in the pixel values of the pixels corresponding to those objects. The color shift of the light source may be the influence of the chromaticity of the light source on the pixel values of the image. For example, compared with the G value of a pixel when an image is captured "when the scene receives light from a first light source having a chromaticity corresponding to white", when capturing a corresponding image of "a scene receiving light from a second light source having a chromaticity corresponding to green", the pixel may have a pixel value including a larger G value.

[0045] The original light source information may be information indicating the chromaticity and / or color of the original light source of at least a part of the scene captured in the input image. The original light source information may include a color cast of the original light source applied to the input image. According to one or more example embodiments, the original light source information may include at least one of an R value, a G value, and a B value indicating the chromaticity and / or color of the original light source. According to one or more example embodiments, the original light source information may indicate the chromaticity and / or color of the original light source of a corresponding partial region of the input image. The original light source information may include pixels corresponding to the pixels of the input image respectively, and each pixel of the original light source information may have a pixel value including an R value, a G value, or a B value indicating the chromaticity and / or color of the original light source.

[0046] The foregoing light source information extraction model may be a model that is generated and / or trained to output corresponding original light source information associated with the original light source applied to an arbitrary input image (by inference). According to one or more example embodiments, the light source information extraction model may be implemented based on a machine learning model and may be / is included, for example, a neural network (e.g., a convolutional neural network (CNN)).

[0047] According to one or more example embodiments, a neural network may include nodes in multiple layers. The nodes in each layer may have connections to the nodes in adjacent layers. The neural network may also include connection weights (e.g., parameters) of the corresponding connections between the nodes. The layers of the neural network may include, for example, an input layer, one or more hidden layers, and an output layer. In the neural network, input data may start from the input layer and propagate through multiple layers to the output layer. During the propagation, abstract feature data (e.g., feature vectors or feature maps) may be extracted from the input data (e.g., the input image), and output data may be generated from the feature data.

[0048] According to one or more example embodiments, training data may be used to train a light source information extraction model, the training data including training input images associated with corresponding ground truth (GT) labels indicating original light source information of the training input images. When the light source information extraction model is being trained (hereinafter referred to as the "temporary light source information extraction model") and applied to the training input images, temporary original light source information may be output. The temporary light source information extraction model may be trained by supervised learning using the GT original light source information as the GT. For example, a loss may be calculated based on the difference between the temporary original light source information and the GT original light source information. The parameters of the temporary light source information extraction model may be iteratively updated based on the calculated loss such that the loss converges to a value less than a predetermined value (e.g., a cutoff threshold). This update process may be performed for each training input image. After the iterative update of the parameters of the temporary light source information extraction model, the update of the parameters of the temporary light source information extraction model may be stopped, and the temporary light source information extraction model may be used as the trained light source information extraction model. As mentioned below, the temporary light source information extraction model may also be trained to output an embedding vector.

[0049] According to one or more example embodiments, when an input image is applied to the trained light source information extraction model, the light source information extraction model of the electronic device may infer and output both (i) the original light source information and (ii) an embedding vector of the input image. The embedding vector may include information related to generating the original light source information by the light source information extraction model.

[0050] The embedding vector of the input image may be / is an vector related to the attributes of the input image. The attributes of the input image may include semantic information about the scene in the input image. For example, the attributes of the input image that may be inferred by the trained light source information extraction model may include: the category of the object in the input image, the atmosphere of the scene in the input image, and / or the texture of at least a part of the input image. When at least one convolutional layer and / or at least one fully connected layer are applied to the input image, the embedding vector of the input image may be obtained.

[0051] As described with respect to operation 120, the embedding vector of the input image may be used to generate intermediate light source information. For example, the embedding vector of the input image may be obtained from the output layer or a hidden layer of the light source information extraction model.

[0052] In operation 120, the electronic device may generate intermediate light source information from the original light source information by applying a light source information generation model.

[0053] The intermediate light source information may indicate a color deviation of a light source (also referred to herein as the "intermediate light source") that will be applied to the input image through a process described later. When applied to the input image based on the original light source information and / or the attributes of the input image, the intermediate light source may be a light source that is popular and / or preferred by a specific user. However, the intermediate light source is not limited thereto, and the intermediate light source may be any arbitrary light source. Similarly, the intermediate light source information needs to represent any specific light source (real or otherwise), and conversely, its functional role within the entire method / device is noteworthy.

[0054] According to one or more example embodiments, the intermediate light source information may include an R value, a G value, and / or a B value indicating the chromaticity of the intermediate light source. According to one or more example embodiments, the intermediate light source information may indicate the chromaticity of the intermediate light source in a corresponding partial region of the input image. The intermediate light source information may include pixels corresponding to the pixels of the input image, and each pixel of the intermediate light source information may have a pixel value including an R value, a G value, or a B value indicating the chromaticity of the intermediate light source.

[0055] For example, when a scene including a lawn is included in the input image, the user may more strongly express the green color of the lawn, and thus it can be known that they prefer to express the lawn more clearly. In this case, the intermediate light source may be determined as a light source having a chromaticity corresponding to accurately expressing the green color of the lawn (e.g., the "true" color). For example, when a scene including the sky is included in the input image, the user may more strongly express the blue color of the sky, and thus it can be known that they prefer to express the sky more clearly. In this case, the intermediate light source may be determined as a light source having a chromaticity corresponding to accurately expressing the blue color of the sky. In various example embodiments of the present disclosure, the "intermediate light source" may also be referred to as the "preferred light source" because it is generally determined as the light source preferred by the user according to the attributes of the input image. However, the preferred light source may be determined in any manner.

[0056] According to one or more example embodiments, the electronic device may generate intermediate light source information based on the input image and the original light source information. For example, the electronic device may generate intermediate light source information from an embedding vector of the input image and the original light source information (both based on the input image). For example, the electronic device may generate intermediate light source information by applying a light source information generation model to the embedding vector of the input image and the original light source information.

[0057] The light source information generation model may be generated and / or trained to output intermediate light source information from the input image (or the embedding vector of the input image) and the original light source information. According to one or more example embodiments, the light source information generation model may be implemented based on a machine learning model and may be / is included, for example, a neural network (e.g., a convolutional neural network (CNN)).

[0058] According to one or more example embodiments, training data may be used to train a light source information generation model, the training data including "training input images" (or embedded vectors of training input images), "training original light source information" mapped to the training input images, and "GT target light source information" mapped to the training input images. Regarding training, the light source information generation model (hereinafter, referred to as the "temporary light source information generation model") is applied to the training input images (or embedded vectors of training input images) and the training original light source information, and temporary intermediate light source information may be output. In this case, the temporary target light source information may be determined based on the training original light source information and based on the temporary intermediate light source information. The weights for the "training original light source information" and the weights for the "temporary intermediate light source information" used to determine the temporary target light source information may be learned as learnable parameters in the training operation of the light source information generation model. The temporary light source information generation model and / or the temporary weights (e.g., the weights for the original light source information being positively learned and the weights for the intermediate light source information being positively learned) may be trained by using the GT target light source information as GT for supervised learning. For example, a loss may be calculated based on the difference between the temporary target light source information and the GT target light source information. In this case, the parameters of the temporary light source information generation model may be iteratively updated based on the calculated loss such that the loss converges to a value less than a predetermined value. After the iterative update of the parameters of the temporary light source information generation model, the update of the parameters of the temporary light source information generation model may be stopped, and the temporary light source information generation model may be used as the light source information generation model.

[0059] Although the light source information extraction model and the light source information generation model are described as being independently trained according to various example embodiments, the examples are not limited thereto. According to one or more example embodiments, the temporary light source information extraction model and the temporary light source information generation model may be trained together in a dependent manner. For example, the training data may include training input images, GT original light source information mapped to the training input images, and GT target light source information mapped to the training input images. When the temporary light source information extraction model is applied to the training input images, a temporary embedded vector and temporary original light source information of the training input images may be obtained by inference. When the temporary light source information generation model is applied to the temporary embedded vector and the temporary original light source information, temporary intermediate light source information may be obtained. When the temporary weights are applied to the temporary original light source information (or GT original light source information) and the temporary intermediate light source information, temporary target light source information may be obtained. A loss may be calculated based on the difference between the GT original light source information and the temporary original light source information and the difference between the temporary target light source information and the GT target light source information. The parameters of the light source information extraction model, the parameters of the light source information generation model, or the weights may be learned or trained by iterative update based on the calculated loss.

[0060] In operation 130, the electronic device may determine target light source information based on the extracted original light source information and the generated intermediate light source information. For example, the electronic device may determine the target light source information by combining a part of the original light source information and a part of the intermediate light source information. In this case, the target light source may be used as the light source applied to the input image instead of the original light source.

[0061] According to one or more example embodiments, the target light source information may include R values, G values, and / or B values indicating the chromaticity of the target light source. According to one or more example embodiments, the target light source information may indicate the chromaticity of the target light source for each partial region of the input image. The target light source information may include pixels corresponding to the pixels of the input image, and each pixel of the target light source information may have a pixel value including R values, G values, and / or B values indicating the chromaticity of the target light source.

[0062] According to one or more example embodiments, the electronic device may obtain a user input specifying weights for the original light source information and / or the intermediate light source information. The electronic device may determine the target light source information by combining the original light source information and the intermediate light source information based on the user input (e.g., based on the specified weights).

[0063] For example, the electronic device may determine the weights for each of the original light source information and the intermediate light source information based on the user input. The electronic device may combine a part of the original light source information and a part of the intermediate light source information based on the determined weights. Alternatively, as described elsewhere herein, the electronic device may copy one or the other light source information (in whole or in part) to the target light source information.

[0064] For example, when the user desires to correct the original light source under which the input image was captured, the user may set a first weight biased towards the original light source information. For example, when the user desires to correct another light source (e.g., a light source similar to the intermediate light source), the user may set a second weight biased towards the intermediate light source information. For example, the weights may be in the range of 0 to 1, where 1 causes only the original light source information to be used, 0.5 causes an equal mixture of the original light source and the intermediate light source to be used, and 0 causes only the intermediate light source to be used.

[0065] According to one or more example embodiments, the sum of the weight for the original light source information and the weight for the intermediate light source information may be a constant (e.g., 1), and the weight for the original light source information and the weight for the intermediate light source information may have values greater than or equal to zero (0) or less than or equal to 1.

[0066] For example, the pixel value of a pixel of the target light source information may be determined based on Equation 1 below.

[0067] Equation 1

[0068] L target = αL ori +(1 - α)L inter

[0069] In Equation 1, L ori represents the original light source information, and L inter represents the intermediate light source information. α represents the weight for the original light source information, and (1 - α) represents the weight for the intermediate light source information. L target represents the target light source information.

[0070] According to one or more example embodiments, assuming that the original light source information, the intermediate light source information, and the target light source information each include a corresponding plurality of pixels, the weight for the original light source information and the weight for the intermediate light source information can be determined independently (e.g., differently) based on the plurality of pixels. For example, to determine the pixel value of each pixel of the target light source information, the electronic device can apply the weight for the original light source information to the pixel value of the original light source information, and can apply the weight for the intermediate light source information to the corresponding pixel value of the intermediate light source information.

[0071] According to one or more example embodiments, the electronic device can determine the target light source information based on the bias of the sensor (e.g., a camera sensor) used to obtain the input image. The bias of the sensor can include the R value, the G value, and the B value measured by the sensor for an achromatic - color object (also simply referred to as an "achromatic object" herein) under standard illumination (e.g., D50 illumination, D55 illumination, D65 illumination, etc.). In some embodiments, the bias can be included in the metadata of the input image. For example, each pixel value of the target light source information can be determined based on Equation 2 below.

[0072] Equation 2

[0073]

[0074] In Equation 2, L ori represents the original light source information, and L inter represents the intermediate light source information. α represents the weight for the original light source information, and (1 - α) represents the weight for the intermediate light source information. L sens represents the sensor bias, and L target represents the target light source information.

[0075] However, various example embodiments of the present disclosure are not limited to combining the original light source information and the intermediate light source information based on weights; the electronic device may determine the original light source information or the intermediate light source information as the target light source information based on regions in the input image. That is, the target light source information 270 may be made up of different parts of the original light source information and the intermediate light source information.

[0076] For example, a first region of the input image may correspond to the original light source information, and a second region of the input image (e.g., the remaining region of the input image other than the first region) may correspond to the intermediate light source information. Pixel values of pixels corresponding to the first region of the input image (among the pixels of the target light source information) may be determined based on the pixel values of the same pixels in the original light source information. For example, it may be set to be the same as the pixel value of the same corresponding pixel in the original light source information. Pixel values of pixels corresponding to the second region of the input image (among the pixels of the target light source information) may be determined based on the pixel values of the same pixels in the intermediate light source information. For example, it may be set to be the same as the pixel value of the same corresponding pixel in the intermediate light source information.

[0077] As can be understood, any one of various mixing / merging / combining techniques may be used to construct the target light source information from the original light source information and the intermediate light source information. For example, mixing may be used, or all values of one or the other may be used. Such techniques may be applied to the entire target light source information or a part thereof. Mixing may be used for a part of the target light source information, and all values of either light source may be used for another part of the target light source. In addition, regions of the target light source information (for receiving the mixing of two light sources or the replication of either light source) may be defined by object detection / recognition (e.g., using a segmentation algorithm). Additionally, although user input is mentioned as a way to control the mixing / replication of source light data, other ways may be used. For example, heuristics, image processing algorithms, etc. may be used to select one source or the other, set the mixing, determine where the mixing or replication will be applied, etc.

[0078] In operation 140, the electronic device may generate an output image based on the result of applying the target light source information to be determined to the input image.

[0079] The output image may be an image of the same scene as in the input image but corrected according to the input image. The output image may also be referred to as a corrected or adjusted version of the input image (referring to the correction / adjustment of the pixel values of the input image to form the output image).

[0080] The electronic device may eliminate the color cast of the original light source from the input image and add the color cast of the target light source information to the input image.

[0081] For example, the pixel value of each pixel of the output image may be determined based on Equation 3:

[0082] Equation 3

[0083]

[0084] In Equation 3, I AWB (where AWB represents automatic white balance) represents the pixel value of the output image, R represents the pixel value obtained by removing the color cast of the original light source from the input image, and L ori represents the original light source information. Additionally, R·L ori represents the pixel value of the input image. L target represents the target light source information, and L inter represents the intermediate light source information. α represents the weight for the original light source information, and (1-α) represents the weight for the intermediate light source information.

[0085] Figure 2 Illustrates an example operation of generating an output image by correcting an input image according to one or more example embodiments.

[0086] According to one or more example embodiments, an electronic device may apply a light source information extraction model 220 to an input image 210 to obtain an embedding vector 230 and original light source information 240 of the input image 210. The electronic device may apply a light source information generation model 250 to the embedding vector 230 and the original light source information 240 of the input image 210 to output intermediate light source information 260. The electronic device may combine the original light source information 240 and the intermediate light source information 260 to generate target light source information 270. The electronic device may remove the color cast of the original light source from the input image 210 based on the original light source information 240 and add the color cast of the target light source based on the target light source information 270, thereby generating an output image 280.

[0087] Figure 3 Illustrates an example operation of generating an output image based on the color diversity of an input image according to one or more example embodiments.

[0088] According to one or more example embodiments, an electronic device may determine whether to apply a light source information extraction model (e.g., light source information extraction model 220) and a light source information generation model (e.g., light source information generation model 250) to an input image 310 based on a color diversity score corresponding to the color diversity of the input image 310. The color diversity of the input image 310 may be, for example, the degree of diversity of colors indicated by the pixel values of the pixels in the input image 310. In Figure 3 the example, the input image 310 is an image of a test pattern object.

[0089] When the color diversity score 330 is less than or equal to the threshold score, the electronic device may determine to apply the light source information extraction model and the light source information generation model. When the color diversity score 330 is greater than the threshold score, the electronic device may determine not to apply the light source information extraction model and the light source information generation model. For example, when the color diversity score 330 is greater than the threshold score, the electronic device may generate an output image from the input image 310 by applying the AWB method.

[0090] The statistics-based AWB method may adjust the pixel values of the pixels of the input image 310 based on the statistical values of the pixel values of the pixels of the input image 310. As a non-limiting example, the statistics-based AWB method may include gray-world, gray-edge, shade of gray, and gray-pixel. The statistics-based AWB method may have a relatively low complexity and a fast algorithm operation speed, but when the input image 310 does not conform to the statistical model corresponding to the method based on statistics of the AWB, the white balance performance of the statistics-based AWB method may be significantly reduced.

[0091] When the color diversity of the input image 310 is high, the WB performance of the operation of correcting the input image 310 to the output image by the statistics-based AWB method may also be high. On the contrary, when the color diversity of the input image 310 is low (for example, when the input image 310 is obtained by capturing a scene mainly including green objects), the statistics-based AWB method may misidentify the pixels of the input image 310 as having similar chromaticities due to the color cast of the original light source, and thus the WB performance of the operation of correcting the input image 310 to the output image may be low. Therefore, according to one or more example embodiments, in a case where the statistics-based AWB method is expected to operate with a relatively high WB performance, the electronic device may apply the statistics-based AWB method with a small amount of computation. Similarly, in a case where the statistics-based AWB method is expected to operate with a relatively low WB performance, the electronic device may perform the WB method based on the light source information extraction model and the light source information generation model with a high WB performance.

[0092] According to one or more example embodiments, the color diversity may be determined based on a linear regression performed on the pixel values. Figure 3An example linear regression 320 result is shown. The electronic device may perform linear regression on the pixel values of the pixels of the input image 310. According to one or more example embodiments, the logarithmic error LE (e.g., LE = 0.68343) may be used to measure the prediction accuracy of the linear regression. Each pixel may have a pixel value that has an R value, a G value, and a B value. For example, the electronic device may perform normalization on the pixel value based on the G value. The electronic device may obtain a first value (also referred to herein as "R / G value") obtained by dividing the R value by the G value and a second value (also referred to herein as "B / G value") obtained by dividing the B value by the G value. For each pixel, the electronic device may obtain a point on a plane following an orthogonal coordinate system where (i) the coordinate of the first axis (e.g., x-axis) is set to the first value (e.g., R / G value) and (ii) the coordinate of the second axis (e.g., y-axis) is set to the second value (e.g., B / G value). The electronic device may obtain the coordinates of the points of the corresponding pixels of the input image 310 in this way. The electronic device may perform linear regression 320 on these points. The linear regression 320 may involve calculations for modeling the linear correlation between the coordinates of the first axis and the coordinates of the second axis. The electronic device may obtain parameters indicating the linear relationship between the coordinates of the first axis and the coordinates of the second axis by performing linear regression 320.

[0093] For example, the electronic device may obtain the linear parameters according to Equation 4:

[0094] Equation 4

[0095] y = a + b·x

[0096] In Equation 4, x represents the coordinate on the first axis, y represents the coordinate on the second axis, and a and b represent the coefficients obtained by performing linear regression 320.

[0097] The electronic device may determine a color diversity score 330 indicating color diversity based on the difference between the result of the linear regression 320 and the pixel value (represented by a point). For example, when the difference between the result of the linear regression 320 and the pixel value (point) increases, the electronic device may determine the color diversity score 330 to be larger.

[0098] For example, the electronic device may obtain the color diversity score 330 according to Equation 5:

[0099] Equation 5

[0100]

[0101] In Equation 5, CD represents the color diversity score 330. x i represents the coordinate of point i on the first axis, and y iRepresents the coordinates of point i on the second axis. a and b represent coefficients obtained by performing linear regression 320 (according to Equation 4).

[0102] Although according to various example embodiments, the color diversity score 330 is described as being mainly determined based on linear regression 320, the examples are not limited thereto.

[0103] According to one or more example embodiments, the electronic device may perform at least one of variance, standard deviation, principal component analysis, and clustering (e.g., K-means clustering) on the pixel values of the pixels of the input image 310, and determine the color diversity score 330 based on the result of the performance.

[0104] According to one or more example embodiments, the electronic device may determine the color diversity score 330 of the input image 310 by applying a color diversity determination model to the input image 310.

[0105] The color diversity decision model may be generated and / or trained to output the color diversity score 330 of the input image 310 from the input image 310. According to one or more example embodiments, the color diversity determination model may be implemented based on a machine learning model and may be / is included, for example, a neural network (e.g., CNN).

[0106] According to one or more example embodiments, training data may be used to train the color diversity determination model, the training data including training input images 310 and GT color diversity scores mapped to the training input images 310 (the training data includes a plurality of such image-GT pairs). For example, the GT color diversity score may be calculated based on linear regression 320, variance, standard deviation, principal component analysis, or clustering performed on the pixel values of the pixels of the training input image 310. When the color diversity determination model being trained (hereinafter referred to as the "temporary color diversity determination model") is applied to the training input image 310, the temporary color diversity score (inferred from the training input image 310) may be output. The temporary color diversity determination model may be trained by supervised learning using the GT color diversity score as the GT value. For example, the loss may be calculated based on the difference between the temporary color diversity score and the GT color diversity score. In this case, the parameters of the temporary color diversity determination model may be iteratively updated based on the calculated loss such that the loss converges to a value less than a predetermined value. After the iterative update of the parameters of the temporary color diversity determination model, the update of the parameters of the temporary color diversity determination model may be stopped, and the temporary color diversity determination model may be obtained as the color diversity determination model. According to various example embodiments, the color diversity determination model is mainly described as being trained by supervised learning, but the examples are not limited thereto, and the color diversity determination model may be trained and obtained by unsupervised learning.

[0107] Figure 4 An example of generating an output image based on the grayscale pixel correlation score of an input image according to one or more example embodiments is shown.

[0108] According to one or more example embodiments, an electronic device may determine whether to apply a light source information extraction model and a light source information generation model to an input image 410 based on a score 430 of grayscale pixels in the input image 410 (also referred to herein as "grayscale pixel correlation score"). The grayscale pixels may correspond to achromatic objects. The grayscale pixel correlation score 430 may be related to pixels of grayscale pixels determined to correspond to true achromatic objects (also referred to herein as "candidate grayscale pixels"). The candidate grayscale pixels may be true grayscale pixels or pseudo grayscale pixels. The true grayscale pixels may correspond to true achromatic objects among the pixels of the input image 410. The pseudo grayscale pixels may correspond to chromatic-color objects (also referred to herein as "chromatic objects"), but appear to correspond to achromatic objects due to the color shift of the original light source. According to one or more example embodiments, as the ratio of pseudo grayscale pixels (or so-estimated pseudo grayscale pixels) among the candidate grayscale pixels of the input image 410 increases, the grayscale pixel correlation score 430 may increase accordingly.

[0109] When the grayscale pixel correlation score 430 is greater than a threshold score, the electronic device may determine to apply the light source information extraction model and the light source information generation model. When the grayscale pixel correlation score 430 is less than or equal to the threshold score, the electronic device may determine not to apply the light source information extraction model and the light source information generation model. For example, when the grayscale pixel correlation score 430 is less than or equal to the threshold score, the electronic device may generate an output image from the input image 410 by applying the statistical-based AWB method as described above.

[0110] When the grayscale pixel correlation score 430 of the input image 410 is relatively low, the performance of correcting the input image 410 to an output image by the statistical-based AWB method may be high. On the contrary, when the grayscale pixel correlation score 430 of the input image 410 is relatively high (e.g., when the probability that the candidate grayscale pixels are pseudo grayscale pixels is high), the statistical-based AWB method may misidentify the pseudo grayscale pixels as true grayscale pixels and accordingly determine the original light source information, so the WB performance of correcting the input image 410 to an output image may be low. Therefore, according to one or more example embodiments, when it is expected that the statistical-based AWB method operates with high WB performance, the electronic device may apply the statistical-based AWB method with a small amount of computation. Similarly, when it is expected that the statistical-based AWB method operates with low WB performance, the electronic device may perform the WB method based on a light source information extraction model and a light source information generation model with high performance.

[0111] Candidate gray pixels can be determined based on a gray locus. The gray locus is a range of pixel values of pixels corresponding to achromatic objects. The gray locus may include pixel values obtained when multiple original light sources are respectively applied to an achromatic object. For example, the gray locus can be obtained as a line corresponding to an achromatic object or an inner region of the line in a coordinate space including two coordinates (e.g., R / G coordinate and B / G coordinate) corresponding to pixel values normalized to the G value. The gray locus can be experimentally determined based on the results of obtaining images from original light sources of various chromaticities for an achromatic object (e.g., an image patch). Optionally, the gray locus can be predetermined for a sensor (e.g., a camera sensor). The electronic device can determine pixels whose points (obtained by normalizing their pixel values to the G value) are included in the gray locus as candidate gray pixels among the pixels of the input image 410.

[0112] According to one or more example embodiments, the electronic device can output a gray pixel-related score 430 of the input image 410 by applying a gray score determination model 420 to the input image 410.

[0113] The gray score determination model 420 can be generated and / or trained to output a gray pixel-related score 430 of the input image 410 from the input image 410. According to one or more example embodiments, the gray score determination model 420 can be implemented based on a machine learning model and can be / is, for example, a neural network (e.g., a CNN). According to one or more example embodiments, the gray score determination model 420 can be trained through supervised learning based on training input images and GT scores mapped to the training input images.

[0114] The GT score can be obtained based on the training input image and the original light source information of the training input image. The original light source information of the training input image may include the color deviation of the original light source applied to the real training input image. Based on the pixel values of the candidate gray pixels of the training input image, the pixel values (hereinafter also referred to as "corrected pixel values of the candidate gray pixels") from which the color deviation of the original light source of the training input image is eliminated can be obtained.

[0115] For example, the GT score can be determined based on the angular errors between the corrected pixel values of the candidate gray pixels of the training input image and the pixel values indicating achromatic color. The angular error can represent the ratio difference of the R value, G value, and B value between the corrected pixel value of the candidate gray pixel and the pixel value indicating achromatic color. For example, when the training input image includes multiple candidate gray pixels, the GT score can be determined as the sum of the angular errors of the candidate gray pixels.

[0116] For example, a pixel value indicating achromatic color may be a pixel value in which the ratio among the R value, the G value, and the B value is 1:1:1. For example, the R value, the G value, and the B value may all include the pixel value 1. For example, when the ratio among the R value, the G value, and the B value of the corrected pixel value of a candidate gray-scale pixel is closer to 1:1:1, the candidate gray-scale pixel is more likely to correspond to an object having a chroma close to achromatic color.

[0117] For example, the angular error may be determined based on Equation 6:

[0118] Equation 6

[0119]

[0120] In Equation 6, AE i represents the angular error of candidate gray-scale pixel i, Γ i represents a vector corresponding to the corrected pixel value of candidate gray-scale pixel i (for example, a vector having a dimension of 1×3), Γ a represents a vector corresponding to the pixel value indicating achromatic color (for example, [1, 1, 1]), and Γ i ·Γ a represents the inner product between vectors Γ i and Γ a .

[0121] For example, when the training input image includes a plurality of candidate gray-scale pixels, the GT score may be determined based on Equation 7:

[0122] Equation 7

[0123]

[0124] In Equation 7, score gray represents the GT score of the training input image, and other symbols may have meanings substantially the same as those expressed in Equation 6.

[0125] According to one or more example embodiments, the electronic device may determine whether to apply the light source information extraction model and the light source information generation model to the input image 410 based on at least one of the color diversity of the input image 410 and the gray-scale pixel related score 430. According to one or more example embodiments, the electronic device may determine whether to apply the light source information extraction model and the light source information generation model to the input image 410 based on whether a first condition of the color diversity of the input image 410 (for example, the color diversity score is less than or equal to a first threshold score) and / or a second condition of the gray-scale pixel related score 430 (for example, the gray-scale pixel related score 430 exceeds a second threshold score) is satisfied.

[0126] For example, when both the first condition and the second condition are satisfied, the electronic device may apply a light source information extraction model and a light source information generation model to the input image 410. That is, when the color diversity score of the input image 410 is less than or equal to a first threshold score and the grayscale pixel correlation score 430 of the input image 410 exceeds a second threshold score, the electronic device may determine to apply the light source information extraction model and the light source information generation model to the input image 410.

[0127] For another example, when at least one of the first condition and the second condition is satisfied, the electronic device may apply a light source information extraction model and a light source information generation model to the input image 410. That is, when the color diversity score of the input image 410 is less than or equal to a first threshold score or the grayscale pixel correlation score 430 of the input image 410 exceeds a second threshold score, the electronic device may determine to apply the light source information extraction model and the light source information generation model to the input image 410.

[0128] Figure 5 An example of using pseudo grayscale pixels to correct an image according to one or more example embodiments is shown.

[0129] According to one or more example embodiments, the electronic device may generate an output image from the input image based on pseudo grayscale pixels included in the input image.

[0130] In operation 510, the electronic device may determine candidate grayscale pixels among the pixels of the input image based on the pixel values and the grayscale trajectory mapped to the achromatic object.

[0131] As described above with reference to Figure 4 The grayscale trajectory, which is the range of pixel values of the pixels corresponding to the achromatic object, may include: a line corresponding to the achromatic object and / or an inner region of the line in a coordinate space including two coordinates (e.g., R / G coordinate and B / G coordinate) corresponding to the pixel values normalized to the G value.

[0132] The electronic device may determine a pixel whose corresponding point (normalized to the G value) is a point on the grayscale trajectory or a pixel within the grayscale trajectory as a candidate grayscale pixel.

[0133] In operation 520, the electronic device may detect pseudo-gray pixels corresponding to a color object among the determined candidate gray pixels. The electronic device may use a gray pixel detection model to detect the pseudo-gray pixels. The gray pixel detection model may be generated and / or trained to output a ground truth gray map from an input image. According to one or more example embodiments, the gray pixel detection model may be implemented based on a machine learning model and may include, for example, a neural network (e.g., a CNN). According to one or more example embodiments, the gray pixel detection model may be trained by supervised learning based on a training input image and a GT ground truth gray map mapped to the training input image.

[0134] The ground truth gray map may indicate whether the pixels in the input image are real gray pixels. The pixels in the ground truth gray map may individually correspond to the pixels in the input image.

[0135] For example, each pixel of the ground truth gray map may have a pixel value indicating whether the pixel of the input image is a real gray pixel. The electronic device may detect the following pixels of the input image as pseudo-gray pixels: having a pixel value among the pixels of the ground truth gray map indicating that the pixel is not a real gray pixel and corresponding to a candidate gray pixel.

[0136] For example, when the pixel of the ground truth gray map is a candidate gray pixel, the pixel may have a pixel value based on the angular error between the corrected pixel value of the candidate gray pixel and the pixel value indicating achromatic color. The electronic device may detect the following pixels of the input image as pseudo-gray pixels: having a pixel value exceeding a threshold angular error and corresponding to a candidate gray pixel among the pixels of the ground truth gray map.

[0137] In operation 530, the electronic device may perform, based on the detected pseudo-gray pixels: generating original light source information, generating intermediate light source information, determining target light source information, or generating an output image.

[0138] For example, the electronic device may also input the ground truth gray map together with the input image into a light source information extraction model (e.g., Figure 2 the light source information extraction model 220) to generate original light source information.

[0139] For example, the electronic device may also input the ground truth gray map together with an embedding vector of the input image (e.g., Figure 2 the embedding vector 230) and / or original light source information (e.g., Figure 2 the original light source information 240) into a light source information generation model (e.g., Figure 2 the light source information generation model 250) to generate intermediate light source information.

[0140] For example, the electronic device may combine the original light source information and the intermediate light source information based on the pixels indicated as pseudo-gray pixels by the ground truth gray map.

[0141] According to one or more example embodiments, an electronic device may generate an output image from an input image by performing a statistics-based AWB method based on detected pseudo-gray pixels. For example, the electronic device may perform a statistics-based AWB method by applying weights to the detected pseudo-gray pixels.

[0142] Figure 6 An example of using multiple WB methods to correct an image according to one or more example embodiments is shown.

[0143] According to one or more example embodiments, an electronic device may calculate target light source information 640 based on multiple WB methods.

[0144] The electronic device may determine first partial target light source information by applying a first WB method 620 based on the input image 610. For example, the first WB method 620 may include operations of determining the target light source information 640 based on the light source information extraction model and the light source information generation model described above with reference to Figure 1 and Figure 2 An operation of determining the target light source information 640.

[0145] The electronic device may determine second partial target light source information by applying a second WB method 630 based on the input image 610. For example, the second WB method 630 may be / include any of the above-described statistics-based WB methods.

[0146] The electronic device may determine the target light source information 640 by combining the first partial target light source information and the second partial target light source information using a ratio or weights corresponding to the first partial target light source information and weights corresponding to the second partial target light source information. According to one or more example embodiments, the sum of the first weight for the first partial target light source information and the second weight for the second partial target light source information may be a constant. The first weight and the second weight may be greater than or equal to zero (0) and less than or equal to 1.

[0147] According to one or more example embodiments, the first weight and / or the second weight may be determined based on the predicted performance from the first WB method 620 and / or the predicted performance from the second WB method 630. For example, the electronic device may determine the first weight based on the color diversity score of the input image 610 or the gray pixel correlation score of the input image 610. When the color diversity score of the input image 610 increases, the electronic device may determine the first weight as a smaller value and the second weight as a larger value. When the gray pixel correlation score of the input image 610 increases, the electronic device may determine the first weight as a larger value and the second weight as a smaller value.

[0148] Figure 7Shows an example configuration of an electronic device according to one or more example embodiments.

[0149] According to one or more example embodiments, the electronic device 700 may include an input image acquirer 710, a processor 720, a memory 730, a communicator 740, and an output image outputter 750. In practice, the processor 720 may be a single processor or a combination of processors (the types of processors are described below). In some embodiments, the electronic device 700 may be a camera including one or more image sensors that capture an image for which WB will be performed.

[0150] The input image acquirer 710 (e.g., a sensor) may acquire an input image. The input image acquirer 710 may send the acquired input image to the processor 720.

[0151] The processor 720 may extract original light source information from the input image based on a light source information extraction model. The processor 720 may generate intermediate light source information from the input image and the original light source information based on a light source information generation model. The processor 720 may determine target light source information based on the original light source information and the intermediate light source information. The processor 720 may generate an output image based on the result of applying the target light source information to the input image.

[0152] The memory 730 may temporarily and / or permanently store at least one of a light source information extraction model, an input image, original light source information, a light source information generation model, intermediate light source information, target light source information, and an output image. The memory 730 may store instructions for extracting original light source information, generating intermediate light source information, determining target light source information, and / or generating an output image. However, this is provided only as an example, and the information to be stored in the memory 730 is not limited thereto.

[0153] The communicator 740 may send and receive at least one of a light source information extraction model, an input image, original light source information, a light source information generation model, intermediate light source information, target light source information, and an output image. The communicator 740 may establish a wired communication channel and / or a wireless communication channel with an external device (e.g., another electronic device 700 and a server). For example, it may establish communication through cellular communication, short-range wireless communication, local area network (LAN) communication, Bluetooth, Wi-Fi Direct, or Infrared Data Association (IrDA), a traditional cellular network, a fourth generation (4G) and / or fifth generation (5G) network, next-generation communication, the Internet, or a telecommunication network (such as a computer network (e.g., LAN or wide area network (WAN))).

[0154] The output image outputter 750 can receive the output image generated by the processor 720 and output the received output image. For example, the output image outputter 750 can be implemented as a display and display the output image (the display may or may not be a component of the electronic device 700). For example, the output image outputter 750 can be implemented as the communicator 740 and send the output image to an external device.

[0155] Herein, regarding Figures 1 to 7The described computing devices, electronic devices, processors, memories, image sensors, displays, information output systems and hardware, storage devices, and other devices, apparatuses, units, modules, and components are implemented by or represent hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (such as logic gate arrays, controllers, and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve a desired result). In one example, a processor or computer includes or is connected to one or more memories that store instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For simplicity, the singular terms "processor" or "computer" may be used in the description of the examples described in this application, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, can implement a single hardware component or two or more hardware components. The hardware components can have any one or more of different processing configurations, examples of different processing configurations including single processors, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0156] Figures 1 to 7The method of performing the operations described in this application, as shown, is executed by computing hardware (e.g., by one or more processors or computers), which is implemented to execute instructions or software as described above to perform the operations performed by the method described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors or a processor and a controller may execute a single operation, or two or more operations.

[0157] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and execute the method as described above may be written as a computer program, code segment, instruction, or any combination thereof, to individually or jointly direct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and method as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executable by one or more processors or computers. In another example, the instructions or software include high-level code executable by one or more processors or computers using an interpreter. The instructions or software may be written in any programming language based on the block diagrams and flowcharts shown in the figures and the corresponding descriptions herein, which disclose algorithms for performing the operations performed by the hardware components and method as described above.

[0158] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and execute the methods as described above, as well as any associated data, data files, and data structures, can be recorded, stored, or fixed in one or more non-transitory computer-readable storage media, or recorded, stored, or fixed on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDD), solid state drives (SSD), flash memory, card-type memories (such as, multimedia card micro or cards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid state disks, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers such that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed by one or more processors or computers in a distributed manner.

[0159] Although the present disclosure includes specific examples, it will be apparent after understanding the disclosure of this application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are considered to be illustrative only and not for purposes of limitation. The description of a feature or aspect in each example should be considered applicable to similar features or aspects in other examples. Suitable results can be achieved if the described techniques are performed in a different order, and / or if the components in the described system, architecture, device, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

[0160] Accordingly, in addition to the above disclosure, the scope of the disclosure may also be defined by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as being included in the disclosure.

Claims

1. A method performed by an electronic device, comprising: Extracting original light source information representing the original light source color cast of the input image from the input image, the extracting step being performed by applying a light source information extraction model to the input image; generating intermediate light source information representing an intermediate light source color cast of the input image from the original light source information, the generating step being performed by applying a light source information generation model to the original light source information; determining target light source information based on the extracted original light source information and based on the generated intermediate light source information; as well as An output image is generated by applying the determined target light source information to the input image.

2. The method according to claim 1, wherein: The step of determining the target light source information includes combining at least a portion of the original light source information and at least a portion of the intermediate light source information.

3. The method according to claim 1, wherein: The steps to generate the output image are: The original illuminant color cast is removed from the input image, and the color cast of the target illuminant information is added to the input image.

4. The method according to claim 1, wherein: The steps of determining the target light source information include: A user input is obtained, and a contribution of the original light source information or the intermediate light source information to the target light source information is controlled according to the user input.

5. The method according to claim 1, further comprising: Outputs the embedding vector produced by the light source information extraction model when extracting the original light source information from the input image, The intermediate light source information is generated by applying a light source information generation model to the embedding vector of the input image and the original light source information.

6. The method according to claim 1, further comprising: Whether to apply the light source information extraction model and the light source information generation model to the input image is determined based on the color diversity of the input image.

7. The method according to claim 6, wherein: The step of determining whether to apply the light source information extraction model and the light source information generation model to the input image based on the color diversity of the input image includes: performing linear regression on the pixel values ​​of the pixels of the input image; and A color diversity score is determined based on a difference between a result of the linear regression and the pixel value, the color diversity score indicating the color diversity.

8. The method according to claim 6, wherein: The step of determining whether to apply the light source information extraction model and the light source information generation model to the input image based on the color diversity of the input image includes: A color diversity score for the input image is determined by applying a color diversity determination model to the input image.

9. The method according to claim 1, further comprising: Whether to apply the light source information extraction model and the light source information generation model to the input image is determined based on the scores of grayscale pixels corresponding to achromatic objects in the input image.

10. The method according to claim 1, further comprising: determining candidate grayscale pixels among pixels of the input image based on the pixel values ​​and the grayscale trajectory mapped to the achromatic object; Detecting pseudo grayscale pixels corresponding to colored objects from the determined candidate grayscale pixels; as well as Based on the detected pseudo grayscale pixels, any one of the following steps is performed: generating original light source information, generating intermediate light source information, determining target light source information, and generating an output image.

11. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, configure the one or more processors to perform the method according to any one of claims 1 to 10.

12. An electronic device comprising: one or more processors; as well as A memory storing instructions configured to cause the one or more processors to: Extracting original light source information representing the original light source color cast of the input image from the input image, the extracting step being performed by applying a light source information extraction model to the input image; Generate intermediate light source information representing the intermediate light source color shift of the input image by applying the light source information generation model to the original light source information; determining target light source information based on the extracted original light source information and based on the generated intermediate light source information; as well as An output image is generated by applying the determined target light source information to the input image.

13. The electronic device according to claim 12, wherein: The instructions are further configured to cause the one or more processors to: The target light source information is determined by combining at least a portion of the original light source information and at least a portion of the intermediate light source information.

14. The electronic device according to claim 12, wherein: The instructions are further configured to cause the one or more processors to: The original illuminant color cast is removed from the input image, and the color cast of the target illuminant information is added to the input image.

15. The electronic device according to claim 12, wherein: The instructions are further configured to cause the one or more processors to: A user input is obtained, and a contribution of the original light source information or the intermediate light source information to the target light source information is controlled according to the user input.

16. The electronic device according to claim 12, wherein: The instructions are further configured to cause the one or more processors to: Output the embedding vector generated by the light source information extraction model when extracting the original light source information from the input image; as well as The intermediate light information is generated by applying the light information generation model to the embedding vector of the input image and the original light information.

17. The electronic device according to claim 12, wherein: The instructions are further configured to cause the one or more processors to: Whether to apply the light source information extraction model and the light source information generation model to the input image is determined based on the color diversity of the input image.

18. The electronic device according to claim 17, wherein: The instructions are further configured to cause the one or more processors to: performing linear regression on the pixel values ​​of the pixels of the input image; and A color diversity score is determined based on a difference between a result of the linear regression and the pixel value, the color diversity score indicating the color diversity.

19. The electronic device according to claim 12, wherein: The instructions are further configured to cause the one or more processors to: Whether to apply the light source information extraction model and the light source information generation model to the input image is determined based on the scores of grayscale pixels corresponding to achromatic objects in the input image.

20. The electronic device according to claim 12, wherein: The instructions are further configured to cause the one or more processors to: determining candidate grayscale pixels among pixels in the input image based on the pixel values ​​and grayscale trajectories mapped to achromatic objects; Detecting pseudo grayscale pixels corresponding to colored objects from the determined candidate grayscale pixels; as well as Based on the detected pseudo grayscale pixels, any one of the following steps is performed: generating original light source information, generating intermediate light source information, determining target light source information, and generating an output image.

Images