Electronic device and method for processing image data

Abstract

A method for processing image data includes: obtaining an original input image captured using a first image sensor under input illumination conditions; obtaining, using a transformation estimator, a color transformation that maps a characteristic of the first image sensor to a characteristic of a second image sensor based on the input lighting condition; and generating an original output image having characteristics of the second image sensor based on the original input image and the color transformation.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and method for converting the colors of an original input image captured using a first sensor under any arbitrary illumination to colors corresponding to a second sensor under arbitrary illumination so that the original output image appears as if it was captured using the second sensor. Background Art

[0002] Capturing raw sensor images in various settings is quite challenging, especially under various lighting and illumination conditions. The process requires adjusting camera settings, using a tripod, setting up the scene, and possibly discovering different lighting conditions and environments. Under such constraints, capturing large-scale datasets of raw images for training neural network models is laborious and time-consuming.

[0003] For example, updating a mobile device with a new image sensor requires recapturing a new training data set because the new image sensor may have different characteristics (e.g., spectral sensitivity, noise distribution, etc.) The time and effort required to capture training data is a significant challenge for manufacturers of mobile devices and other smartphone and camera companies.

[0004] Therefore, there is a need for a simplified process for creating a new dataset of raw images that can be used to train a neural network model to be used with a new image sensor. Summary of the invention

[0005] Technical Solutions

[0006] One or more embodiments of the present disclosure provide an electronic device and method for converting the colors of a raw sensor image captured by a first sensor into colors corresponding to a second sensor so that the raw sensor image captured by the first sensor appears as if it has been captured by the second sensor.

[0007] According to one aspect of the present disclosure, an electronic device for processing image data includes: at least one memory configured to store instructions; and at least one processor configured to execute the instructions to: obtain an original input image captured using a first image sensor under input lighting conditions; based on the input lighting conditions, obtain a color transform using a transform estimator that maps characteristics of the first image sensor to characteristics of a second image sensor; and generate an original output image having the characteristics of the second image sensor based on the original input image and the color transform.

[0008] According to one aspect of the present disclosure, a method for processing image data is performed by at least one processor, including: obtaining an original input image captured using a first image sensor under input lighting conditions; based on the input lighting conditions, obtaining a color transform that maps characteristics of the first image sensor to characteristics of a second image sensor using a transform estimator; and generating an original output image having the characteristics of the second image sensor based on the original input image and the color transform.

[0009] According to one aspect of the present disclosure, a computer-readable medium is configured to store instructions that, when executed by at least one processor of a device for processing image data, cause the at least one processor to: obtain an original input image captured using a first image sensor under input lighting conditions; based on the input lighting conditions, obtain a color transform using a transform estimator that maps characteristics of the first image sensor to characteristics of a second image sensor; and generate an original output image having the characteristics of the second image sensor based on the original input image and the color transform.

[0010] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other aspects, features and aspects of the embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a diagram of an electronic device for performing color conversion according to an embodiment of the present invention;

[0013] Figure 2 is a diagram showing an example of capturing an image under arbitrary lighting conditions using different image sensors according to an embodiment of the present disclosure;

[0014] Figure 3A , Figure 3B , Figure 4 and Figure 5 A method of indexing an image to provide an illumination dataset according to an embodiment of the present disclosure is shown;

[0015] Figure 6 Examples of arbitrary input lighting not present in the lighting dataset are shown;

[0016] Figure 7 is a flowchart of a process for training an AI model to convert images captured under arbitrary lighting conditions according to an embodiment of the present disclosure;

[0017] Fig. 8A and Figure 8BA method of collecting an illumination dataset using two sensors for estimating color transformation according to an embodiment of the present disclosure is shown;

[0018] Fig. 9 A method of calculating a transformation between two sensors based on an illumination data set according to an embodiment of the present disclosure is shown;

[0019] Fig. 10A A method of training a neural network-based transform estimator according to an embodiment of the present disclosure is shown;

[0020] Fig. 10B shows the structure of a neural network configured to estimate color transformation according to an embodiment of the present disclosure;

[0021] Fig.11 A method of converting an original input image captured using a first sensor into an original output image having characteristics of a second sensor according to an embodiment of the present disclosure is shown;

[0022] Fig.12 The application of the color conversion method according to the embodiment of the present disclosure is shown;

[0023] Fig.13 The application of the color conversion method according to the embodiment of the present disclosure is shown;

[0024] Fig.14 is a diagram of an apparatus for performing color conversion according to an embodiment of the present disclosure; and

[0025] Fig.15 According to the embodiment of the present disclosure Fig.14 A diagram of the components of one or more devices.

[0026] Fig.16 A method for processing image data according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] Example embodiments are described in more detail below with reference to the accompanying drawings.

[0028] In the following description, the same reference numerals are used for the same elements even in different drawings. Matters defined in the specification, such as detailed construction and elements, are provided to assist in a comprehensive understanding of the exemplary embodiments. However, it is apparent that the exemplary embodiments can be practiced without those specifically defined matters. In addition, well-known functions or structures are not described in detail because they would obscure the description with unnecessary detail.

[0029] Expressions such as "at least one of..." when following a list of elements modify the entire list of elements rather than modifying the individual elements in the list. For example, the expression "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or any variation of the foregoing examples.

[0030] Although terms such as "first", "second", etc. may be used to describe various elements, these elements must not be limited to the above terms. The above terms may be used only to distinguish one element from another.

[0031] The term "module" or "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0032] It is apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0033] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible implementations includes the combination of each dependent claim with every other claim in the claim set.

[0034] Unless explicitly described as such, elements, actions, or instructions used herein should not be interpreted as critical or essential. In addition, as used herein, the articles "a" and "an" are intended to include one or more items and can be used interchangeably with "one or more." In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and can be used interchangeably with "one or more." Figure 1 In the case of multiple items, the term "a" or similar language is used. In addition, as used herein, the terms "has," "have," "having," and the like are intended to be open-ended terms. In addition, unless expressly stated otherwise, the phrase "based on" is intended to mean "based, at least in part, on."

[0035] One or more embodiments of the present disclosure provide a method and apparatus for performing an image processing process, which includes estimating a color transform that maps an original input image captured using a first image sensor to a second image sensor, and applying the color transform to the original input image to obtain an original output image having characteristics of the second image sensor.

[0036] The raw input image may be an unprocessed raw image output from the first image sensor. The term "raw image" may refer to the unprocessed digital output of the image sensor of the camera, and may also be referred to as a "raw burst image" or a "Bayer image". Light or photons incident from a scene are digitized and recorded by the camera sensor, and the raw image consists of digital pixel intensity values ​​recorded by the camera sensor before any processing is applied. For example, a raw image is an image that has not been processed by an image signal processor (ISP) or an image processing unit (IPU), and may have a raw Bayer format. When a camera sensor includes sensor elements arranged in a pattern of red, green, and blue channels, which is referred to as a Bayer array, the image recorded on the camera sensor by the Bayer array is referred to as a Bayer image.

[0037] In embodiments, the term "raw image" may also refer to a partially processed image, such as an otherwise unprocessed image to which demosaicing or denoising has been applied. Additionally, the term "raw image" may refer to a fused raw image obtained from a burst of raw image frames, such as in a format such as ProRAW, ExpertRAW, etc. In embodiments, the term "raw image" may refer to an image to which processing such as white balance correction and subsequent color manipulation has not yet been applied.

[0038] Figure 1 is a diagram of a system for performing color transformation according to an embodiment of the present disclosure.

[0039] like Figure 1 As shown, the system 100 may include a first camera 100A, a second camera 100B, at least one processor 200 , at least one memory 300 , an input interface 400 , a display 500 , and a communication interface 600 .

[0040] The first camera 100A may include a first lens 110A and a first image sensor 120A, the first image sensor 120A may include a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor, and a color filter array (CFA). The first camera 100A may capture an image based on a user input received via the input interface 400, and may output an unprocessed raw image to the processor 200. The input interface 400 may be implemented as at least one of a touch panel, a keyboard, a mouse, a button, a microphone, and the like.

[0041] Similarly, the second camera 100B may include a second lens 110B and a second image sensor 120B, which may include a CMOS sensor or a CCD sensor and a CFA. The second camera 100B may capture an image based on a user input received via the input interface 400 and may output an unprocessed raw image to the processor 200.

[0042] Although the first camera 100A and the second camera 100B are Figure 1 100A and 100B are shown as being directly connected to the processor 200, but the embodiment is not limited thereto. For example, each of the first camera 100A and the second camera 100B may be connected to a separate processor 200. As an example, one or both of the first camera 100A and the second camera 100B may be indirectly connected to one or more processors 200, and an image captured using one or both of the first camera 100A and the second camera 100B may be transmitted to the one or more processors 200 in a wired or wireless manner. For example, one or both of the first camera 100A and the second camera 100B may be included in an external electronic device outside the system 100, and the processor 200 may receive an original image captured using one or both of the first image sensor 120A and the second image sensor 120B through the communication interface 600.

[0043] The processor 200 may include an artificial intelligence (AI)-based image processing model including an automatic white balance module 210, a color transformation estimation module 220, and a color transformation module 230. Although the automatic white balance module 210 is shown as being disposed outside the first camera 100A and the second camera 100B, according to an embodiment, the automatic white balance module 210 may be disposed inside one or both of the first camera 100A and the second camera 100B. The processor 200 may receive a raw input image from at least one of the first camera 100A and the second camera 100B or from an external device via the communication interface 600. The raw input image may be a raw image that is not processed by an image signal processor (ISP) and / or has a Bayer format.

[0044] Based on user input for editing the original input image, for example, user input requesting a color transform to convert the original input image into an original output image with different characteristics, the processor 200 may perform a color transform on the original input image and output the color transformed original input image as the original output image. In an embodiment, the characteristics of the image sensor may include spectral sensitivity or noise profile. The processor 200 may store the original output image in the memory 300 as an enhanced image or a synthesized image, for example as part of an image data set.

[0045] In an embodiment, the processor 200 may further include an image signal processor (ISP) that may perform image processing on at least one of the original input image and the original output image to generate a processed image that may be displayed on the display 500. In an embodiment, the processed image may also be referred to as a display-referred image or an sRGB image.

[0046] In an embodiment of the present disclosure, the processor 200 may estimate a color transform that maps a characteristic of the first image sensor 120A to a characteristic of the second image sensor 120B, and may apply the color transform to an original input image captured using the first image sensor 120A to obtain an original output image having the characteristic of the second image sensor 120B. Similarly, the processor 200 may estimate a color transform that maps a characteristic of the second image sensor 120B to a characteristic of the first image sensor 120A, and may apply the color transform to an original input image captured using the second image sensor 120B to obtain an original output image having the characteristic of the first image sensor 120A. In an embodiment, the characteristic of the first image sensor 120A may be or include an RGB response of the first image sensor 120A. In an embodiment, the RGB response of the first image sensor 120A may be affected by or correspond to at least one of the spectral sensitivity of the first image sensor 120A and the noise distribution of the first image sensor 120A. In an embodiment, the characteristic of second image sensor 120B may be or include an RGB response of second image sensor 120B. In an embodiment, the RGB response of second image sensor 120B may be affected by or correspond to at least one of a spectral sensitivity of second image sensor 120B and a noise distribution of second image sensor 120B.

[0047] In an embodiment, the processor 200 may estimate a color transform based on receiving information about one or both of the first image sensor 120A and the second image sensor 120B through the input interface 400. For example, based on a user input requesting a color transform to convert an original input image captured using the first image sensor 120A into an original output image having characteristics of the second image sensor 120B, the processor 200 may identify the second image sensor 120B based on the user input, and may obtain a color transform based on the identified second image sensor 120B.

[0048] In an embodiment, the auto white balance module 210 may identify the input illumination under which the original input image was captured, and the color transform estimation module 220 may estimate a color transform corresponding to the input illumination.

[0049] In an embodiment of the present disclosure, the color transform T may be estimated using an artificial intelligence model (e.g., a neural network model or a machine learning model) trained to estimate color transforms. Specifically, a machine learning-based method may be used to predict a color transform that maps characteristics of the first image sensor 120A to characteristics of the second image sensor 120B and / or that maps characteristics of the second image sensor 120B to characteristics of the first image sensor 120A. In an embodiment, the artificial intelligence model may be referred to as a transform estimator and may be included in or implemented by the color transform estimation module 220. When estimating the color transform, a small data set of images of color calibration patterns captured under various lighting conditions may be used without the need for a large training data set. In one embodiment, the various lighting conditions may include daylight, fluorescent light, or incandescent light. Reference is made below to Figure 7 , Fig. 10A and Fig. 10B The method of estimating the color transform T using the neural network is described in more detail. In an embodiment, the color transform T may be estimated by at least one of the transform estimator and the color transform estimation module 220 using any other desired method, such as a linear mapping method, a nonlinear mapping method, and a k-nearest neighbor method. The color-transformed original input image may be stored in the memory 300 as an original output image.

[0050] All elements of the electronic device may be included in a single device, or may be included in more than one device. For example, at least one of the first camera 100A, the second camera 100B, the input interface 400, and the display 500 may be included in a client device (e.g., a smart phone), and the AI-based image processing model of the processor 200 may be included in a server. When the AI-based image processing model is included in the server, the client device may send an original input image and input illumination to the server, request the server to perform color transformation on the original input image to obtain an original output image, and may receive the original output image from the server.

[0051] Figure 2 is a diagram showing examples of images captured under arbitrary lighting conditions using different image sensors according to an embodiment of the present disclosure. Figure 2As shown, scene 201 may be illuminated by particular illumination 202. A raw image 203A of scene 201 may be captured using first image sensor 120A, which may have corresponding spectral sensitivity 204A, and a raw image 203B of scene 201 may be captured using second image sensor 120B, which may have corresponding spectral sensitivity 204B. Raw images 203A and 203B may be provided to ISP 205, which may be included in, for example, processor 200. Raw image 203A may be processed by ISP 205 to obtain processed image 206A, and raw image 203B may be processed by ISP 205 to obtain processed image 206B.

[0052] In an embodiment, the image values ​​of the raw images 203A and 203B may depend on the corresponding spectral sensitivities 204A and 204B, and the spectral distribution of the lighting condition 202. In an embodiment, due to the difference between the spectral sensitivities 204A and 204B, the image values ​​of the raw image 203A may be different from the image values ​​of the raw image 203B, even though they are both images of the same scene 201 captured under the same lighting condition 202. In an embodiment, the ISP 205 may apply image processing, such as white balance correction and other color manipulations, to the raw images 203A and 203B to generate processed images 206A and 206B. After the ISP 205 applies the image processing, the color values ​​of the processed images 206A and 206B may be relatively similar, e.g., more similar than the color values ​​of the raw images 203A and 203B. Figure 2 As shown, in order to process the original images 203A and 203B to generate processed images 206A and 206B having relatively similar color values, the ISP 205 may use different ISP parameters, such as first ISP parameters 205A corresponding to the first image sensor 120A and second ISP parameters 205B corresponding to the second image sensor 120B.

[0053] Because different imaging sensors have different spectral sensitivities and therefore produce raw images with different color properties even if the images are captured under the same lighting conditions and are of the exact same scene, many AI models used to process raw images may be sensor specific. In an embodiment, the color characteristics of an imaging sensor may refer to the color response of the imaging sensor, such as the RGB response of the imaging sensor, but the embodiment is not limited thereto. For example, an AI model trained based on an image data set including raw images captured using the first image sensor 120A may provide incorrect or undesirable results when applied to raw images captured using the second image sensor 120B. Therefore, when a new sensor is developed and manufactured, the training image data set for the AI-based algorithm may need to be recaptured using a new sensor in order to train a new AI model suitable for the new sensor. Capturing a new training image data set may require a lot of time and resources, and this problem may be complicated by the fact that many devices (such as smart phones) may include multiple image sensors per device.

[0054] Therefore, embodiments of the present disclosure may relate to methods and apparatus for transforming an input raw image having characteristics corresponding to a first image sensor into an output raw image having characteristics of a second image sensor. Therefore, embodiments of the present disclosure may allow the need to capture new training image data sets to be significantly reduced. For example, when a new image sensor is developed and manufactured, an embodiment may allow a previous training image data set captured using a previous image sensor to be converted into a new training image data set having the characteristics of the new image sensor, so that an AI model can be trained for use with the new image sensor without the need to recapture training data using the new image sensor. In addition, an embodiment may allow a previous AI model to be used with images captured using a new image sensor by converting images captured using the new image sensor into images suitable for the previous AI model.

[0055] Some methods of image conversion can be performed using processed images instead of raw images. These methods may not take into account lighting because the ISP used to generate the processed images can remove the lighting. In addition, because the color changes between processed images from different sensors may be relatively small compared to the color changes between raw images from different sensors, the training data set required for the processed images may be relatively small. However, these processed image-based methods may provide less accurate results than raw image-based methods because ISP processing may involve nonlinear manipulations that may be irreversible. In addition, training data sets generated based on processed images may not be suitable for training certain AI models because some information (such as lighting information) may be lost or changed during ISP processing.

[0056] Figure 3A , Figure 3B , Figure 4 and Figure 5 A method of indexing images to provide an illumination dataset according to an embodiment of the present disclosure is shown. Raw images captured under multiple different illumination conditions can be stored and indexed to provide an illumination dataset. Raw images captured under multiple different illumination conditions can be referred to as color calibrated images. Fig. 8A Examples of light boxes that may be used to capture color calibration images are described in more detail.

[0057] Figure 3A and Figure 3B Examples of images of the color calibration pattern 20 captured using the same image sensor under different lighting conditions are illustrated. Specifically, Figure 3A shows a color calibration image 310 captured using an image sensor under a first lighting condition, and Figure 3B A color calibration image 320 captured using the same image sensor under a second lighting condition is shown. The color calibration pattern 20 may include a neutral patch 301, which may also be referred to as a colorless patch or a gray patch, and may be used to determine RGB color values ​​that indicate the lighting under which the color calibration images 310 and 320 were captured. For example, the lighting in the color calibration image 310 may correspond to the RGB value And the illumination in the color calibration image 320 may correspond to the RGB value .

[0058] like Figure 4 As shown, each of the color calibration images in the illumination dataset can be indexed based on the scene illumination using the two-dimensional coordinates [R / G, B / G] and stored as the illumination dataset. For example, the first illumination condition and the second illumination condition can be indexed using the two-dimensional coordinates L 1 = [R- 1 / G 1 , B 1 / G 1 ] and L 2 = [R- 2 / G 2 , B 2 / G 2 ] to index and be stored as an illumination data set. In an embodiment, a set of two-dimensional coordinates [R / G, B / G] may be referred to as a chromaticity space.

[0059] In an embodiment, the corresponding illumination value L recorded by the first image sensor may be used. 1 To L MThe first image sensor 120A captures a first illumination dataset under a set of illumination conditions, and the second image sensor 120B may be used to capture a second illumination dataset under the same illumination conditions. j , the transformation that maps the first color-calibrated image to the second color-calibrated image can be determined In an embodiment, the transformation may be calculated using, for example, the least squares method .

[0060] In the embodiment, corresponding to the illumination L j Transformation A plurality of parameters t may be included, which may be arranged in matrix form according to Equation 1 below:

[0061] “Equation 1”

[0062]

[0063] Despite the transformation is shown as a 3×3 matrix, but the embodiment is not limited thereto, and the transformation can be represented as a matrix of any size, or in another form such as a polynomial. In an embodiment, the illumination L j may correspond to a color calibration image included in the first illumination data set and a color calibration image included in the second illumination dataset ,in and It can be expressed according to the following equations 2 and 3:

[0064] “Equation 2”

[0065]

[0066] “Equation 3”

[0067]

[0068] In an embodiment, the color calibration image may be estimated using the least squares method by solving the following equation 4: and Transformation between :

[0069] “Equation 4”

[0070]

[0071] As used in Equation 2, Equation 3, and Equation 4, P represents the number of color samples included in the color calibration image.

[0072] like Figure 5 As shown, the first illumination data set may include a first illumination L recorded with a first image sensor. 1 The color calibration image 310A captured under the corresponding first lighting condition, and the second lighting data set may include the color calibration image 310B captured under the first lighting condition, and the transformation may be calculated using the color calibration images 310A and 310B. The transformation The first illumination data set may be used to convert an image captured using the first image sensor 120A under a first illumination condition into an image having the characteristics of the second image sensor 120B under the first illumination condition. Similarly, the first illumination data set may include a first illumination data set corresponding to the second illumination L recorded by the first image sensor 120A. 2 The second illumination data set may include a color calibration image 320B captured under the second illumination condition, and the transformation may be calculated using the color calibration images 320A and 320B. The transformation It can be used to convert an image captured using the first image sensor 120A under a second lighting condition into an image having the characteristics of the second image sensor 120B under the second lighting condition.

[0073] Therefore, the transformation dataset can be constructed , which can be used to transform the lighting L 1 To L M Any captured image.

[0074] Figure 6 An example of an arbitrary input lighting that is not present in the lighting dataset is shown. u May not be included in the scheduled lighting L 1 To L M So the transformation may not be known. If you transform the dataset The transformations included in One is used to convert the corresponding lighting L u If the input raw image is distorted, the resulting output raw image may have inaccurate or incorrect colors.

[0075] Thus, embodiments of the present disclosure may use a learning-based approach to predict transformations that map between different image sensors under arbitrary lighting conditions. For example, embodiments of the present disclosure may involve training a neural network, such as a multilayer perceptron, using a relatively small training data set, and using the neural network to predict transformations corresponding to arbitrary lighting (including lighting not included in the data set). In an embodiment, the training data set may correspond to the transformation data set discussed above. , and may be generated based on, for example, 50 to 100 pairs of color calibration images captured using the first image sensor 120A and the second image sensor 120B.

[0076] Figure 7 A process for predicting transformation according to an embodiment of the present disclosure is shown. In an embodiment, one or more operations of process 700 may be performed by any of the elements discussed above (e.g., at least one of system 100, processor 200, color transformation estimation module 220, and color transformation module 230). In an embodiment, process 700 may be referred to as a color transformation method.

[0077] In operation S701, process 700 may include collecting a first illumination data set and a second illumination data set, the first illumination data set including a color calibration image captured using a first image sensor (e.g., first image sensor 120A), and the second illumination data set including a color calibration image captured using a second image sensor (e.g., second image sensor 120B). Fig. 8A and Figure 8B An example of a process for capturing an illumination dataset is described in more detail.

[0078] In operation S702, process 700 may include extracting illumination and calculating a transformation corresponding to a first illumination data set and a second illumination data set. Fig. 9 An example describing the process of extracting lighting and computing transforms in more detail.

[0079] In operation S703, process 700 may include training a transform estimator using the illumination and transform obtained in operation S702. Fig. 10A and Fig. 10B An example of a process for training a transform estimator is described in more detail.

[0080] In operation S704, process 700 may include converting at least one raw input image captured using the first image sensor into a raw output image having characteristics of the second image sensor using the transform predicted by the transform estimator. Fig.11 An example of the process of converting an original input image is described in more detail.

[0081] Although the above process 700 involves converting an original input image captured using the first image sensor 120A into an original output image having characteristics of the second image sensor 120B, the embodiment is not limited thereto. For example, in an embodiment, a transformation estimator may be trained to predict a transformation for converting from the second image sensor 120B to the first image sensor 120A, and operation S704 may include converting at least one original input image captured using the second image sensor 120B into an original output image having characteristics of the first image sensor 120A.

[0082] Fig. 8A and Figure 8B A process of collecting an illumination dataset for estimating color transform according to an embodiment of the present disclosure is shown.

[0083] like Fig. 8A As shown, the light box 10 can be used to illuminate the color calibration pattern 20 under different lighting conditions to obtain the illumination L 1 To L M In an embodiment, the light box 10 may be variable or tunable to emit light within a range of wavelengths or frequencies, which may simplify the process of collecting illumination data sets. For example, the light box 10 may be variable or tunable to emit light that is consistent with the illumination L 1 To L M Each corresponding light in , the lighting may include, for example, sunlight (or daylight), light emitting diode (LED) light, incandescent lamp, fluorescent lamp, etc. Fig. 8A In the embodiment, the color calibration pattern 20 is illustrated as a color chart, but the embodiments of the present disclosure are not limited thereto, and various examples of color materials may be used instead of the color chart. For example, in an embodiment, the color calibration pattern 20 may include various samples of materials with different textures and colors, or a collection of various items may be used as the color calibration pattern 20.

[0084] exist Figure 7 In operation S701, the first image sensor 120A and the second image sensor 120B may be used to capture images of the color calibration pattern 20 under different lighting conditions to obtain a first lighting data set and a second lighting data set. For example, a first pair of color calibration images under sunlight, a second pair of color calibration images under LED light, a third pair of color calibration images under incandescent light, and a fourth pair of color calibration images under fluorescent light may be captured and collected as training data for the transformation estimator. Examples of the first lighting data set and the second lighting data set are shown in FIG. Figure 8B Shown in.

[0085] Fig. 9FIG. 5 shows a method for calculating a transformation based on an illumination dataset according to an embodiment of the present disclosure. In operation S702, an illumination L may be extracted from a color calibration image included in a first illumination dataset. 1 To L M In an embodiment, the illumination L may be determined using the automatic white balance module 210 or based on the neutral block 301 included in the color calibration pattern 20. 1 To L M Then, based on the extracted illumination L 1 To L M and color calibration images to compute the transformation arrive For example, the transformation can be calculated using the least squares method as in Equation 4 arrive . Lighting 1 To L M and transformation arrive may be included in a training data set used to train a transform estimator. In an embodiment, the training data set may correspond to the transform data set discussed above. .

[0086] Fig. 10A A method for training a neural network-based transform estimator according to an embodiment of the present disclosure is shown.

[0087] In an embodiment, Fig. 10A The method shown can correspond to Figure 7 The operation S703 of the present invention can be used to train a transformation estimator, the transformation estimator comprising at least one neural network, the at least one neural network being trained to predict the transformation estimator for performing Figure 7 The transformation of operation S704.

[0088] The transformation estimator can be implemented as a neural network such as a multilayer perceptron (MLP) and can have Fig. 10B The network structure shown. The transformation estimator can receive the input illumination L j As input, it can output the estimated transformation The difference between the estimated transformation and the ground-truth transformation can be calculated as the loss of the transformation estimator, and the loss can be back-propagated to the neural network to update the node weights of the neural network. The transformation estimator can be trained until the loss reaches a predetermined minimum value, or converges to a constant value with a preset margin. In an embodiment, the training data set discussed above (which can be, for example, a transformation data set) can be obtained. ), obtain the base true transformation.

[0089] Fig. 10BThe structure of a neural network configured to estimate a transformation according to an embodiment of the present disclosure is shown.

[0090] like Fig. 10B As shown, the neural network may include an input layer, a set of hidden layers, and an output layer. For example, the input layer may include two nodes to receive an input illumination L represented as a two-dimensional coordinate [R / G, B / G] u For example, each of the hidden layer and the output layer may include nine nodes, but the number of nodes is not limited thereto. The output layer may output the output signal to be used in the illumination L u The image captured by the first image sensor 120A is converted into an image having the characteristics of the second image sensor 120B under the same lighting conditions. In particular, the output layer can output a color transform The matrix elements (t1, t2, t3, t4, t5, t6, t7, t8, t9) of , which can be converted into a matrix having the form shown in Equation 1.

[0091] As described above, the transformation can be obtained using an artificial intelligence (AI) based model such as a neural network, but the embodiments of the present disclosure are not limited thereto. For example, the transformation can be calculated based on a machine learning method (such as a k-nearest neighbor method) or based on a mathematical algorithm using the transformation data set discussed above.

[0092] Fig.11 A method for converting an original input image captured by a first sensor into an original output image having characteristics of a second sensor according to an embodiment of the present invention is described. In an embodiment, process 110 may correspond to Figure 7 Operation S704 of process 110, and one or more operations of process 110 may be performed by any of the elements discussed above (eg, system 100, processor 200, color transformation estimation module 220, and color transformation module 230). In an embodiment, process 110 may be referred to as a color transformation method.

[0093] In operation S111, process 110 may include obtaining an original input image. In an embodiment, the input illumination L u The original input image is captured using the first image sensor 120A. In an embodiment, the original input image may have characteristics of the first image sensor 120A. For example, the original input image may have a characteristic of the input illumination L uand a color value corresponding to a color response of first image sensor 120A. In an embodiment, the color response of first image sensor 120A may correspond to or be affected by at least one of a spectral sensitivity of first image sensor 120A and a noise distribution of first image sensor 120A.

[0094] In operation S112, process 110 may include obtaining an estimated transformation In an embodiment, the estimated transformation can be estimated or predicted by the color transformation estimation module 220, which can, for example, transform the input illumination L u is provided to the trained transform estimator model. In an embodiment, the estimated transform Can be based on the input lighting L u (which may be referred to as input lighting conditions) A color transform that maps characteristics of the first image sensor to characteristics of the second image sensor.

[0095] In operation S113, process 110 may include transforming the estimated Applied to the original input image. In an embodiment, the color transform module 230 may be used to apply the estimated transform .

[0096] In operation S114, process 110 may include converting the estimated The result of applying the original input image to obtain the original output image. In an embodiment, the original output image may have the characteristics of the second image sensor 120B. For example, the original output image may have the same characteristics as the input illumination L u and a color value corresponding to a color response of the second image sensor 120B. In an embodiment, the color response of the second image sensor 120B may correspond to or be affected by at least one of a spectral sensitivity of the second image sensor 120B and a noise distribution of the second image sensor 120B.

[0097] Fig.12 The use application of the color transformation method according to an embodiment of the present disclosure is shown. In an embodiment, one or more operations of the process 1200 may be performed by any of the elements discussed above (eg, the system 100, the processor 200, the color transformation estimation module 220, and the color transformation module 230).

[0098] In operation S1201, process 1200 may include capturing a raw image dataset using first image sensor 120A. In an embodiment, each of the raw images included in the raw image dataset may have characteristics of first image sensor 120A. For example, the raw image dataset may be a dataset corresponding to first image sensor 120A, and each of the raw images included in the raw image dataset may have characteristics corresponding to illumination L 1 To L M and a color value corresponding to the color response of the first image sensor 120A. In an embodiment, the color response of the first image sensor 120A may correspond to or be affected by at least one of the spectral sensitivity of the first image sensor 120A and the noise distribution of the first image sensor 120A.

[0099] In operation S1202, the process 1200 may include training a first AI model based on the original image data set. For example, the first AI model may be an AI model suitable for processing images captured using the first image sensor 120A.

[0100] In operation S1203, process 1200 may include performing sensor mapping on the raw image dataset. For example, the trained transform estimator may be used to transform the raw image dataset into a transformed raw image dataset. In an embodiment, each of the transformed raw images included in the transformed raw image dataset may have characteristics of the second image sensor 120B. For example, the transformed raw image dataset may be a dataset corresponding to the second image sensor 120B, and each of the transformed raw images included in the transformed raw image dataset may have a characteristic corresponding to the lighting condition used to obtain the raw image dataset (e.g., corresponding to the lighting L 1 To L M In an embodiment, the color response of the second image sensor 120B may correspond to or be affected by at least one of the spectral sensitivity of the second image sensor 120B and the noise distribution of the second image sensor 120B.

[0101] In an embodiment, operation S1203 may further include training a transformation estimator. For example, the transformation corresponding to the illumination L may be calculated. 1 To L M Multiple transformations of arrive , you can construct a transformation data set And use it to train the transformation estimator.

[0102] In operation S1204, the process 1200 may include training a second AI model using the transformed raw image data set. For example, the second AI model may be an AI model suitable for processing images captured using the second image sensor 120B.

[0103] In an embodiment, at least one of the original image dataset and the transformed original image dataset may be stored in an image database, for example, in the memory 300 .

[0104] In an embodiment, one or more operations of process 1200 may correspond to one or more operations discussed above. For example, in an embodiment, operation S1203 may correspond to Figure 7 Course 700 or Fig.11 One or more operations of process 110. For example, the original output image obtained in operation S114 may be included in the transformed original image dataset and may be used to train the second AI model.

[0105] In an embodiment, the first AI model and the second AI model may be AI models for performing illumination estimation in an input raw image. In an embodiment, illumination estimation may refer to a process of estimating the unknown illumination of a scene in order to remove the color cast in the raw image caused by the illumination. For example, the first AI model may be an AI model trained to perform illumination estimation based on an input raw image captured using the first image sensor 120A. In an embodiment, if the first AI model is used to perform illumination estimation on a raw image captured using the second image sensor 120B, the performance may be reduced due to the difference between the raw image data set used to train the first AI model and the input image captured using the second image sensor 120B, which may be referred to as a domain gap. These differences may be caused by differences between the characteristics of the first image sensor 120A and the second image sensor 120B. Therefore, process 1200 may be used to train a second AI model, and when applied to a raw image captured using the second image sensor 120B, the second AI model may have improved performance compared to the first AI model. Table 1 below shows an example of mean angular error obtained by performing illumination estimation on an image captured using the second image sensor 120B using a first AI model, a second AI model, and a third AI model trained on a raw image dataset directly captured by the second image sensor 120B.

[0106] Table 1

[0107] AI Models Testing the sensor Mean angular error (lower is better) First AI Model B 4.46 Second AI Model B 2.47(↓44.6%) The third AI model B 1.61

[0108] As can be seen from Table 1, training an AI model based on transformed image data (e.g., the second AI model) improves performance by approximately 45% compared to training an AI model based on the original image dataset (e.g., the first AI model).

[0109] Fig.13 The use application of the color transformation method according to an embodiment of the present disclosure is shown. In an embodiment, one or more operations of the process 1300 may be performed by any of the elements discussed above (eg, the system 100, the processor 200, the color transformation estimation module 220, and the color transformation module 230).

[0110] In operation S1301, process 1300 may include capturing a first raw input frame / video using first image sensor 120A. In embodiments, the first raw input frame / video may include or correspond to the raw input image discussed above. In embodiments, the first raw input frame / video may have characteristics of first image sensor 120A. For example, the first raw input frame / video may have a characteristic corresponding to input illumination L u and a color value of a color response of first image sensor 120A. In an embodiment, the color response of first image sensor 120A may correspond to or be affected by at least one of a spectral sensitivity of first image sensor 120A and a noise distribution of first image sensor 120A.

[0111] In operation S1302, process 1300 may include performing sensor mapping on a first raw input frame / video. For example, a trained transform estimator may be used to transform the first raw input frame / video into a transformed raw frame / video. In an embodiment, the transformed raw frame / video may be referred to as an output raw frame / video and may include or correspond to the output raw image discussed above. In an embodiment, the transformed raw frame / video may have characteristics of the second image sensor 120B. For example, the transformed raw frame / video may have a characteristic corresponding to the input illumination L u and a color value of a color response of the second image sensor 120B. In an embodiment, the color response of the second image sensor 120B may correspond to or be affected by at least one of a spectral sensitivity of the second image sensor 120B and a noise distribution of the second image sensor 120B.

[0112] In operation S1303, process 1300 may include capturing a second raw frame / video using second image sensor 120B. In an embodiment, the second raw frame / video may have characteristics of second image sensor 120B. For example, the second raw frame / video may have a color value corresponding to the input illumination L uAnd corresponds to at least one of the spectral sensitivity of the second image sensor 120B and the noise distribution of the second image sensor 120B.

[0113] In operation S1304, process 1300 may include obtaining a video based on the transformed original frame / video and the second original frame / video. As an example, the video may be a live stream to a display. As an example, the video may be or may include a pre-stored video. For example, in an embodiment, the first image sensor 120A and the second image sensor 120B may be included in a mobile device such as a smart phone and may be used to capture images or videos with different zoom levels or fields of view. Because the first image sensor 120A and the second image sensor 120B may have different characteristics, the video generated using the first image sensor and the second image sensor may have an inconsistent color appearance. For example, it may be difficult to produce a consistent color appearance for a video generated using the first image sensor 120A and the second image sensor 120B. For example, the mobile device may have a separate ISP to process the original frame / video captured using the first image sensor 120A and the second image sensor 120B. When the user performs a zoom function, the captured video may switch from one camera to another camera based on a certain zoom factor reached. To avoid undesirable changes in color characteristics, the ISP parameters of the separate ISPs may be carefully tuned so that they produce output images / videos with the same color characteristics. However, such tuning may be difficult to perform and implement, and may produce results that are not entirely satisfactory. Therefore, according to an embodiment, a raw frame / video captured using the first image sensor 120A may be transformed into a raw frame / video having the characteristics of the second image sensor 120B, and a single ISP may be used for both image sensors.

[0114] In an embodiment, one or more operations of process 1300 may correspond to one or more operations discussed above. For example, in an embodiment, operation S1303 may correspond to Figure 7 Course 700 or Fig.11 One or more operations of process 110, for example, the original output image obtained in operation S114 may correspond to the transformed first original image obtained in operation S1302.

[0115] Fig.14 is a diagram of an apparatus for performing color transform estimation according to an embodiment of the present disclosure. Fig.14 The system includes a user device 110, a server 110 and a network 110. The user device 110 and the server 110 may be interconnected via a wired connection, a wireless connection, or a combination of wired and wireless connections. Figure 1The electronic device shown may correspond to the user device 110 or a combination of the user device 110 and the server 110. For example, Figure 1 All or at least a portion of the processor 200 shown may be included in the server 110, and Figure 1 The remaining elements shown may be included in user equipment 110 .

[0116] User device 110 includes one or more devices configured to generate raw output images. For example, user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a camera device (e.g., Figure 1 The first camera 100A or the second camera 100B shown), a wearable device (e.g., a pair of smart glasses or a smart watch) or a similar device.

[0117] The server 110 includes one or more devices configured to receive an image and perform AI-based image processing on the image according to a request from the user device 110 to obtain a color-transformed image.

[0118] The network 110 includes one or more wired and / or wireless networks. For example, the network 110 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0119] Fig.14 The number and arrangement of devices and networks shown in are provided as examples. In practice, there may be Fig.14 The devices and / or networks shown in FIG. 1 may include additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks than those shown in FIG. Fig.14 Two or more of the devices shown in may be implemented in a single device, or Fig.14 A single device shown in the may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (eg, one or more devices) may perform one or more functions described as being performed by another set of devices.

[0120] Fig.15 According to the embodiment of the present disclosure Fig.14The electronic device 2000 may correspond to the user device 110 and / or the server 110.

[0121] The electronic device 2000 includes a bus 2010 , a processor 2020 , a memory 2030 , an interface 2040 , and a display 2050 .

[0122] The bus 2010 includes a circuit for connecting the components 2020 to 2050 to each other. The bus 2010 serves as a communication system for transmitting data between the components 2020 to 2050 or between electronic devices.

[0123] The processor 2020 includes one or more of a central processing unit (CPU), a graphics processor unit (GPU), an accelerated processing unit (APU), many integrated cores (MIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a machine learning accelerator, and a neural processing unit (NPU). The processor 2020 may be a single-core processor or a multi-core processor. The processor 2020 is capable of executing control of any one or any combination of the other components of the electronic device 2000, and / or performing communication-related operations or data processing. For example, the processor 2020 may include Figure 1 All or at least a portion of the elements of the processor 200 shown. The processor 2020 executes one or more programs stored in the memory 2030.

[0124] The memory 2030 may include volatile and / or non-volatile memory. The memory 2030 stores information related to at least one other component of the electronic device 2000 and used to drive and control the electronic device 2000, such as one or more of commands, data, programs (one or more instructions), applications 2034, etc. For example, the commands and / or data may formulate an operating system (OS) 2032. The information stored in the memory 2030 may be executed by the processor 2020. In particular, the memory 2030 may store original images and processed images (e.g., color-converted images).

[0125] Application 2034 includes the above-mentioned embodiments. Specifically, application 2034 may include a Figure 1The automatic white balance module 210, the color transformation estimation module 220 and the color transformation module 230 of the present invention and a program for performing one or more operations discussed above. These functions can be performed by a single application or multiple applications, each of which performs one or more of these functions. For example, the application 2034 may include a photo editing application. When the photo editing application receives a user request to convert the color of an original input image captured using a first image sensor to the color of an original image captured using a second sensor, the photo editing application can use the automatic white balance module to identify the input illumination of the image, estimate at least one color transformation, and apply at least one color transformation to the original input image to transform the color of the original input image to the color of the original image captured using the second sensor. The photo editing application can display and store the color-transformed image.

[0126] The display 2050 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a micro-electromechanical system (MEMS) display, or an electronic paper display. The display 2050 may also be a depth perception display, such as a multi-focus display. The display 2050 is capable of presenting, for example, various contents, such as text, images, videos, icons, and symbols.

[0127] Interface 2040 includes an input / output (I / O) interface 2042, a communication interface 2044, and / or one or more sensors 2046. I / O interface 2042 serves as an interface that can transmit commands and / or data between a user and / or other external devices and other components of electronic device 2000, for example.

[0128] The communication interface 2044 can realize the communication between the electronic device 2000 and other external devices via a wired connection, a wireless connection, or a combination of a wired and wireless connection. The communication interface 2044 can allow the electronic device 2000 to receive information from another device and / or provide information to another device. For example, the communication interface 2044 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc. The communication interface 2044 can receive the original image, the processed image, and the target illumination from the external device or send the original image, the processed image, and the target illumination to the external device.

[0129] The sensor 2046 of the interface 2040 can measure physical quantities or detect the activation state of the electronic device 2000, and convert the measured or detected information into an electrical signal. For example, the sensor 2046 may include one or more cameras (e.g., Figure 1) or other imaging sensors for capturing images of scenes. Sensor 2046 may also include any one or any combination of a microphone, a keyboard, a mouse, and one or more buttons for touch input. Sensor 2046 may also include an inertial measurement unit. In addition, (one or more) sensors 2046 may include control circuits for controlling at least one sensor included herein. Any of these sensors 2046 may be located within electronic device 2000 or coupled to electronic device 2000.

[0130] The color conversion method may be written as a computer executable program or instructions that may be stored in a medium.

[0131] The medium may store computer executable programs or instructions continuously, or temporarily store computer executable programs or instructions for execution or downloading. In addition, the medium may be any of a variety of recording media or storage media in which single or multiple pieces of hardware are combined, and the medium is not limited to media directly connected to an electronic device, but may be distributed over a network. Examples of media include magnetic media (such as hard disks, floppy disks, and magnetic tapes), optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as optical magnetic disks), and ROM, RAM, and flash memory, which are configured to store program instructions. Other examples of media include recording media and storage media managed by application stores that distribute applications or by websites, servers, etc. that provide or distribute various other types of software.

[0132] The color conversion method may be provided in the form of downloadable software. The computer program product may include a product in the form of a software program that is electronically distributed through a manufacturer or an electronic marketplace (e.g., a downloadable application). For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server or a storage medium of a server.

[0133] The model related to the above-mentioned neural network can be implemented via a software module. When the model is implemented via a software module (eg, a program module including instructions), the model can be stored in a computer-readable recording medium.

[0134] In addition, the model can be part of the above-mentioned electronic device by being integrated in the form of a hardware chip. For example, the model can be manufactured in the form of a dedicated hardware chip for artificial intelligence, or can be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-specific processor (e.g., a GPU).

[0135] The purpose of the present disclosure is to solve various technical problems, which are not limited to the above-mentioned technical problems. Those skilled in the art will easily understand any other technical problems not explicitly stated herein based on the following disclosure.

[0136] According to one aspect of the present disclosure, an electronic device 2000 for processing image data includes: at least one memory 2030, which is configured to store instructions; and at least one processor 2020, which is configured to execute instructions to: obtain an original input image captured using a first image sensor under input lighting conditions; based on the input lighting conditions, obtain a color transform that maps characteristics of the first image sensor to characteristics of a second image sensor using a transform estimator; and generate an original output image having the characteristics of the second image sensor based on the original input image and the color transform.

[0137] The characteristic of the first image sensor may include a spectral sensitivity of the first image sensor, and the characteristic of the second image sensor may include a spectral sensitivity of the second image sensor.

[0138] The transform estimator may include an artificial intelligence (AI) model trained based on multiple raw image pairs corresponding to multiple lighting conditions, and each raw image pair may include a first raw image captured using a first image sensor under a lighting condition among the multiple lighting conditions, and a second raw image captured using a second image sensor under the lighting condition.

[0139] A plurality of raw image pairs may be obtained using a variable light source configured to apply a plurality of lighting conditions to the object.

[0140] The object may include a color calibration pattern including a plurality of colors, and the color calibration pattern may include a plurality of textures corresponding to the plurality of colors.

[0141] The original input image and the original output image may not be subjected to white balance correction.

[0142] The electronic device 2000 may further include an input interface, and the at least one processor may be further configured to obtain a color transform and generate a raw output image based on receiving information about the second image sensor through the input interface.

[0143] The first image sensor may be included in an external device, and the second image sensor may be included in an electronic device, the electronic device may further include a communication interface configured to receive an original input image from the external device, and at least one processor may further be configured to execute instructions to convert the original input image received from the external device into an original output image having characteristics of the second image sensor based on a color transformation.

[0144] At least one processor 2020 may also be configured to execute instructions to: obtain an image dataset, which may include multiple original first images captured using a first image sensor; obtain multiple color transforms that map characteristics of the first image sensor to characteristics of a second image sensor using a transform estimator, wherein the multiple color transforms may include a color transform; create a transformed dataset based on the multiple original first images and the multiple color transforms, the transformed dataset may include multiple original second images having characteristics of the second image sensor, wherein the multiple original second images may include an original output image; and input the transformed dataset into an artificial intelligence (AI)-based image processing model to train the AI-based image processing model.

[0145] The electronic device 2000 may also include a first image sensor and a second image sensor, the original input image may be a first original input video frame, the original output image may be an original output video frame, and the at least one processor may also be configured to: obtain a second original input video frame captured using the second image sensor; and generate a video based on the original output video frame and the second original input video frame.

[0146] Fig.16 A method 1600 for processing image data according to an embodiment of the present disclosure is shown.

[0147] According to one aspect of the present disclosure, a method 1600 for processing image data is executed by at least one processor, including: obtaining S1602 an original input image captured using a first image sensor under input lighting conditions; based on the input lighting conditions, obtaining S1604 a color transform that maps characteristics of the first image sensor to characteristics of a second image sensor using a transform estimator; and generating S1606 an original output image having the characteristics of the second image sensor based on the original input image and the color transform.

[0148] The characteristic of the first image sensor may include a spectral sensitivity of the first image sensor, and the characteristic of the second image sensor may include a spectral sensitivity of the second image sensor.

[0149] The transform estimator may include an artificial intelligence (AI) model trained based on multiple raw image pairs corresponding to multiple lighting conditions, and each raw image pair may include a first raw image captured using a first image sensor under a lighting condition among the multiple lighting conditions, and a second raw image captured using a second image sensor under the lighting condition.

[0150] A plurality of raw image pairs may be obtained using a variable light source configured to apply a plurality of lighting conditions to the object.

[0151] The object may include a color calibration pattern including a plurality of colors, and the color calibration pattern may include a plurality of textures corresponding to the plurality of colors.

[0152] A color transform may be obtained and a raw output image generated based on information received through the input interface regarding the second image sensor.

[0153] The first image sensor may be included in a first electronic device, and the second image sensor may be included in a second electronic device including at least one processor, the second electronic device may include a communication interface configured to receive an original input image from the first electronic device, and the method may also include converting the original input image received from the first electronic device into an original output image having characteristics of the second image sensor based on a color transformation.

[0154] The method may also include: obtaining an image dataset, which may include multiple original first images captured using the first image sensor; obtaining multiple color transforms that map characteristics of the first image sensor to characteristics of the second image sensor using a transform estimator, wherein the multiple color transforms may include color transforms; creating a transformed dataset including multiple original second images having characteristics of the second image sensor based on the multiple original first images and the multiple color transforms, wherein the multiple original second images may include an original output image; and inputting the transformed dataset into an artificial intelligence (AI)-based image processing model to train the AI-based image processing model.

[0155] The at least one processor, the first image sensor and the second image sensor may be included in an electronic device, the original input image may be a first original input video frame, the original output image may be an original output video frame, and the method may further include: obtaining a second original input video frame captured using the second image sensor; and generating a video based on the original output video frame and the second original input video frame.

[0156] According to one aspect of the present disclosure, a computer-readable medium is configured to store instructions that, when executed by at least one processor of a device for processing image data, cause the at least one processor to: obtain an original input image captured using a first image sensor under input lighting conditions; based on the input lighting conditions, obtain a color transform using a transform estimator that maps characteristics of the first image sensor to characteristics of a second image sensor; and generate an original output image having the characteristics of the second image sensor based on the original input image and the color transform.

[0157] Although embodiments of the present disclosure have been described with reference to the accompanying drawings, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope defined by the following claims.

Claims

1. An electronic device (2000) for processing image data, the electronic device (2000) comprising: at least one memory (2030) configured to store instructions; as well as At least one processor (2020) is configured to execute the instructions to: obtaining an original input image captured using a first image sensor under input lighting conditions; obtaining, using a transform estimator, a color transform that maps a characteristic of the first image sensor to a characteristic of a second image sensor based on the input lighting condition; as well as An original output image having the characteristics of the second image sensor is generated based on the original input image and the color transform.

2. The electronic device (2000) according to claim 1, wherein: The characteristic of the first image sensor includes a color response of the first image sensor, the color response corresponding to a spectral sensitivity of the first image sensor, and The characteristic of the second image sensor includes a color response of the second image sensor, and the color response corresponds to a spectral sensitivity of the second image sensor.

3. The electronic device (2000) according to any one of claims 1 to 2, wherein: The transform estimator includes an artificial intelligence (AI) model trained based on a plurality of raw image pairs corresponding to a plurality of lighting conditions, and Each original image pair includes a first original image captured using the first image sensor under an illumination condition among the multiple illumination conditions, and a second original image captured using the second image sensor under the illumination condition.

4. The electronic device (2000) according to claim 3, wherein: The plurality of raw image pairs are obtained using a variable light source configured to apply the plurality of lighting conditions to an object.

5. The electronic device (2000) according to claim 4, wherein: The object includes a color calibration pattern including a plurality of colors, and The color calibration pattern includes a plurality of textures corresponding to the plurality of colors.

6. The electronic device (2000) according to any one of claims 1 to 5, wherein: The original input image and the original output image are not subjected to white balance correction.

7. The electronic device (2000) according to any one of claims 1 to 6, further comprising an input interface, in, The at least one processor is further configured to obtain the color transform and generate the raw output image based on receiving information about the second image sensor through the input interface.

8. A method (1600) for processing image data, the method being performed by at least one processor and comprising: Obtaining ( S1602 ) an original input image captured using a first image sensor under input lighting conditions; Based on the input lighting condition, obtaining (S1604) a color transform that maps a characteristic of the first image sensor to a characteristic of a second image sensor using a transform estimator; as well as An original output image having characteristics of the second image sensor is generated ( S1606 ) based on the original input image and the color transformation.

9. The method (1600) of claim 8, wherein: The characteristic of the first image sensor includes a color response of the first image sensor, the color response corresponding to a spectral sensitivity of the first image sensor, and The characteristic of the second image sensor includes a color response of the second image sensor, and the color response corresponds to a spectral sensitivity of the second image sensor.

10. The method (1600) according to any one of claims 8 to 9, wherein: The transform estimator includes an artificial intelligence (AI) model trained based on a plurality of raw image pairs corresponding to a plurality of lighting conditions, and Each original image pair includes a first original image captured using the first image sensor under an illumination condition among the multiple illumination conditions, and a second original image captured using the second image sensor under the illumination condition.

11. The method (1600) according to any one of claims 8 to 10, wherein: The color transform is obtained and the raw output image is generated based on information about the second image sensor received through an input interface.

12. The method (1600) according to any one of claims 8 to 11, wherein: The first image sensor is included in a first electronic device, and the second image sensor is included in a second electronic device including the at least one processor, The second electronic device comprises a communication interface configured to receive the original input image from the first electronic device, and The method further includes converting the original input image received from the first electronic device into the original output image having the characteristics of the second image sensor based on the color transformation.

13. The method (1600) according to any one of claims 8 to 12, wherein: The method further comprises: obtaining an image data set including a plurality of original first images captured using the first image sensor; obtaining, using the transform estimator, a plurality of color transforms that map characteristics of the first image sensor to characteristics of the second image sensor, wherein the plurality of color transforms includes the color transform; creating a transformed data set based on the plurality of original first images and the plurality of color transforms, the transformed data set comprising a plurality of original second images having characteristics of the second image sensor, wherein the plurality of original second images comprises the original output image; and The transformed data set is input into an artificial intelligence (AI)-based image processing model to train the AI-based image processing model.

14. The method (1600) according to any one of claims 8 to 13, wherein: The at least one processor, the first image sensor, and the second image sensor are included in an electronic device, The original input image includes a first original input video frame. Wherein, the original output image includes an original output video frame, and Wherein, the method further comprises: obtaining a second original input video frame captured using the second image sensor; and A video is generated based on the original output video frame and the second original input video frame.

15. A computer readable medium configured to store instructions which, when executed by at least one processor of an apparatus for processing image data, cause the at least one processor to: obtaining an original input image captured using a first image sensor under input lighting conditions; based on the input lighting conditions, obtaining, using a transform estimator, a color transform that maps characteristics of the first image sensor to characteristics of a second image sensor; and An original output image having the characteristics of the second image sensor is generated based on the original input image and the color transform.

Images