Method and apparatus for reproducing color of image

Abstract

An image color reproduction method according to an embodiment of the present disclosure may comprise the steps of: converting a raw image signal collected through an image sensor into a raw image matrix; correcting a spectral response component of the raw image matrix; and correcting an illumination component of the raw image matrix, wherein the spectral response component is determined according to the image sensor, and the illumination component is determined according to illumination of an environment in which the raw image signal is collected.

Description

Method and device for reproducing color in images

[0001] Cross-citation with related applications

[0002] This invention claims the benefit of priority from Korean Patent Application No. 10-2023-0187620, filed on December 20, 2023, and all contents of the document in that Korean Patent Application are incorporated herein by reference.

[0003] Technology field

[0004] The present invention relates to an image reproduction method and device, and more particularly, to an image color reproduction method and device capable of obtaining a high-quality image without a color correction step for a raw image.

[0005] Raw image signals collected through an image sensor under lighting with an arbitrary color temperature can compensate for pixel-to-pixel deviations through defective pixel correction, black level compensation, and lens shading compensation.

[0006] The raw image signal with the pixel-to-pixel deviation corrected can be expressed as a raw image matrix.

[0007] The raw image matrix may be a set of matrices generated by transforming a raw image signal collected from each of a plurality of pixels included in the image sensor.

[0008] A raw image matrix can contain as components the color values ​​for each channel of the pixels contained in the image sensor.

[0009] For example, if an image sensor includes pixels with RGB channels, the raw image matrix can be expressed in a 3X1 format, and each component can mean an R channel color value, a G channel color value, and a B channel color value.

[0010] The color values ​​for each channel included in the raw image matrix may be distorted by the spectral response component (image sensor spectral response), the inherent reflectance component of the image capture object, and the lighting component, and may have values ​​different from the colors seen by the human eye under standard lighting (D50 standard lighting, color temperature 5000K).

[0011] The spectral response component may be affected by the optical lens, optical filter, and quantum efficiency (QE) of the photoelectric conversion element included in the image sensor, and may be a distortion component with different characteristics for each image sensor.

[0012] Conventional image color reproduction methods can perform white balancing and color correction on the raw image matrix.

[0013] White balancing may be a step of calculating a white balance gain matrix on the raw image matrix to match the color channel color values ​​for the achromatic area pixels.

[0014] For example, in an image sensor containing RGB channels, white balancing may be a step of multiplying the raw image matrix for the achromatic area pixels by a white balance gain matrix to match the R channel color values, G channel color values, and B channel color values.

[0015] In the conventional image color reproduction method, there was a concern that the raw image matrix for pixels other than the achromatic area pixels would be distorted by the spectral response component during the white balancing process.

[0016] The color correction step may be a step of adjusting the weight of each color value by calculating a color correction matrix on the raw image matrix.

[0017] The color correction matrix can be experimentally determined depending on the type of image sensor that collects the image signal and the color temperature conditions of the lighting from which the image signal is collected.

[0018] The color correction step may be a step of estimating the color temperature of the lighting from which the image data was collected by referring to the white balance gain, and then applying experimentally predetermined color correction matrices to the raw image matrix after completing white balancing.

[0019] In the color correction step, based on the raw image matrices to which the color correction matrices are applied, the signal values ​​for each channel included in the raw image matrices can be corrected to be similar to the color signal values ​​seen by humans under standard lighting.

[0020] However, conventional image color reproduction methods may cause distortion of the color correction matrix due to spectral response components by first performing the white balancing step before the color correction step of the raw image matrix.

[0021] In addition, the conventional image color reproduction method has a problem in that the white balance gain matrix is ​​affected by the spectral response component, and therefore different white balance gain matrices are applied to image data collected by different image sensors under the same lighting.

[0022] The white balance gain matrix is ​​used to estimate the color temperature of lighting, and if the spectral response component affects the white balance gain matrix, it may become difficult to determine the color temperature.

[0023] The technical idea of ​​the present disclosure is to provide an improved image color reproduction method and device.

[0024] The technical idea of ​​the present disclosure is to provide an image color reproduction method and device that can easily determine a white balance gain matrix and selectively apply color correction by removing a spectral response component from a raw image matrix before white balancing.

[0025] The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the descriptions below.

[0026] An image color reproduction method according to one embodiment of the present disclosure includes a step of converting a raw image signal collected through an image sensor into a raw image matrix, a step of correcting a spectral response component of the raw image matrix, and a step of correcting an illumination component of the raw image matrix in which the spectral response component is corrected, wherein the spectral response component may correspond to the image sensor, and the illumination component may correspond to illumination of an environment in which the raw image signal is collected.

[0027] According to one embodiment, the step of correcting the spectral response component may include the step of computing an inverse color response transformation matrix and the step of multiplying the raw image matrix by the inverse color response transformation matrix.

[0028] According to one embodiment, when the image sensor is a 3-channel image sensor, the inverse color response transformation matrix may be a 3X3 matrix.

[0029] According to one embodiment, when the image sensor is an N-channel image sensor, the inverse color response transformation matrix may be a 3XN matrix.

[0030] According to one embodiment, the step of computing the inverse color response transformation matrix includes the steps of collecting a first raw image signal by photographing a preset subject under a preset reference illumination, converting the first raw image signal into a first raw image matrix, and algebraically computing the inverse color response transformation matrix based on a reference image matrix for the preset subject under the preset reference illumination and the first raw image matrix, wherein the reference image matrix may be a matrix including ideal image signal values ​​for the preset subject.

[0031] According to one embodiment, the step of computing the inverse color response transformation matrix includes the steps of collecting a first raw image signal by photographing a preset subject under a preset reference illumination, converting the first raw image signal into a first raw image matrix, multiplying the first raw image matrix by the inverse color response transformation matrix, computing a first loss function based on a difference between the first raw image matrix multiplied by the inverse color response transformation matrix and a reference image matrix in a color space, and recursively computing the inverse color response transformation matrix by reflecting the first loss function, wherein the reference image matrix may be a matrix including ideal image signal values ​​for the preset subject.

[0032] According to one embodiment, the color space may be any one of an RGB space, a YUV space, an XYZ space, or a Lab space.

[0033] According to one embodiment, the step of calculating the inverse color response transformation matrix comprises the steps of: photographing a preset subject under a preset reference lighting to collect a first raw image signal; converting the first raw image signal into a first raw image matrix; photographing the preset subject under any lighting to collect a second raw image signal; converting the second raw image signal into a second raw image matrix; multiplying the first raw image matrix by the inverse color response transformation matrix; calculating a first loss function based on a difference between the first raw image matrix multiplied by the inverse color response transformation matrix and a reference image matrix in a color space; multiplying the second raw image matrix by the inverse color response transformation matrix; white balancing the second raw image matrix multiplied by the inverse color response transformation matrix; calculating a second loss function based on a difference between the second raw image matrix on which the white balancing is performed and the reference image matrix in the color space; and calculating the inverse color response transformation matrix by reflecting the first loss function and the second loss function. A step of performing a recursive operation is included, wherein the reference image matrix may be a matrix including ideal image signal values ​​for the preset subject.

[0034] According to one embodiment, the color space may be any one of an RGB space, a YUV space, an XYZ space, or a Lab space.

[0035] An image color reproduction device according to another embodiment of the present disclosure includes an image matrix generation unit that converts a raw image signal collected by an image sensor into a raw image matrix, an inverse color response conversion unit that corrects a spectral response component of the raw image matrix, and a white balancing unit that corrects an illumination component of the raw image matrix in which the spectral response component is corrected, wherein the spectral response component may be determined according to the image sensor, and the illumination component may be determined according to illumination of an environment in which the raw image matrix is ​​collected.

[0036] According to another embodiment, the inverse color response transformation unit can correct the raw image matrix using an inverse color response transformation matrix.

[0037] According to another embodiment, the image color reproduction device may further include a color response correction unit that obtains the inverse color response transformation matrix.

[0038] According to another embodiment, the image matrix generation unit converts a first raw image signal collected by the image sensor by photographing a preset subject under preset reference lighting into a first raw image matrix, and the color response correction unit algebraically calculates the inverse color response transformation matrix based on the first raw image matrix and a reference image matrix for the preset subject under the reference lighting, and the reference image matrix may be a matrix including ideal image signal values ​​for the preset subject.

[0039] According to another embodiment, the image matrix generation unit converts a first raw image signal collected by the image sensor by photographing a preset subject under a preset reference illumination into a first raw image matrix, the color response correction unit multiplies the first raw image matrix by the inverse color response transformation matrix, calculates a first loss function based on a difference between the first raw image matrix multiplied by the inverse color response transformation matrix and a reference image matrix in a color space, and recursively calculates the inverse color response transformation matrix by reflecting the first loss function, and the reference image matrix may be a matrix including ideal image signal values ​​for the preset subject.

[0040] According to another embodiment, the image matrix generation unit converts a first raw signal collected by the image sensor by photographing a preset subject under a preset reference illumination into a first raw image matrix, converts a second raw image signal collected by the image sensor by photographing the preset subject under an arbitrary illumination into a second raw image matrix, and the color response correction unit multiplies the first raw image matrix by the inverse color response transformation matrix, and calculates a first loss function based on a difference between the first raw image matrix multiplied by the inverse color response transformation matrix and the reference image matrix in a color space, multiplies the second raw image matrix by the inverse color response transformation matrix, and performs white balancing on the second raw image matrix multiplied by the inverse color response transformation matrix, and calculates a second loss function based on a difference between the second raw image matrix on which the white balancing was performed and the reference image matrix in the color space, and recursively calculates the inverse color response transformation matrix by reflecting the first loss function and the second loss function, and the reference image matrix is It may be a matrix containing ideal image signal values ​​for the subject.

[0041] The image color reproduction method and device of the present disclosure can prevent distortion of a white balancing gain matrix due to spectral response by removing the influence of spectral response from a raw image matrix before performing white balancing.

[0042] The image color reproduction method and device of the present disclosure can achieve natural image color reproduction even when collecting a raw image signal in an extremely low-light situation or collecting a raw image signal with an extremely limited achromatic region by first removing distortion caused by a spectral response from a raw image matrix before performing white balancing.

[0043] In addition, the image color reproduction method and device of the present disclosure can achieve excellent color reproduction without color correction.

[0044] In addition, various effects may be provided, either directly or indirectly, through this document.

[0045] FIG. 1 is a flowchart illustrating an image color reproduction method according to one embodiment of the present disclosure.

[0046] FIG. 2 illustrates an operational flowchart of one embodiment of step S200 illustrated in FIG. 1.

[0047] FIG. 3 illustrates an operational flowchart of one embodiment of step S220 illustrated in FIG. 2.

[0048] FIG. 4 illustrates an operational flowchart of another embodiment for step S220 illustrated in FIG. 2.

[0049] FIG. 5 illustrates an operational flowchart of another embodiment for step S220 illustrated in FIG. 2.

[0050] FIG. 6 illustrates an image color reproduction device according to one embodiment of the present disclosure.

[0051] FIG. 7 is a block diagram of a computing system for executing an image color reproduction method according to an embodiment of the present disclosure.

[0052] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. The advantages and features of the present invention, and methods for achieving them, will become clearer with reference to the embodiments described in detail below, along with the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments.

[0053] The present invention is not limited to the embodiments, but may be implemented in various different forms, and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present invention.

[0054] In addition, when adding reference signs to components of each drawing, it should be noted that identical components are given the same signs as much as possible even if they are shown on different drawings.

[0055] In describing embodiments of the present invention, if a detailed description of a related known configuration or function is judged to hinder understanding of the embodiments of the present invention, the detailed description is omitted.

[0056] In the specification, the singular includes the plural unless the context specifically states otherwise. The words "comprises" and / or "comprising" as used in the specification do not exclude the presence or addition of one or more other components, steps, operations, and / or elements.

[0057] Embodiments of the present disclosure relate to a correction method for reproducing a raw image similar to an image taken under white light (D50 standard light), the gist of which is to remove the spectral response of a raw image matrix generated from a raw image signal before performing white balancing.

[0058] Embodiments of the present disclosure can prevent the raw image matrix from being distorted by the spectral response when performing white balancing by converting a raw image signal collected through an image sensor into a raw image matrix and removing a spectral response component included in the raw image matrix before white balancing.

[0059] Embodiments of the present disclosure can prevent color values ​​collected from pixels other than achromatic area pixels from being distorted by white balancing by removing spectral response using an inverse color response transformation matrix.

[0060] In addition, it is possible to easily remove spectral response components from raw image signals collected in low-light environments or complex lighting environments, for which it is difficult to determine white balance gain, thereby enabling high-quality image reproduction.

[0061] Methods and devices according to embodiments of the present disclosure are described with reference to FIGS. 1 to 7.

[0062] FIG. 1 is a flowchart illustrating an image color reproduction method according to one embodiment of the present disclosure.

[0063] A method for reproducing image color by correcting a raw image signal collected through an image sensor is specifically described through Fig. 1.

[0064] Referring to FIG. 1, a raw image signal is collected through an image sensor, and the collected raw image signal can be converted into a raw image matrix (S100).

[0065] An image sensor may include a plurality of pixels. Each pixel included in the image sensor may generate an electrical signal corresponding to incident light received.

[0066] Each pixel can include multiple channels, and each pixel can use the channels to separate signals according to the wavelength of incident light.

[0067] For example, for a pixel having RGB channels, the R channel color value, G channel color value, and B channel color value of incident light can be obtained. The color value for each channel can correspond to the intensity of the wavelength contained in the incident light.

[0068] For example, the R channel color value may be a component corresponding to the intensity of red light contained in the incident light, the G channel color value may mean a component corresponding to the intensity of green light contained in the incident light, and the B channel color value may mean a component corresponding to the intensity of blue light contained in the incident light.

[0069] The raw image signal collected from the image sensor can be a signal obtained by capturing an arbitrary object under lighting having an arbitrary color temperature.

[0070] The raw image signal can be collected as many times as the number of pixels included in the image sensor, and each raw image signal can include channel components matching the number of channels.

[0071] According to an embodiment, some noise may be removed from the raw image signal through defect pixel correction, black level compensation, and lens shading correction.

[0072] Defect pixel correction may be a method of replacing raw image signals of pixels having abnormal values ​​among pixels included in an image sensor based on raw image signals of surrounding pixels.

[0073] Black level compensation can be the correction of the image signal of the pixel corresponding to the darkest area based on the exact black value.

[0074] Lens shading correction may be a correction to prevent the intensity (saturation and sensitivity) of the raw image signal of pixels located at the edge of the lens from being reduced due to the physical characteristics of the lens included in the image sensor.

[0075] After removing noise from the raw image signal, the denoised image signal can be converted into a raw image matrix.

[0076] The raw image matrix may be a matrix of color values ​​for each of multiple channels contained in a raw image signal from which noise has been removed.

[0077] For example, if an image sensor includes pixels with RGB channels, the raw image matrix can be expressed in a 3X1 format, and each component can mean an R channel color value, a G channel color value, and a B channel color value.

[0078] According to one embodiment of the present disclosure, after converting a raw image signal into a raw image matrix, a spectral response component of the raw image matrix can be corrected (S200).

[0079] The raw image signal may be distorted by the spectral response component, the inherent reflectance component of the image capturing object, and the lighting component, and may be collected with values ​​that differ from the colors seen by the human eye under standard lighting (D50 standard lighting, color temperature 5000K).

[0080] The spectral response component is a distortion component caused by the image sensor that collected the raw image signal, and can be affected by factors such as the optical lenses, optical filters, and quantum efficiency (QE) of the photoelectric conversion elements contained in the image sensor. In other words, the spectral response component corresponds to the image sensor, and may be a different component depending on the image sensor.

[0081] The amount of light reflected may vary depending on the color of the subject. However, the pixels contained in an image sensor cannot distinguish the differences in reflectance based on color. The inherent reflectance component may be a distortion component caused by the differences in reflectance based on the color of the subject.

[0082] The lighting component can be a distortion caused by the lighting under which the image sensor collects the image signal. For example, the color values ​​of the image signal can be distorted depending on the color temperature of the lighting.

[0083] The raw image signal collected by the image sensor is converted into a color value (C) per channel contained in each pixel. k ) can be expressed as in <Mathematical Formula 1> below.

[0084] [Mathematical Formula 1]

[0085]

[0086] For example, among multiple channels, the R channel color value is C r It can be, and the G channel color value is C g may be, and the B channel color value is C b can be. The color value is the spectral response component (Q) with respect to the wavelength (λ). k (λ)), the illumination component (L(λ)) and the reflectance component (O(λ)).

[0087] Based on the above integral formula, the R channel color value (C r ), G channel color value (C g ) and B channel color values ​​(C b ) can be expressed as a raw image matrix as in <Mathematical Formula 2> below.

[0088] [Equation 2]

[0089]

[0090] The raw image matrix (C) can be expressed as the product of the spectral response matrix (Q), the illumination component matrix (L), and the reflectance component matrix (O).

[0091] , , ,

[0092] Each component Q contained in the spectral response matrix (Q) kλ In , k is each channel index (r, g or b), and λ can mean the long wavelength region (l), the medium wavelength region (m) and the short wavelength region (s), respectively.

[0093] The spectral response matrix (Q) written as an example is for a 3-channel (R channel, B channel and G channel) image sensor and may be a 3X3 matrix. In another embodiment, for an image sensor including an additional channel (e.g., an IR channel), the raw image matrix (C) may be a 4X1 matrix and the spectral response matrix (Q) may be a 4X3 matrix. In other words, the spectral response matrix (Q) may be an NX3 matrix (N is the number of channels).

[0094] Alternatively, if the raw image matrix (C) includes more offset rows, the raw image matrix (C) can be a matrix with (N+1) rows (where N is the number of channels of the image sensor), and the spectral response matrix (Q) can be an (N+1)X3 matrix. In addition, each component L included in the illumination component matrix (L) k In , k is each channel index (r, g or b), and each component O contained in the reflectance component matrix (O) k In , k can be each channel index (r, g or b). The illumination component matrix (L) can be modeled as a 3X(N+1) matrix depending on the form of the raw image matrix (C) and the spectral response matrix (Q).

[0095] As can be seen in <Mathematical Equation 2>, the spectral response matrix (Q) may not be calculated independently for each channel, unlike the illumination component matrix (L).

[0096] Conventional image color reproduction methods use a raw image matrix (C) corresponding to the achromatic area. gray) calculates a white balance gain matrix that ensures that all channel color values ​​included in the image have the same value, and performs white balancing by multiplying the white balance gain matrix by all raw image matrices (C).

[0097] The white balance gain matrix according to the conventional image color reproduction method can be complexly distorted by the components included in the spectral response matrix (Q).

[0098] In addition, the conventional image color reproduction method performs color correction after calculating the white balance gain matrix so as to obtain a value corresponding to the color value under standard lighting (D50 lighting).

[0099] However, in the case of conventional image color reproduction methods, there was a difficulty in that color correction was performed after the raw image matrix (C) was distorted by the components included in the spectral response matrix (Q), and components other than the illumination component (e.g., spectral response matrix components) had to be considered.

[0100] Additionally, there is a disadvantage in that it is difficult to calculate the white balance gain matrix in situations where it is difficult to obtain a raw image corresponding to an extremely low-light situation or an achromatic colorless area.

[0101] However, the image color reproduction method according to the present disclosure can prevent distortion of the raw image matrix due to the spectral response component by correcting the spectral response component of the raw image matrix before the illumination component.

[0102] The correction of the spectral response component can be expressed in a formula as shown in <Mathematical Formula 3> below.

[0103] [Equation 3]

[0104]

[0105] In the above formula, the inverse matrix (Q) of the spectral response matrix (Q) -1) can be called the inverse color response transform matrix.

[0106] For 3-channel image sensors, the inverse color response transformation matrix (Q -1 ) can be modeled as a 3X3 matrix. For image sensors that include additional channels (e.g., IR channels), the inverse color response transformation matrix (Q -1 ) can be a 3X4 matrix.

[0107] In other words, the inverse color response transformation matrix (Q -1 ) can be a 3XN matrix (N is the number of channels).

[0108] Additionally, if the raw image matrix (C) contains more offset rows, the inverse color response transformation matrix (Q -1 ) can be a 3X(N+1) matrix (where N is the number of channels of the image sensor).

[0109] Inverse color response transformation matrix (Q -1 ) can be obtained, the raw image matrix can be multiplied by the inverse color response transformation matrix to obtain an image matrix (LO) in which the spectral response component is removed and only the influence of the illumination component remains.

[0110] In addition, the white balance gain matrix required for the raw image matrix multiplied by the inverse color response transformation matrix can be expressed as in <Mathematical Formula 4> below.

[0111] [Equation 4]

[0112]

[0113] L g L is a scalar value representing an arbitrary standard illumination component. w It can be expressed as M wb may be a white balance gain matrix for compensating for lighting components.

[0114] Therefore, if we can obtain the inverse color response transformation matrix, we can correct the raw image matrix through a white balance gain matrix that contains only the illumination component.

[0115] The specific correction method of the spectral response component will be described in detail through Figs. 2 to 7.

[0116] According to one embodiment of the present disclosure, after the spectral response component of the raw image matrix is ​​corrected, the illumination component of the raw image matrix can be corrected (S300).

[0117] Unlike the conventional technology described above, the image color reproduction method of the present disclosure can correct the illumination component after correcting the spectral response component.

[0118] When the spectral response component is corrected from the raw image matrix, distortion caused by the spectral response component is eliminated when correcting the illumination component, and separate color correction may not be required after correcting the illumination component.

[0119] The white balance gain matrix used for lighting component correction is M as described above. wb It could be.

[0120] According to one embodiment of the present disclosure, after correcting the illumination component of the raw image matrix, post-processing of the raw image matrix can be performed (S400).

[0121] Post-processing of the raw image matrix may include gamma correction, edge enhancement, and color space conversion.

[0122] FIG. 2 illustrates an operational flowchart of one embodiment of step S200 illustrated in FIG. 1.

[0123] A method for correcting the spectral response component of a raw image matrix is ​​illustrated in FIG. 2.

[0124] Referring to FIG. 2, the step (S200) of correcting the spectral response component of the raw image matrix may include the step (S220) of calculating an inverse color response transformation matrix.

[0125] The inverse color response transformation matrix can be computed either algebraically or recursively.

[0126] The inverse color response transformation matrix can be calculated based on the raw image matrix and the reference image matrix. The calculation method of the inverse color response transformation matrix will be described in detail through FIGS. 3 to 7.

[0127] Once the inverse color response transformation matrix is ​​determined by step S220, the raw image matrix can be multiplied by the calculated inverse color response transformation matrix (S230). By multiplying the raw image matrix by the inverse color response transformation matrix, the spectral response component can be removed.

[0128] FIG. 3 illustrates an operational flowchart of one embodiment of step S220 illustrated in FIG. 2.

[0129] A method for algebraically computing the inverse color response transformation matrix can be explained through Fig. 3.

[0130] Referring to FIG. 3, the step of calculating the inverse color response transformation matrix (S220) may include the step of collecting a first raw image signal (S221a).

[0131] The first raw image signal may be a raw image signal obtained by photographing a preset subject under a preset reference illumination.

[0132] For example, the reference light may be a D50 standard light with a color temperature of 5000K. Alternatively, the preset subject may be a color checker with known color values.

[0133] The color checker can use a reference image matrix based on the ideal image signal values ​​for the reference illumination to compute the inverse color response transformation matrix.

[0134] After collecting the first raw image signal, the first raw image signal can be converted into a first raw image matrix (S222a).

[0135] The method for converting an image signal into an image matrix may be the same as described above in Fig. 1.

[0136] An inverse color response transformation matrix can be algebraically computed based on the reference image matrix and the first raw image matrix (S223a).

[0137] An inverse color response transformation matrix can be algebraically computed based on the first raw image matrix and the reference image matrix corresponding to the first raw image matrix. The algebraic computation method is explained with reference to <Mathematical Formula 5> below.

[0138] [Equation 5]

[0139]

[0140] In <Mathematical Formula 5>, the first raw image matrix (C1) and the inverse color response transformation matrix (Q -1 ) is the reference image matrix (LO) for a preset subject under a preset reference illumination. r ) can be collected in pairs corresponding to each other.

[0141] The inverse color response transformation matrix can be calculated using the first raw image matrix (C1) and its pseudo-inverse matrix, and can be expressed as <Mathematical Formula 6> below.

[0142] [Equation 6]

[0143]

[0144] The first raw image matrix (C1) can be obtained based on the collected first raw image signal, and the reference image matrix (LO) r ) is composed of predefined values, the first raw image matrix (C1) and the reference image matrix (LO). r ) Once enough pairs are available, the inverse color response transformation matrix (Q -1 ) can be obtained.

[0145] FIG. 4 illustrates an operational flowchart of another embodiment for step S220 illustrated in FIG. 2.

[0146] FIG. 4 is an example of collecting an inverse color response transformation matrix when collecting an image signal under reference lighting.

[0147] Referring to FIG. 4, step S220 of calculating an inverse color response transformation matrix may include a step (S221b) of collecting a first raw image signal.

[0148] As described above in FIG. 3, the first raw image signal may be a raw image signal obtained by photographing a preset subject under a preset reference illumination, and the reference illumination may be a D50 standard illumination having a color temperature of 5000 K. Additionally, the preset subject may be a color checker having a known color value.

[0149] After collecting the first raw image signal, the first raw image signal can be converted into a first raw image matrix (S222b).

[0150] The first raw image matrix transformation may be a transformation of a plurality of channel-specific color values ​​included in the first raw image signal into a matrix.

[0151] For example, if the image sensor includes pixels with RGB channels, the first raw image matrix (C1) can be expressed in a 3X1 format as follows.

[0152]

[0153] Each component (r1, g1, b1) included in the first raw image matrix (C1) may be a signal value for each channel calculated from the first raw image signal.

[0154] Afterwards, the first raw image signal matrix can be multiplied by the inverse color response transformation matrix to be obtained (S223b).

[0155] For example, the inverse color response transformation matrix (Q) for a 3-channel image sensor -1 ) can be expressed as a 3X3 matrix containing arbitrary elements (a, b, c, d, e, f, g, h, i) as follows.

[0156]

[0157] Inverse color response transformation matrix (Q -1 ) and the first raw image matrix (C1) can be called the first color response correction matrix (I1).

[0158]

[0159] Each of the components (R1, G1, B1) included in the first color response correction matrix (I1) is converted to the inverse color response transformation matrix (Q) by each of the components (r1, g1, b1) included in the first raw image matrix (C1). -1 ) may have been corrected.

[0160] In the color space, the inverse color response transformation matrix (Q -1 ) can be calculated based on the difference between the first raw image matrix (C1) multiplied by the reference image matrix (S224b).

[0161] Inverse color response transformation matrix (Q -1 ) is called the first color response correction matrix (I1) when the matrix obtained by multiplying the first raw image matrix (C1) by the first color response correction matrix (I1) is the inverse color response transformation matrix (Q) of the first raw image matrix (C1).-1 ) may be a matrix corrected by .

[0162] The reference image matrix may be a matrix whose components are ideal channel-specific color values ​​obtained when a preset subject is photographed under a preset reference illumination and there is no distortion due to a spectral response component.

[0163] Each component of the reference image matrix can have as its component an ideal color value for a preset subject, the color checker, and the ideal color value can be a value provided by the seller of the color checker.

[0164] The reference image matrix (T) can be represented as a 3X1 matrix as follows.

[0165]

[0166] As explained above, each component (R) contained in the reference image matrix (T) i , G i , B i ) may be the ideal color value provided by the seller of the color checker.

[0167] A color space is a space that expresses a color display system in three dimensions, and can include RGB space, YUV space, XYZ space, or Lab space.

[0168] The first color response correction matrix (I1) and the reference image matrix (T) can be expressed as arbitrary vectors in the color space.

[0169] According to an embodiment, a first loss function (E1) can be computed based on the difference between a first color response correction matrix (I1) and a corresponding reference image matrix (T) in a color space.

[0170] At this time, the difference between the first color response correction matrix (I1) and the reference image matrix (T) in the color space can be obtained using angular error, cosine error, MSE (mean squared error), or MAE (mean absolute error).

[0171] An exemplary first loss function calculation method may be as shown in <Mathematical Formula 7> below.

[0172] [Equation 7]

[0173]

[0174] The first loss function (E1) calculation method of <Mathematical Formula 7> may be based on the cosine error. The inverse color response transformation matrix (Q -1 ) is multiplied by the first raw image matrix and the reference image matrix, and the first loss function (E1) can be calculated based on the sum of the differences.

[0175] The inverse color response transformation matrix can be computed recursively by reflecting the computed first loss function (E1) (S225b).

[0176] Recursively computing the inverse color response transformation matrix means that the inverse color response transformation matrix (Q) that was multiplied to the first raw image matrix (C1) to compute the first loss function (E1) -1 ) is corrected based on the first loss function (E1), and the corrected inverse color response transformation matrix (Q -1 ) can be used again to go through steps S223b to S224b. More specifically, the inverse color response transformation matrix (Q) can be used to reduce the first loss function (E1). -1 ) can be retrospectively corrected.

[0177] A more accurate inverse color response transformation matrix can be obtained through recursive operations.

[0178] FIG. 5 illustrates an operational flowchart of another embodiment for step S220 illustrated in FIG. 2.

[0179] FIG. 5 is an example of collecting an inverse color response transformation matrix when collecting image signals under multiple illuminations.

[0180] Any overlapping content described in Figures 3 and 4 above is omitted.

[0181] Referring to FIG. 5, step S220 of calculating an inverse color response transformation matrix may include a step (S221c) of collecting a first raw image signal.

[0182] As described above in FIG. 3, the first raw image signal may be a raw image signal obtained by photographing a preset subject under a preset reference illumination.

[0183] Additionally, the reference light may be a D50 standard light with a color temperature of 5000K, and the preset subject may be a color checker with known color values.

[0184] After collecting the first raw image signal, the first raw image signal can be converted into a first raw image matrix (S222c).

[0185] The first raw image matrix transformation may be a transformation of a plurality of channel-specific color values ​​included in the first raw image signal into a matrix.

[0186] As described in Fig. 4, the first raw image matrix (C1) can be expressed in a 3X1 format as follows.

[0187]

[0188] Step S220 of calculating the inverse color response transformation matrix may include a step (S223c) of collecting a second raw image signal.

[0189] The second raw image signal may be a raw image signal obtained by photographing a preset subject under arbitrary lighting.

[0190] The preset subject may be a color checker whose color values ​​are known as described above.

[0191] After collecting the second raw image signal, the second raw image signal can be converted into a second raw image matrix (S224c).

[0192] Afterwards, the first raw image signal matrix can be multiplied by the inverse color response transformation matrix to be obtained (S225c).

[0193] Inverse color response transformation matrix (Q -1 ) can be expressed as a 3X3 matrix containing arbitrary elements (a, b, c, d, e, f, g, h, i) as follows.

[0194]

[0195] Inverse color response transformation matrix (Q -1 ) and the first raw image matrix (C1) can be called the first color response correction matrix (I1).

[0196]

[0197] Each of the components (R1, G1, B1) included in the first color response correction matrix (I1) is converted to the inverse color response transformation matrix (Q) by each of the components (r1, g1, b1) included in the first raw image matrix (C1). -1 ) may have been corrected.

[0198] In the color space, the inverse color response transformation matrix (Q -1 ) can be calculated based on the difference between the first raw image matrix (C1) multiplied by the reference image matrix (S226c).

[0199] The calculation method of the first loss function may be substantially the same as that described in Fig. 4.

[0200] Afterwards, the second raw image signal matrix can be multiplied by the inverse color response transformation matrix to be obtained (S227c).

[0201]

[0202] Inverse color response transformation matrix (Q -1 ) is as above, and the second raw image matrix (C2) may be as follows.

[0203]

[0204] Additionally, the inverse color response transformation matrix (Q -1 ) and the second raw image matrix (C2) can be called the second color response correction matrix (I2).

[0205]

[0206] The second color response correction matrix (I2) is the second raw image matrix (C2) converted to the inverse color response transformation matrix (Q -1 ) may be a matrix corrected by .

[0207] The second color response correction matrix (I2) is a matrix calculated based on the second raw image matrix (C2) collected under arbitrary lighting, and may have distortion due to lighting components.

[0208] Therefore, distortion caused by lighting components may need to be corrected.

[0209] White balancing can be performed on the second raw image signal matrix multiplied by the inverse color response transformation matrix (S228c).

[0210] White balancing can be performed by multiplying the white balance gain matrix described above. In step S227c, the inverse color response transformation matrix (Q -1) by correcting the second color response correction matrix (I2), so that only the lighting component can be removed by the white balance gain matrix.

[0211] Thereafter, a second loss function can be calculated based on the difference between the second raw image matrix on which white balancing has been performed and the reference image matrix in the color space (S229c).

[0212] The reference image matrix may be a matrix whose components are ideal channel-specific color values ​​obtained when a preset subject is photographed under a preset reference illumination and there is no distortion due to a spectral response component.

[0213] Each component of the reference image matrix can have as its component an ideal color value for a preset subject, the color checker, and the ideal color value can be a value provided by the seller of the color checker.

[0214] A color space is a space that expresses a color display system in three dimensions, and can include RGB space, YUV space, XYZ space, or Lab space.

[0215] The second color response correction matrix (I2) and the reference image matrix (T) on which white balancing is performed can be expressed as arbitrary vectors in the color space.

[0216] According to an embodiment, a second loss function (E2) can be calculated based on the difference between a second color response correction matrix (I2) that has performed white balancing on a color space and a corresponding reference image matrix (T).

[0217] As explained above, the difference between the second color response correction matrix (I2) and the reference image matrix (T) in the color space can be obtained using angular error, cosine error, mean squared error (MSE), or mean absolute error (MAE).

[0218] The second color response correction matrix (I2) on which white balancing is performed can be calculated using the first loss function as shown in <Mathematical Formula 8> below.

[0219] [Equation 8]

[0220]

[0221] The calculation method of the second loss function (E2) of <Mathematical Formula 8> may be based on the cosine error. The inverse color response transformation matrix (Q -1 ) and calculate the difference (△e2) between the second raw image matrix that has undergone white balancing and the reference image matrix, and a second loss function (E2) can be calculated based on the sum of the differences.

[0222] The inverse color response transformation matrix can be retrospectively calculated by reflecting the calculated first loss function (E1) and second loss function (E2) (S230c).

[0223] Recursively computing the inverse color response transformation matrix means that the inverse color response transformation matrix (Q) that was multiplied by the first raw image matrix (C1) and the second raw image matrix (C2) to compute the first loss function (E1) and the second loss function (E2) -1 ) is corrected based on the first loss function (E1) and the second loss function (E2), and the corrected inverse color response transformation matrix (Q -1 ) may mean going through steps S225c to S229c again. More specifically, the inverse color response transformation matrix (Q) is used to reduce the sum of the first loss function (E1) and the second loss function (E2). -1 ) can be retrospectively corrected.

[0224] FIG. 6 illustrates an image color reproduction device according to one embodiment of the present disclosure.

[0225] An image color reproduction device (100) according to one embodiment of the present disclosure may include an image matrix generation unit (110) that receives an image signal from an image sensor that collects a raw image signal and converts the received raw image signal into a raw image matrix.

[0226] The image matrix generation unit (110) can perform bad pixel correction, black level correction, and lens shading correction on the raw image signal collected by the image sensor.

[0227] Noise in the raw image signal can be removed through bad pixel correction, black level correction, and lens shading correction. After removing noise, the image matrix generation unit (110) can generate a matrix whose components include color values ​​for each of the multiple channels included in the raw image signal.

[0228] For example, if the pixels included in the image sensor include RGB channels, the image matrix generation unit (110) can convert the raw image signal with noise removed into a 3X1 matrix. At this time, each component included in the 3X1 matrix can mean an R channel color value, a G channel color value, and a B channel color value.

[0229] The image color reproduction device (100) may include a color response correction unit (120) that calculates an inverse color response transformation matrix.

[0230] The color response correction unit (120) can calculate an inverse color response transformation matrix based on the first raw image matrix and the reference image matrix corresponding to the first raw image matrix.

[0231] The first raw image matrix may be a matrix generated based on a first raw image signal obtained by photographing a preset subject under a preset reference illumination.

[0232] For example, the reference light may be a D50 standard light with a color temperature of 5000K. Alternatively, the preset subject may be a color checker with known color values.

[0233] According to one embodiment, the inverse color response transformation matrix can be computed algebraically or recursively based on the first raw image matrix and the reference image matrix.

[0234] In another embodiment, the inverse color response transformation matrix can be computed recursively using the second raw image matrix in addition to the first raw image matrix and the reference image matrix.

[0235] The second raw image matrix may be a matrix generated based on a second raw image signal obtained by photographing a preset subject under arbitrary lighting.

[0236] The algebraic operation method and the recursive operation method of the inverse color response transformation matrix have been described above through Figures 3 to 5, and thus a detailed description thereof is omitted.

[0237] The image color reproduction device (100) may include an inverse color response transformation unit (130) that corrects the spectral response component of the raw image matrix.

[0238] The inverse color response transformation unit (130) can remove the spectral response component of the raw image matrix using the inverse color response transformation matrix.

[0239] An image color reproduction device (100) according to one embodiment of the present disclosure can remove a spectral response component in advance before white balancing by including an inverse color response converter (130).

[0240] Since the spectral response component is removed in advance, distortion of the raw image matrix due to the spectral response component may not occur when performing white balancing.

[0241] In addition, by removing the spectral response component that has different characteristics for each image sensor, only the lighting component can be left in the raw image matrix, enabling intuitive image color reproduction.

[0242] An image color reproduction device according to one embodiment of the present disclosure can obtain an inverse color response transformation matrix corresponding to each image sensor even if the image sensor is replaced / changed, and can perform image color reproduction based on the inverse color response transformation matrix, so that image color reproduction can be made simpler than a conventional image color reproduction method.

[0243] In addition, image colors can be easily reproduced even if the spectral response characteristics of the replaced image sensor are different from the spectral response characteristics of the image sensor before replacement.

[0244] The image color reproduction device (100) may include a white balancing unit (130) that removes the lighting component after calculating the inverse color response transformation matrix.

[0245] Unlike conventional image color reproduction methods, the white balancing unit (130) removes the lighting component of the raw image matrix after the spectral response component is removed, so that white balancing distortion may not occur in the raw image matrix due to the spectral response component.

[0246] The image color reproduction device (100) may further include an image post-processing unit (not shown) that post-corrects a raw image matrix from which lighting components have been removed through white balancing. According to an embodiment, the image post-processing unit may perform gamma correction, edge enhancement, and color space conversion.

[0247] FIG. 7 is a block diagram of a computing system for executing an image color reproduction method according to an embodiment of the present disclosure.

[0248] Referring to FIG. 7, the image color reproduction method according to an embodiment of the present disclosure described above may also be implemented through a computing system. The computing system (2000) may include at least one processor (2100), memory (2300), user interface input device (2400), user interface output device (2500), storage (2600), and network interface (2700) connected via a system bus (2200).

[0249] The processor (2100) may be a central processing unit (CPU) or a semiconductor device that executes processing on instructions stored in memory (2300) and / or storage (2600). The memory (2300) and storage (2600) may include various types of volatile or non-volatile storage media. For example, the memory (2300) may include a read-only memory (ROM) (2310) and a random access memory (RAM) (2320).

[0250] Accordingly, the steps of a method or algorithm described in connection with the embodiments disclosed herein may be implemented directly in hardware, a software module, or a combination of the two executed by the processor (2100). The software module may reside in a storage medium (i.e., memory (2300) and / or storage (2600)), such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM. An exemplary storage medium is coupled to the processor (2100), such that the processor (2100) can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral with the processor (2100). The processor (2100) and the storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within a user terminal. Alternatively, the processor (2100) and the storage medium may reside as discrete components within the user terminal.

[0251] While the embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical spirit or essential characteristics thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. A step of converting a raw image signal collected through an image sensor into a raw image matrix; a step of correcting the spectral response component of the above raw image matrix; and Comprising a step of correcting the illumination component of the raw image matrix whose spectral response component is corrected, The above spectral response component corresponds to the image sensor, The above illumination component is an image color reproduction method corresponding to the illumination of the environment in which the raw image signal is collected.

2. In paragraph 1, The step of correcting the above spectral response component is: a step of computing an inverse color response transformation matrix; and An image color reproduction method comprising the step of multiplying the raw image matrix by the inverse color response transformation matrix.

3. In paragraph 2, An image color reproduction method wherein the inverse color response transformation matrix is ​​a 3X3 matrix when the image sensor is a 3-channel image sensor.

4. In paragraph 2, An image color reproduction method wherein the inverse color response transformation matrix is ​​a 3XN matrix when the image sensor is an N-channel image sensor.

5. In paragraph 2, The step of calculating the above inverse color response transformation matrix is A step of collecting a first raw image signal by photographing a preset subject under a preset reference illumination; a step of converting the first raw image signal into a first raw image matrix; and A step of algebraically computing the inverse color response transformation matrix based on the reference image matrix for the preset subject and the first raw image matrix under the preset reference illumination, An image color reproduction method, wherein the above reference image matrix is ​​a matrix containing ideal image signal values ​​for the preset subject.

6. In paragraph 2, The step of calculating the above inverse color response transformation matrix is A step of collecting a first raw image signal by photographing a preset subject under a preset reference illumination; A step of converting the first raw image signal into a first raw image matrix; A step of multiplying the first raw image matrix by the inverse color response transformation matrix; In the color space, calculating a first loss function based on the difference between the first raw image matrix and the reference image matrix multiplied by the inverse color response transformation matrix; and A step of recursively computing the inverse color response transformation matrix by reflecting the first loss function, An image color reproduction method, wherein the above reference image matrix is ​​a matrix containing ideal image signal values ​​for the preset subject.

7. In paragraph 6, The above color space is an image color reproduction method that is one of RGB space, YUV space, XYZ space, or Lab space.

8. In paragraph 2, The step of calculating the above inverse color response transformation matrix is A step of collecting a first raw image signal by photographing a preset subject under a preset reference illumination; A step of converting the first raw image signal into a first raw image matrix; A step of collecting a second raw image signal by photographing the preset subject under any lighting; A step of converting the second raw image signal into a second raw image matrix; A step of multiplying the first raw image matrix by the inverse color response transformation matrix; In the color space, a step of computing a first loss function based on the difference between the first raw image matrix and the reference image matrix multiplied by the inverse color response transformation matrix; A step of multiplying the second raw image matrix by the inverse color response transformation matrix; A step of white balancing the second raw image matrix multiplied by the above inverse color response transformation matrix; In the color space, a step of calculating a second loss function based on the difference between the second raw image matrix on which the white balancing is performed and the reference image matrix; and A step of recursively computing the inverse color response transformation matrix by reflecting the first loss function and the second loss function, An image color reproduction method, wherein the above reference image matrix is ​​a matrix containing ideal image signal values ​​for the preset subject.

9. In paragraph 8, The above color space is an image color reproduction method that is one of RGB space, YUV space, XYZ space, or Lab space.

10. Image matrix generation unit that converts the raw image signal collected by the image sensor into a raw image matrix; An inverse color response transform unit for correcting the spectral response component of the above raw image matrix; and A white balancing unit is included for correcting the illumination component of the raw image matrix in which the spectral response component is corrected. The above spectral response component is determined according to the image sensor, The above illumination component is an image color reproduction device determined according to the illumination of the environment in which the raw image matrix was collected.

11. In clause 10, The above inverse color response transformation unit is an image color reproduction device that corrects the raw image matrix using an inverse color response transformation matrix.

12. In paragraph 11, An image color reproduction device further comprising a color response correction unit for obtaining the above inverse color response transformation matrix.

13. In paragraph 12, The above image matrix generation unit converts a first raw image signal collected by the image sensor by photographing a preset subject under preset reference lighting into a first raw image matrix, The above color response correction unit algebraically calculates the inverse color response transformation matrix based on the reference image matrix for the preset subject and the first raw image matrix under the reference lighting, An image color reproduction device wherein the above reference image matrix is ​​a matrix containing ideal image signal values ​​for the preset subject.

14. In paragraph 12, The above image matrix generation unit converts a first raw image signal collected by the image sensor by photographing a preset subject under preset reference lighting into a first raw image matrix, The color response correction unit multiplies the first raw image matrix by the inverse color response transformation matrix, calculates a first loss function based on the difference between the first raw image matrix multiplied by the inverse color response transformation matrix and the reference image matrix in the color space, and recursively calculates the inverse color response transformation matrix by reflecting the first loss function. An image color reproduction device wherein the above reference image matrix is ​​a matrix containing ideal image signal values ​​for the preset subject.

15. In paragraph 14, The above image matrix generation unit converts a first raw signal collected by the image sensor by photographing a preset subject under preset reference lighting into a first raw image matrix, and converts a second raw image signal collected by the image sensor by photographing the preset subject under arbitrary lighting into a second raw image matrix. The color response correction unit multiplies the first raw image matrix by the inverse color response transformation matrix, calculates a first loss function based on a difference between the first raw image matrix multiplied by the inverse color response transformation matrix and the reference image matrix in a color space, multiplies the second raw image matrix by the inverse color response transformation matrix, performs white balancing on the second raw image matrix multiplied by the inverse color response transformation matrix, calculates a second loss function based on a difference between the second raw image matrix on which the white balancing was performed and the reference image matrix in the color space, and recursively calculates the inverse color response transformation matrix by reflecting the first loss function and the second loss function. An image color reproduction device wherein the above reference image matrix is ​​a matrix containing ideal image signal values ​​for the preset subject.

Images