Image processing method and apparatus

Abstract

An image processing method and apparatus are disclosed. The image processing method includes receiving an image including a plurality of frames taken over time in a light environment including alternating current (AC) light, extracting AC pixels corresponding to the AC light from a plurality of pixels in the image, estimating a visual space of the AC pixels based on values of the AC pixels in the plurality of frames, estimating information of the AC light included in the image based on the visual space estimation, and processing the image based on the information of the AC light.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0060985, filed May 24, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference for all purposes. TECHNICAL FIELD

[0002] The following description relates to an image processing method and apparatus for estimating a color of an alternating current (AC) light source using a high-speed camera. BACKGROUND

[0003] According to a statistical-based color constancy technique, a light is estimated using statistical properties of an image. For example, a statistical-based light estimation technique corrects a light by correcting red, green, and blue (RGB) average values of an input image to 1:1:1 (assuming that an average RGB ratio of the image is achromatic). Since a statistical-based research requires low technical complexity, a fast algorithm operation is provided, and many researches have been conducted, some examples of which are gray world, gray edge, gray shade, and gray pixel.

[0004] When a statistical-based technique is not applicable to a corresponding statistical model despite the low complexity of the statistical-based technique, a color performance is greatly distorted. In general, in order to accurately perform light estimation, various colors should be complexly present in an image. SUMMARY

[0005] This summary is provided to introduce a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0006] In one general aspect, there is provided an image processing method including receiving an image, the image including a plurality of frames taken over time in a light environment including an alternating current (AC) light; extracting, from a plurality of pixels in the image, AC pixels corresponding to the AC light; estimating a visual space of the AC pixels based on values of the AC pixels in the plurality of frames; estimating information of the AC light included in the image based on the visual space; and processing the image based on the information of the AC light.

[0007] The estimating of the visual space can include estimating a visual space indicating an illumination component and a diffuse component of the AC pixels in the plurality of frames.

[0008] The estimating of the visual space can include estimating a dichromatic plane of the AC pixels based on a dichromatic model.

[0009] The values of the AC pixels can comprise red component values, green component values and blue component values of the AC pixels, and the step of estimating the visual space can comprise estimating the visual space based on a linear combination of the red component values, the green component values and the blue component values of the AC pixels in the plurality of frames.

[0010] The step of estimating the visual space can comprise extracting parameters of the visual space that minimize a perpendicular distance between a plane and the values of the AC pixels in the plurality of frames.

[0011] The parameters can be extracted based on a least squares method.

[0012] The step of estimating the information of the AC light can comprise estimating color information of the AC light.

[0013] The information of the AC light can comprise a ratio of red component, green component and blue component of the AC light.

[0014] The step of processing the image can comprise correcting a color of the image based on the color information of the AC light.

[0015] The step of extracting the AC pixels can comprise extracting the AC pixels from the plurality of pixels that exhibit a signal distortion due to noise that is less than a threshold value.

[0016] The step of extracting the AC pixels can comprise extracting the AC pixels based on variations in the values of the plurality of pixels in the plurality of frames.

[0017] The step of extracting the AC pixels can comprise modeling the values of the plurality of pixels in the plurality of frames as a sinusoidal curve, calculating respective differences between the pixel values of the modeled sinusoidal curve and the values of the plurality of pixels in the plurality of frames, and extracting, from the plurality of pixels, a pixel having a sum of the calculated differences that is less than a threshold value as an AC pixel.

[0018] The step of modeling can comprise modeling the values of the plurality of pixels in the plurality of frames as a sinusoidal curve based on a Gauss-Newton method.

[0019] The step of estimating the information of the AC light can comprise determining, based on the visual space, light vector candidates corresponding to the AC light, determining, based on prior information of the AC light, a light vector from among the light vector candidates, and estimating the information of the AC light based on the light vector.

[0020] The prior information can be obtained based on Planckian locus information.

[0021] The step of estimating the information of the AC light can comprise estimating intersection lines of the visual space, estimating, based on a maximum a posteriori (MAP) estimation, an intersection line from among the intersection lines that minimizes a cost function, and estimating the information of the AC light based on the determined intersection line.

[0022] The determining step can include: calculating a probability that the intersection line is perpendicular to the visual space; and determining the intersection line that minimizes the cost function from among the intersection lines based on the prior information of the AC light and the probability.

[0023] In another general aspect, an image processing device includes a processor configured to: receive an image, the image including a plurality of frames taken over time in a light environment including an alternating current (AC) light; extract, from a plurality of pixels in the image, AC pixels corresponding to the AC light; estimate a visual space of the AC pixels based on values of the AC pixels included in the plurality of frames; estimate information of the AC light included in the image based on the visual space estimate; and process the image based on the information of the AC light.

[0024] The processor can be configured to estimate the visual space that indicates an illumination component and a diffuse component of the AC pixels in the plurality of frames.

[0025] The processor can be configured to estimate a dichromatic plane of the AC pixels based on a dichromatic model.

[0026] The values of the AC pixels can include red component values, green component values, and blue component values of the AC pixels, and the processor can be configured to estimate the visual space based on a linear combination of the red component values, the green component values, and the blue component values of the AC pixels in the plurality of frames.

[0027] The processor can be configured to extract parameters of the visual space that minimize a perpendicular distance between a plane and the values of the AC pixels in the plurality of frames.

[0028] The processor can be configured to estimate color information of the AC light.

[0029] The processor can be configured to correct a color of the image based on the color information of the AC light.

[0030] The processor can be configured to extract, from the plurality of pixels, pixels that show a signal distortion due to noise less than a threshold value as the AC pixels.

[0031] The processor can be configured to extract the AC pixels based on variations in values of the plurality of pixels included in the plurality of frames.

[0032] The processor can be configured to model the values of the plurality of pixels in the plurality of frames as sinusoidal curves; calculate respective differences between pixel values of the modeled sinusoidal curves and the values of the plurality of pixels in the plurality of frames; and extract, from the plurality of pixels, pixels having a sum of the calculated differences less than a threshold value as the AC pixels.

[0033] The processor can be configured to model the values of the plurality of pixels included in the plurality of frames as sinusoidal curves based on a Gauss-Newton method.

[0034] The processor can be configured to determine a light vector candidate corresponding to the AC light based on the visual space, determine a light vector from among the light vector candidates based on prior information of the AC light, and estimate information of the AC light based on the light vector.

[0035] The processor can be configured to estimate intersection lines of the visual space, determine an intersection line that minimizes a cost function from among the intersection lines based on maximum a posteriori (MAP) estimation, and estimate information of the AC light based on the determined intersection line.

[0036] The processor can be configured to calculate a probability that the intersection line is perpendicular to the visual space, and determine an intersection line that minimizes a cost function from among the intersection lines based on prior information of the AC light and the probability.

[0037] In another general aspect, an image processing apparatus includes a sensor configured to capture an image, the image including a plurality of frames captured over time in a light environment including an alternating current (AC) light, a processor configured to extract AC pixels corresponding to the AC light from a plurality of pixels in the image, estimate a visual space of the AC pixels based on values of the AC pixels included in the plurality of frames, determine information of the AC light included in the image based on the visual space, and process the image based on the information of the AC light, and an output configured to display the processed image.

[0038] The image processing apparatus includes a non-transitory computer-readable storage medium storing the AC pixels, the image, and instructions, in response to the processor executing the instructions, the AC pixels are extracted, the information of the AC light is determined, and the image is processed.

[0039] Other features and aspects will be apparent from the following detailed description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 An example of an image processing method is shown.

[0041] Figure 2A And Figure 2B An example of reflected light is shown.

[0042] Figure 3 An example of a visual space is shown.

[0043] Figure 4A And Figure 4B An example of estimating information of an alternating current (AC) light included in an image based on a visual space is shown.

[0044] Figure 5 An example of an image processing method is shown.

[0045] Figures 6A to 6D An example of extracting an AC pixel is shown.

[0046] Figure 7 An example of estimating a visual space is shown.

[0047] Figures 8A to 8D and Figure 9 An example of estimating information of an AC light is shown.

[0048] Figure 10 An example of an image processing method is shown.

[0049] Figure 11 An example of an image processing apparatus is shown.

[0050] Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing references will be understood to represent the same elements, features, and structures. The drawings can not be to scale, and the relative dimensions, proportions, and depiction of elements in the drawings can be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0051] The following DETAILED DESCRIPTION is provided to assist the reader in understanding the methods, apparatuses, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatuses, and / or systems described herein can be apparent to the skilled reader after understanding the disclosure provided herein. For example, the order of the operations described herein is merely an example, and is not limited to those set forth herein, but can be changed as will be apparent to one of ordinary skill in the art after understanding the disclosure provided herein, except for operations that must occur in a particular order. Also, the description of features known in the art can be omitted for the sake of clarity and conciseness.

[0052] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to show some of the many ways in which the methods, apparatuses, and / or systems described herein can be implemented after understanding the disclosure provided herein.

[0053] Although the terms of "first" or "second" are used to explain various components, the components are not limited to the terms. The terms should be used only to distinguish one component from another component. For example, within the scope of the right of the idea according to the present disclosure, a "first" component can be referred to as a "second" component, or similarly, a "second" component can be referred to as a "first" component.

[0054] In the description, although a first component can be connected, coupled, or linked to a second component, a third component can be "connected," "coupled," or "linked" between the first component and the second component if the first component "connects," "couples," or "links" to the second component. Conversely, if a first component is described as being "directly connected," "directly coupled," or "directly linked" to a second component, there can be no third component. Similarly, expressions such as "between" and "directly between" and "adjacent" and "directly adjacent" can also be interpreted in the manner described above. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0055] Words describing spatial relationships, such as "below," "under," "below," "lower," "bottom," "above," "on," "upper," "top," "left," and "right," can be used to conveniently describe the spatial relationship of one device or element to other devices or elements. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation, in addition to the orientations depicted in the figures. For example, if a device in the figures is turned over, elements described as "above" or "on" other elements or features would then be oriented "below" or "on" the other elements or features. Thus, the term "above" can encompass both an upward and a downward orientation depending on the particular orientation of the figure. The devices can be otherwise oriented (e.g., rotated 90 degrees or at other orientations) and the spatially relative descriptions used herein interpreted accordingly.

[0056] Descriptions with respect to any one axis (x-axis, y-axis, or z-axis) can also be applied in the same manner to any other axis. Such words are to be interpreted as encompassing devices oriented as shown in the figures as well as devices in other orientations in use or operation.

[0057] As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0058] In the following description, examples will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like elements.

[0059] Figure 1 An example of an image processing method is shown.

[0060] Referring to Figure 1 The light environment includes alternating current (AC) light 111. InFigure 1 In particular, based on capturing an image including a plurality of frames over time, the color of the light is accurately estimated based on a change in intensity according to a rapid change in pixel value of the light. By estimating the color of the light, the image quality of the color of the captured image 131 is improved.

[0061] The AC light 111 represents a light beam whose intensity periodically changes over time. In one example, the AC light 111 is a sinusoidal light beam having a frequency of 60 Hertz (Hz). The AC light 111 is generated by an AC light source. In one example, the light environment includes an environment including only the AC light, and a light environment in which direct current (DC) light and the AC light are mixed.

[0062] The image processing method includes a capturing operation 110, an image analysis operation 120, and an image processing operation 130.

[0063] In the capturing operation 110, an image including a plurality of frames is generated by capturing an object 112 over time in a light environment including the AC light 111. In one example, the camera 113 captures the object 112 at a capturing speed (frames per second (fps)) greater than or equal to the frequency (Hz) of the AC light. For example, when the AC light has a frequency of 60 Hz, the camera 113 captures the object 112 at a speed of 60 fps or more.

[0064] In the image analysis operation 120, an image including a plurality of frames 121, 122, and 123 captured over time (t, t+1, …) is analyzed in the time domain. The image including the plurality of frames 121, 122, and 123 is generated in a light environment including the AC light 111, and thus has image information that changes over time. For example, even a pixel corresponding to the same position in each frame 121, 122, and 123 has a different pixel value over time.

[0065] The pixel value includes intensity information of a pixel, and is also referred to as pixel intensity. For example, the pixel value is a value between “0” and “255”. The greater the value indicates the brighter the pixel. In addition, the pixel value is represented by a plurality of sub-pixel values. In one example, a pixel of a color image is represented as (red component pixel value, green component pixel value, blue component pixel value). In another example, a pixel of a color image is represented in the form of a 3 x 1 matrix. In the case of a color image, various colors are created by combining a red component, a green component, and a blue component. For example, when the pixel value is represented by a value between “0” and “255”, 16777216 colors corresponding to 256 3 colors are created.

[0066] In the image analysis operation 120, values of a plurality of pixels included in the plurality of frames are obtained, and information of the AC light is estimated using variations of the plurality of pixel values. Specifically, in the image analysis operation 120, color information of the AC light is estimated. In one example, the color information of the AC light represents a ratio of red, green, and blue colors of the color of the AC light.

[0067] In the image processing operation 130, the existing image 131 is processed to generate a corrected image 132 based on the information of the AC light. For example, the image 132 in which the color of the existing image 131 is corrected is generated based on the color information of the AC light. The color-corrected image 132 is an image in which the color corresponding to the AC light is removed from the existing image 131.

[0068] Figure 2A and Figure 2B An example of reflected light is shown. Referring to Figure 2A , according to a dichromatic model, the incident light beam 210 is split into two reflection components 220 and 230. In one example, the reflected light from the object includes a specular reflection component 220 and a diffuse reflection component 230. The specular reflection component 220 is a reflection component that is reflected on the surface of the object. The diffuse reflection component 230 is a reflection component that occurs when the light beam penetrates the object and scatters.

[0069] The reflected light from the object includes a combination of the specular reflection component 220 and the diffuse reflection component 230. The reflected light from the object is represented by the pixel value I c of the image. The pixel value I c of the image is represented by Equation 1.

[0070] [Equation 1]

[0071] I c = m d Λ c + m s Γ c

[0072] In Equation 1, Λ c represents a diffuse chromaticity, Γ c represents a specular chromaticity, m d represents a diffuse parameter, and m s represents a specular parameter.

[0073] According to equation 1, the pixel value of the image comprises an illumination component and a diffuse component. The illumination component corresponds to the specular component 220. The specular chroma of the specular component 220 comprises information about the color of the light, and the specular parameter of the specular component 220 comprises information about the intensity of the light. The diffuse component corresponds to the diffuse component 230. The diffuse chroma of the diffuse component 230 comprises information about the color of the object, and the diffuse parameter of the diffuse component 230 comprises information about the brightness of the object. The specular chroma is also referred to as a light vector, and the diffuse chroma is also referred to as an object vector.

[0074] With reference to Figure 2B In one example, the diffuse chroma and the specular chroma are represented in the form of vectors. For example, the diffuse chroma and the specular chroma are represented by 1 -dimensional vectors (e.g., in the form of (red component (R), green component (G), blue component (B)) or (R, G, B)). Thus, the pixel value of the image determined by the combination of the illumination component 240 and the diffuse component 250 is also represented in the form of (red component pixel value, green component pixel value, blue component pixel value) or (R, G, B). The matrix.

[0075] For example, equation 1 is represented in the form of a matrix as given in equation 2.

[0076] [Equation 2]

[0077]

[0078] In equation 2, I R represents the red component of the pixel value I c , I G represents the green component of the pixel value I c , and I B represents the blue component of the pixel value I c . Γ R represents the red component of the specular chroma Γ c , Γ G represents the green component of the specular chroma Γ c , and Γ B represents the blue component of the specular chroma Γ c . Λ R represents the red component of the diffuse chroma Λ c , Λ G represents the green component of the diffuse chroma Λ c , and Λ B represents the blue component of the diffuse chroma Λ c .

[0079] ​In one example, the unit vector of the illumination component 240 is the specular chromaticity, and the magnitude of the illumination component 240 is the specular parameter. Furthermore, the unit vector of the diffuse component 250 is the diffuse chromaticity, and the magnitude of the diffuse component 250 is the diffuse parameter.

[0080] Furthermore, the values of the pixels included in the image exist on a space 260 indicating the illumination component 240 and the diffuse component 250, the image comprising a plurality of frames taken over time in a light environment comprising AC light. The space 260 is referred to as the visual space or the dichromatic plane. Reference will be made below to Figure 3 The visual space is further described.

[0081] Some methods estimate light by mainly using pixels at different locations in the image to estimate a plane or a line. In this example, to accurately estimate the plane or the line, pixels at different locations in the image and having the same diffuse chromaticity and the same specular chromaticity need to be extracted. To this end, light is estimated by detecting the specular regions and using the pixels in the detected regions to estimate the plane or the line. However, when the specular regions in the image are insufficient, the light estimation performance is greatly reduced.

[0082] According to one example, instead of using pixels included in the specular regions, light is estimated by selecting AC pixels by modeling the variation in intensity of the AC light of the pixels to minimize the effect of noise on the image, so the light is more accurately estimated.

[0083] Figure 3 An example of the visual space is shown.

[0084] Reference is made to Figure 3 The values of the pixels included in the plurality of frames exist on the visual space 330. The visual space 330 comprises a space in which the values of the pixels included in the plurality of frames exist.

[0085] In one example, the values of the pixels included in the image are represented in the time domain as given by Equation 3, the image comprising a plurality of frames taken over time in a light environment comprising AC light.

[0086] [Equation 3]

[0087] I(t) = m d (t) Λ c + m s (t) Γ c

[0088] Since the specular chrominance 320 includes information about the color of light and the diffuse chrominance 310 includes information about the color of the object, the specular chrominance 320 and the diffuse chrominance 310 do not change over time. Also, since the specular parameter and the diffuse parameter include information about the intensity of the reflected light with respect to the AC light, the specular parameter and the diffuse parameter change over time. In detail, the specular parameter and the diffuse parameter are represented by Equation 4.

[0089] [Equation 4]

[0090]

[0091] B c = ∫ Ω S d (λ, x)E(λ)q c (λ)dλ

[0092] G c = ∫ Ω E(λ)q c (λ)dλ

[0093] In Equation 4, E(λ) denotes a spectral energy distribution. i, B c , S d , G c , λ, x, and q c (λ) are parameters that define weights of a dichromatic model and are used for an existing dichromatic model. Specifically, is a geometric structure parameter of the specular reflection, and i is an index indicating a color such as red, green, and blue. B c and G c denote parameters that show the intensity of the incident light that shows diffuse reflection and specular reflection, respectively. λ denotes a wavelength. S d and q c (λ) denote a spectral diffuse reflectance and a sensor sensitivity of a sensor for each wavelength, respectively. x denotes an image coordinate.

[0094] Since the specular chrominance 320 and the diffuse chrominance 310 do not change over time, the values of the pixels included in the plurality of frames exist on the visual space 330. For example, I(t) is the same as the vector 331, I(t+1) is the same as the vector 332, and I(t+2) is the same as the vector 333. As described above, if the values of the pixels included in two frames are provided for the pixels existing at one location, the visual space 330 is determined.

[0095] Figure 4A and Figure 4B show an example of estimating information of the AC light included in an image based on the visual space.

[0096] Referring toFigure 4A A single visual space 410, 420 is determined for each pixel included in the image. The plurality of pixels included in the image have different diffuse chrominances including information about the color of the object according to the position of the pixel, but have the same specular chrominance including information about the color of the light. This is because the color of the light does not change regardless of the position of the pixel. Therefore, the intersection line 430 of the visual space 410 of the first pixel and the visual space 420 of the second pixel indicates the specular chrominance including information about the color of the light. In one example, the temporal space of N pixels has a single intersection line, and the intersection line indicates the specular chrominance.

[0097] In the example shown in Figure 4B , noise such as color noise causes multiple intersection lines. In one example, the temporal space of N pixels has up to intersection lines. Therefore, in order to more accurately estimate the specular chrominance including information about the color of the light, additional constraints can be required. The additional constraints will be further described below with reference to Figures 5 to 10 .

[0098] Figure 5 An example of an image processing method is shown. Although Figure 5 the operations in can be performed in the order and manner as shown, the order of some operations can be changed, or some operations can be omitted, without departing from the spirit and scope of the described illustrative examples. Figure 5 Many of the operations shown in Figure 5 can be performed in parallel or concurrently. The blocks of the image processing method of Figure 5 are performed by an image processing device. In one example, the image processing device is implemented by a computer and a device based on dedicated hardware that performs a specific function, such as a processor, or by a combination of dedicated hardware and computer instructions. In addition to the description of Figures 1 to 4B below, Figure 5 the description of also applies to

[0099] and is incorporated herein by reference. Therefore, the above description is not repeated here.

[0100] In operation 510, the image processing device receives an image including a plurality of frames taken over time in a light environment including AC light. Figure 4A Figure 4B In operation 520, the image processing device extracts AC pixels corresponding to the AC light from a plurality of pixels included in the image. As described above with reference to Figures 6A to 6DFurther examples of extracting AC pixels are described.

[0101] In operation 530, the image processing device estimates a visual space of the AC pixels based on values of the AC pixels included in the plurality of frames. For example, the values of the AC pixels can include red component values, green component values, and blue component values of the AC pixels. For each AC pixel, a single visual space corresponding to each AC pixel is estimated. Examples of estimating a visual space are described below with reference to FIGS. 6A and 6B. Figure 7 Further examples of estimating a visual space are described.

[0102] In operation 540, the image processing device estimates information of the AC light included in the image based on the visual space. The information of the AC light includes color information of the AC light, which indicates a ratio of red, green, and blue of a color of the AC light. Examples of estimating information of the AC light are described below with reference to FIGS. 7A and 7B. Figures 8A to 8D and Figure 9 Further examples of estimating information of the AC light are described.

[0103] In operation 550, the image processing device processes the image based on the information of the AC light. The image processing device corrects a color of the image based on the color information of the AC light. For example, by removing a color corresponding to the AC light in the existing image, an image quality performance associated with the color of the existing image is improved.

[0104] Figures 6A to 6D Examples of extracting AC pixels are shown.

[0105] When selecting the AC pixels by analyzing a change in intensity of light, a pixel showing a smallest signal distortion due to noise is selected to more accurately estimate a visual space.

[0106] Figure 6A A pixel value I1(t) 601 of the first pixel at time t and a pixel value I1(t+1) 602 of the first pixel at time t+1 are shown. Figure 6B A pixel value I2(t) 611 of the second pixel at time t and a pixel value I2(t+1) 612 of the second pixel at time t+1 are shown.

[0107] When the pixel value as shown in FIG. 6A relatively slightly changes over time, an effect of noise is greater. A time space generated based on the pixel shows a relatively low accuracy. When the pixel value as shown in FIG. 6B relatively greatly changes over time, an effect of AC change of light is greater. Figure 6A Figure 6B

[0108] In order to obtain more accurate AC light information, a pixel showing a small signal distortion due to noise (e.g., a pixel showing a signal distortion due to noise less than a threshold value) is extracted as an AC pixel, and a visual space and AC light information are estimated based on the extracted AC pixel.​​

[0109] Reference Figure 6C and Figure 6D , based on the intensity of AC light x i The change of pixel value y i For example, when the change in the intensity of the AC light is a sinusoidal curve, the change in the pixel value is modeled using a sinusoidal curve, as shown in Equation 5.

[0110] [Equation 5]

[0111] y i =f(x i )=a sin(Fs·x i +b)+c+n

[0112] In Equation 5, F represents the AC frequency, s represents the captured frame rate, b represents the phase, c represents the offset, and n represents the noise. Since the AC frequency (60 Hz) and the captured frame rate are known, the frequency is predictable, so the parameters to be estimated are the amplitude, phase, and offset of the sinusoid.

[0113] In a similar manner, the variation of pixel values ​​is also modeled as shown in Equation 6.

[0114] [Equation 6]

[0115] y i =f(x i )=|asin(bx i +c)|+d+n

[0116] When using the Gauss-Newton scheme, the parameters are estimated accurately in a short time through iteration. Figure 6C The change of pixel value is modeled based on Equation 5 as shown in Figure 6D The variation of pixel values ​​is modeled based on Equation 6 as shown in FIG.

[0117] When the parameter estimation is completed, if the difference between the modeled sinusoidal curve and the actual change in pixel value is less than a threshold, a pixel is selected from the plurality of pixels as an AC pixel. In one example, for a plurality of pixels (e.g., N pixels), the pixel value of the modeled sinusoidal curve is The difference between the values ​​y of the pixels in each frame is calculated. Pixels with a sum of the differences calculated for each frame less than a threshold (e.g., as indicated by an AC fitting error, but not limited thereto) are extracted as AC pixels. The AC fitting error Err is represented by Equation 7.

[0118] [Equation 7]

[0119]

[0120]

[0121]

[0122] In Equation 7, the Gauss Newton function represents a function for Gauss Newton iteration, represents an estimated value of the parameter a in Equation 5. In order to reduce the algorithm complexity, a set step is used instead of performing a sinusoidal curve modeling for all pixels. For example, only 60 AC pixels are extracted from a plurality of pixels included in an image.

[0123] Figure 7 An example of estimating a visual space is shown.

[0124] Referring to Figure 7 In the visual space estimation operation, the light is estimated using the time samples of the AC pixels obtained in the previous operation. The visual space of the AC pixels is estimated based on the values of the AC pixels included in a plurality of frames. The visual space indicating the illumination component and the diffuse component of the AC pixels included in a plurality of frames is estimated for the AC pixels. The visual space is a dichromatic plane of the AC pixels.

[0125] In one example, the parameters of the plane are three directional parameters of a normal vector of a three-dimensional plane, and are obtained using a least square method. The plane that minimizes the average perpendicular distance from the values of the AC pixels included in a plurality of frames is estimated based on the perpendicular distance between the plane and the values of the AC pixels included in a plurality of frames.

[0126] For example, using a least square method, the normal vector 720 of the plane 710 minimizes the average perpendicular distance from the perpendicular distance from the pixel value 701 at a first time point, the pixel value 702 at a second time point, the pixel value 703 at a third time point, the pixel value 704 at a fourth time point, the pixel value 705 at a fifth time point, and the pixel value 706 at a sixth time point.

[0127] Figures 8A to 8D and Figure 9 An example of estimating information of AC light is shown.

[0128] Referring to Figure 8A Three AC pixels 811, 812, and 813 are extracted from the image 810. Figure 8B A visual space corresponding to the first pixel 811 is shown, Figure 8C A visual space corresponding to the second pixel 812 is shown, Figure 8D A visual space corresponding to the third pixel 813 is shown.

[0129] An intersection line of the visual space of the first pixel 811, the visual space of the second pixel 812, and the visual space of the third pixel 813 indicates a specular chrominance including information about a color of light. The specular chrominance includes color information of the AC light, which indicates a ratio of red, green, and blue of the color of the AC light.

[0130] A light vector candidate corresponding to the AC light is determined based on the visual space. The light vector is determined among the light vector candidates based on prior information of the AC light. Information of the AC light is estimated based on the light vector.

[0131] In one example, since all the visual spaces share the same light vector, the intersection line of the visual spaces should correspond to a single light vector. However, in reality, due to camera characteristics and noise in the video, there are many intersection lines. In one example, to estimate the best light vector among the intersection lines, a maximum a posteriori probability (MAP) estimation is used.

[0132] An intersection line of the visual space is estimated. In one example, the intersection line that minimizes a cost function is determined among the intersection lines based on the MAP estimation. Information of the AC light is estimated based on the determined intersection line.

[0133] The MAP estimation is a method for detecting the best parameter by combining provided observations and "prior knowledge (prior probability)". The MAP estimation is represented by Equation 8.

[0134] [Equation 8]

[0135]

[0136] Referring to Equation 8, the MAP estimation includes two terms, in detail, a physical term indicating a physical characteristic and a statistical term indicating a statistical characteristic.

[0137] The physical term indicates a physical relationship between the AC light and the visual space. In one example, a vector of light should be perpendicular to a normal vector of all the visual spaces. Thus, an angle using an inner product of the light vector and the normal vector of the visual space is calculated and used. In one example, when the angle is close to 90 degrees, the intersection line is determined to have more accurate light. When a cosine value obtained using a cosine function is close to "0" (i.e., when the value is small), the intersection line is estimated to have more accurate light. The foregoing is represented by Equation 9.

[0138] [Equation 9]

[0139]

[0140] The statistical term uses Figure 9of the Planck locus as prior information. The Planck locus indicates the change of color of a black body with respect to temperature. These colors include the colors of most light sources that can exist in nature. In one example, on the ultraviolet (UV) domain, the intersection line at a short perpendicular distance from the Planck locus is estimated to have the accurate light.

[0141] The intersection line that minimizes the sum of the value of the physical term and the value of the statistical term is detected, and the intersection line is estimated as the light vector.

[0142] In Equation 8 and Equation 9, P k represents a plane (an index), n represents the total number of planes (k = 1, 2, …, n). λ represents a balance weight between the likelihood (arccos part) and the prior term (d plankcian ). λ is determined through experiments. Γ i represents the i-th light vector candidate. d plankcian is the Euclidean projection distance on the CIE 1960 ultraviolet (UV) chromaticity domain, and is the distance between Γ i and the Planck locus.

[0143] Figure 10 An example of an image processing method is shown.

[0144] Referring to Figure 10 , the image processing method includes an operation 1010 of extracting AC pixels corresponding to AC light from a plurality of pixels included in an image, an operation 1020 of estimating a visual space of the AC pixels based on values of the AC pixels included in a plurality of frames, and an operation 1030 of estimating information of the AC light included in the image based on the visual space and processing the image based on the information of the AC light.

[0145] In operation 1010, AC pixels are extracted from a plurality of pixels included in an image 1011, the image 1011 including a plurality of frames taken over time in a light environment including AC light. In one example, pixels having a sum of differences between pixel values of a modeled sinusoidal curve 1013 and pixel values of the plurality of frames that is less than a threshold value are extracted as AC pixels.

[0146] In operation 1020, a visual space is estimated based on pixel values of extracted AC pixels 1021 over time. In one example, intersection lines of the visual space can be indicated by a set 1023.

[0147] In operation 1030, a set 1031 of intersection lines of the visual space of the AC pixels is extracted. A single intersection line is extracted from the intersection lines based on a MAP estimate. The extracted intersection line includes color information of the AC light, and correction 1030 of a color of the image is performed based on the color information of the AC light.

[0148] Figure 11 An example of an image processing device is illustrated.

[0149] Referring to Figure 11 The image processing device 1100 includes a processor 1110. The image processing device 1100 further includes an input / output interface 1120, a memory 1130, a communication interface 1150, and one or more sensors 1170. The processor 1110, the memory 1130, the communication interface 1150, and the one or more sensors 1170 communicate with each other through a communication bus 1105.

[0150] The processor 1110 is, for example, a device configured to execute instructions or programs or control the image processing device 1100. The processor 1110 includes, for example, a central processing unit (CPU), a processor core, a multi-core processor, a reconfigurable processor, a multi-processor, an application-specific integrated circuit (ASIC), and a field-programmable gate array (FPGA), a graphics processing unit (GPU), or any other type of multi-processor configuration or single-processor configuration. More details about the processor 1110 are provided below.

[0151] The processor 1110 receives an image including a plurality of frames photographed over time in a light environment including AC light, extracts AC pixels corresponding to the AC light from a plurality of pixels included in the image, estimates a visual space of the AC pixels based on values of the AC pixels included in the plurality of frames, estimates information of the AC light included in the image based on the visual space, and processes the image based on the information of the AC light.

[0152] The memory 1130 includes a registration database containing images and pixel values. The memory 1130 is a volatile memory or a non-volatile memory. The memory 1130 includes a mass storage medium such as a hard disk for storing various data. More details about the memory 1130 are provided below.

[0153] The one or more sensors 1170 include, for example, a camera configured to photograph an image at a photographing speed (fps) greater than or equal to a frequency (Hz) of the AC light. The one or more sensors 1170 collect various image information.

[0154] In one example, the input / output interface 1120 can be a display that receives input from a user or provides output of the image processing device 1100. In one example, the input / output interface 930 can function as an input device and receive input from a user through a conventional input method (e.g., a keyboard and a mouse) and a new input method (e.g., touch input, voice input, and image input). Accordingly, the input / output interface 1120 can include, for example, a keyboard, a mouse, a touch screen, a microphone, and other devices that can detect input from a user and transmit the detected input to the image processing device 1100.

[0155] In one example, the input / output interface 1120 can serve as an output device and provide an output of a processed image of the AC light-based information to a user. The input / output interface 1120 can include, for example, a display, a touch screen, and other devices that can provide an output to a user. However, the input / output interface 1120 is not limited to the above-described example, and any other display operatively connected to the image processing apparatus 1100, such as, for example, a computer monitor and an eyeglass display (EGD), can be used without departing from the spirit and scope of the described illustrative examples. In one example, the input / output interface 1120 is a physical structure including one or more hardware components that provide the ability to present a user interface, present a display, and / or receive user input.

[0156] In one example, the image processing apparatus 1100 is connected to an external device (e.g., a microphone, a keyboard, or an image sensor) via the communication interface 1150 and exchanges data.

[0157] In one example, the processor 1110 estimates a visual space indicating illumination components and diffuse components of AC pixels included in a plurality of frames. In one example, the processor 1110 estimates a dichromatic plane of the AC pixels based on a dichromatic model. In one example, the processor 1110 estimates the visual space based on a linear combination of red component values, green component values, and blue component values of the AC pixels in the plurality of frames. In one example, the processor 1110 estimates a visual space that would be produced by a linear combination of red component values, green component values, and blue component values of the AC pixels included in the plurality of frames. In one example, the processor 1110 estimates parameters of the visual space that minimize a perpendicular distance between a plane and values of the AC pixels included in the plurality of frames. In one example, the processor 1110 estimates color information of the AC light. In one example, the processor 1110 corrects colors of an image based on the color information of the AC light. In one example, the processor 1110 extracts AC pixels based on changes in values of a plurality of pixels included in the plurality of frames. In one example, the processor 1110 models values of a plurality of pixels included in the plurality of frames as sinusoidal curves, calculates differences between pixel values of the modeled sinusoidal curves and values of the pixels in the plurality of frames for the plurality of pixels, and extracts, from the plurality of pixels, pixels having a sum of the calculated differences that is less than a threshold value as the AC pixels. In one example, the processor 1110 models values of the pixels included in the plurality of frames as sinusoidal curves based on a Gauss-Newton method. In one example, the processor 1110 determines light vector candidates corresponding to the AC light based on the visual space, determines a light vector among the light vector candidates based on prior information of the AC light, and estimates information of the AC light based on the light vector. In one example, the processor 1110 estimates intersection lines of the visual space, determines an intersection line that minimizes a cost function among the intersection lines based on a MAP estimation, and estimates information of the AC light based on the determined intersection line. In one example, the processor 1110 calculates a probability that the intersection line is perpendicular to the visual space, and determines an intersection line that minimizes a cost function among the intersection lines based on prior information of the AC light and the probability.

[0158] Further, the processor 1110 performs at least one method described with reference to or an algorithm corresponding to the at least one method. The processor 1110 executes programs and controls the image processing device 1100. Program codes executed by the processor 1110 are stored in the memory 1130. Figures 2A to 10 Further, the processor 1110 performs at least one method described with reference to or an algorithm corresponding to the at least one method. The processor 1110 executes programs and controls the image processing device 1100. Program codes executed by the processor 1110 are stored in the memory 1130.

[0159] The image processing device 1100 is mounted in various devices and / or systems such as, for example, a smart phone, a mobile phone, a wearable device such as a ring, a watch, a pair of glasses, a glasses-type device, a bracelet, an anklet, a belt, a necklace, an earring, a headband, a helmet, a device embedded in clothing, or an electronic glasses display (EGD), a computing device (e.g., a server, a laptop computer, a notebook computer, a subnotebook computer, a netbook, a tablet personal computer (tablet), a phablet, a mobile internet device (MID), a personal digital assistant (PDA), an enterprise digital assistant (EDA), an ultra-mobile personal computer (UMPC), a portable laptop PC), an electronic product (e.g., a robot, a digital camera, a digital camcorder, a portable game console, an MP3 player, a portable / personal multimedia player (PMP), a handheld electronic book, a global positioning system (GPS) navigator, a personal navigation device, a portable navigation device (PND), a handheld game console, an electronic book, a television (TV), a high-definition television (HDTV), a smart TV, a smart home appliance, a smart home device, or a security device for door control, a smart speaker), various Internet of Things (IoT) devices, or a kiosk), and the image processing device 1100 can be executed by an application, middleware, or an operating system mounted on a user device, or by a program of a server that interoperates with a corresponding application.

[0160] Herein for Figures 1 to 11The described image processing device 1100, as well as other devices, units, modules, apparatuses, and other components described herein, are implemented by way of hardware components. Examples of hardware components include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic module units, adders, subtracters, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components of the described image processing device 1100 are implemented by way of computing hardware, for example, by one or more processors or computers. A processor or computer can be implemented by way of one or more processing elements, such as logical gates arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond to and implement instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes or is connected to one or more memories that store instructions or software for execution by the processor or computer. The hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or additional processors and additional controllers. The one or more processors, or a processor and a controller, can implement a single hardware component or two or more hardware components. The hardware components can have any one or more of various processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessors, single-instruction multiple-data (SIMD) multiprocessors, multiple-instruction single-data (MISD) multiprocessors, and multiple-instruction multiple-data (MIMD) multiprocessors.

[0161] Figures 1 to 11The methods illustrated in the middle perform the operations described in this application by computing hardware (e.g., by one or more processors or computers) implemented to execute instructions or software as described above to perform the operations described in this application as performed by the methods. For example, a single operation or two or more operations can be performed by a single processor or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or further processors and further controllers. The one or more processors, or a processor and a controller, can perform a single operation or two or more operations.

[0162] The instructions or software for controlling the processor or computer to implement the hardware components and perform the methods as described above are written as a computer program, a code segment, instructions, or any combination thereof, to individually or collectively instruct or configure the processor or computer to operate as a machine or a special-purpose computer to perform the operations performed by the hardware components and the methods as described above. In one example, the instructions or software include at least one of a small program, a dynamic link library (DLL), middleware, firmware, a device driver, and an application program that stores an image processing method. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. A programmer of ordinary skill in the art can easily write the instructions or software based on the block diagrams and flowcharts illustrated in the drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and the methods as described above.

[0163] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and perform the methods as described above, as well as any associated data, data files, and data structures, can be recorded, stored, or fixed in one or more non-transitory computer-readable storage media, or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random- access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid-state drive (SSD), card-type memory such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card, magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store instructions or software, as well as any associated data, data files, and data structures, in a non-transitory manner and provide the instructions or software, as well as any associated data, data files, and data structures, to a processor or computer so that the processor or computer can execute the instructions. In one example, the instructions or software, as well as any associated data, data files, and data structures, are distributed over a network-connected computer system so that one or more processors or computers store, access, and execute the instructions, software, and any associated data, data files, and data structures in a distributed manner.

[0164] While the disclosure includes certain examples, it will be clear to those skilled in the art that various changes can be made without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each example should be considered as available for combination with other features or aspects in other examples. If the described technology is performed in a different order, and / or if the described architectures, systems, devices, or circuits are combined in a different manner, and / or by other components or their equivalents, suitable results can be achieved. Accordingly, the scope of the disclosure is not intended to be limited by the specific embodiments described herein, but only by the claims and their equivalents, and the entire disclosure is intended to be included within the scope of the claims and their equivalents.

Claims

1. An image processing method, comprising: receiving an image comprising a plurality of frames captured over time in a light environment including AC light; extracting AC pixels corresponding to the AC light from a plurality of pixels in the image; estimating a visual space of an AC pixel in the image based on values ​​of the AC pixel in the plurality of frames, the AC pixel in the image being determined based on a combination of an illumination component and a diffuse component of the AC pixel; estimating information of AC light included in the image based on a visual space, wherein the step of estimating the information of the AC light comprises: estimating an intersection line of the visual space, wherein the intersection line indicates specular chromaticity including information related to color of light; and estimating the information of the AC light based on the intersection line; and The image is processed based on the information of the AC light.

2. The image processing method according to claim 1, wherein: The step of estimating the visual space includes estimating the visual space indicative of illumination components and diffuse components of AC pixels in the plurality of frames.

3. The image processing method according to claim 1, wherein: The step of estimating the visual space includes estimating a two-color plane of the AC pixels based on a two-color model.

4. The image processing method according to claim 1, wherein: The value of the AC pixel includes the red component value, the green component value, and the blue component value of the AC pixel, and The step of estimating the visual space includes estimating the visual space based on a linear combination of red component values, green component values, and blue component values ​​of AC pixels in the plurality of frames.

5. The image processing method according to claim 1, wherein: The step of estimating the visual space includes extracting parameters of the visual space that minimize a vertical distance between a plane and the values ​​of the AC pixels in the plurality of frames. The image processing method according to claim 5 , wherein: The step of extracting the parameters includes: extracting the parameters based on a least squares method.

7. The image processing method according to claim 1, wherein: The step of estimating the information of the AC light includes estimating color information of the AC light.

8. The image processing method according to claim 1, wherein: The information of the AC light includes ratios of a red component, a green component, and a blue component of the AC light.

9. The image processing method according to claim 1, wherein: The step of processing the image includes correcting the color of the image based on the color information of the AC light.

10. The image processing method according to claim 1, wherein: The step of extracting the AC pixel includes extracting, from the plurality of pixels, AC pixels exhibiting signal distortion due to noise that is less than a threshold value.

11. The image processing method according to claim 1, wherein: The step of extracting the AC pixels includes extracting the AC pixels based on changes in values ​​of the plurality of pixels in the plurality of frames.

12. The image processing method according to claim 1, wherein: The steps to extract AC pixels include: modeling values ​​of the plurality of pixels in the plurality of frames as a sinusoidal curve; calculating respective differences between pixel values ​​of the modeled sinusoid and values ​​of the plurality of pixels in the plurality of frames; and Pixels having a calculated sum of differences smaller than a threshold value are extracted from the plurality of pixels as AC pixels.

13. The image processing method according to claim 12, wherein: The modeling step includes modeling the values ​​of the plurality of pixels in the plurality of frames as a sine curve based on a Gauss-Newton method.

14. The image processing method according to claim 1, wherein: The steps for estimating the AC light information include: determining a light vector candidate corresponding to the AC light based on the visual space; determining a light vector from among light vector candidates based on prior information of the AC light; and Estimate AC light information based on light vectors.

15. The image processing method according to claim 14, wherein: The prior information is obtained based on the Planck locus information.

16. The image processing method according to claim 1, wherein: The steps for estimating the AC light information include: Estimating multiple intersecting lines in visual space; determining an intersection line that minimizes a cost function from among the estimated intersection lines based on the maximum a posteriori probability estimate; and Information of the AC light is estimated based on the determined intersection line.

17. The image processing method according to claim 16, wherein: The steps to determine include: Compute the probability that the intersecting line is perpendicular to visual space; and Based on the prior information of the AC light and the probability, an intersection line that minimizes the cost function is determined from among the estimated intersection lines. 18 . A non-transitory computer-readable storage medium storing instructions, wherein when the instructions are executed by a processor, the processor is caused to perform the image processing method according to claim 1 .

19. An image processing device comprising: A processor configured to: receive an image comprising a plurality of frames captured over time in a light environment including AC light; extracting AC pixels corresponding to AC light from a plurality of pixels in an image; estimating a visual space of the AC pixels in the image based on values ​​of the AC pixels included in the plurality of frames, the AC pixels in the image being determined based on a combination of an illumination component and a diffuse component of the AC pixels; estimating information of the AC light included in the image based on the visual space; and processing the image based on the information of the AC light, The processor is configured to: estimate an intersection line of a visual space, wherein the intersection line indicates specular chromaticity including information related to color of light; and estimate information of the AC light based on the intersection line.

20. The image processing apparatus according to claim 19, wherein The processor is further configured to estimate a visual space indicative of illumination components and diffuse components of AC pixels in the plurality of frames.

21. The image processing apparatus according to claim 19, wherein The processor is further configured to estimate a dichromatic plane of the AC pixel based on a dichromatic model.

22. The image processing apparatus according to claim 19, wherein The value of the AC pixel includes the red component value, the green component value, and the blue component value of the AC pixel, and The processor is further configured to estimate a visual space based on a linear combination of red component values, green component values, and blue component values ​​of the AC pixels in the plurality of frames.

23. The image processing apparatus according to claim 19, wherein The processor is further configured to extract a parameter of the visual space that minimizes a vertical distance between the plane and the values ​​of the AC pixels in the plurality of frames.

24. The image processing apparatus according to claim 19, wherein The processor is further configured to estimate color information of the AC light.

25. The image processing apparatus according to claim 19, wherein The processor is further configured to correct the color of the image based on the color information of the AC light.

26. The image processing apparatus according to claim 19, wherein The processor is further configured to extract pixels showing signal distortion due to noise less than a threshold value from the plurality of pixels as AC pixels.

27. The image processing apparatus according to claim 19, wherein The processor is further configured to extract AC pixels based on changes in values ​​of the plurality of pixels included in the plurality of frames.

28. The image processing apparatus according to claim 19, wherein The processor is further configured to: model the values ​​of the plurality of pixels in the plurality of frames as a sinusoidal curve; calculate respective differences between pixel values ​​of the modeled sinusoidal curve and the values ​​of the plurality of pixels in the plurality of frames; A pixel having a calculated difference sum smaller than a threshold value is extracted from the plurality of pixels as an AC pixel.

29. The image processing apparatus according to claim 28, wherein The processor is further configured to model the values ​​of the plurality of pixels included in the plurality of frames as a sinusoidal curve based on a Gauss-Newton method.

30. The image processing apparatus according to claim 19, wherein The processor is further configured to: determine light vector candidates corresponding to the AC light based on the visual space; determine a light vector from among the light vector candidates based on prior information of the AC light; and estimate information of the AC light based on the light vector.

31. The image processing apparatus according to claim 19, wherein The processor is further configured to: estimate intersection lines of the visual space; determine an intersection line that minimizes a cost function from among the estimated intersection lines based on maximum a posteriori probability estimation; and estimate information of the AC light based on the determined intersection line.

32. The image processing apparatus according to claim 31, wherein The processor is further configured to: calculate a probability that the intersection line is perpendicular to the visual space; and determine an intersection line that minimizes the cost function from among the estimated intersection lines based on the prior information of the AC light and the probability.

33. An image processing device comprising: a sensor configured to capture an image comprising a plurality of frames captured over time in a light environment including AC light; The processor is configured to: extracting AC pixels corresponding to the AC light from a plurality of pixels in the image; estimating a visual space of AC pixels in the image based on values ​​of the AC pixels included in the plurality of frames, the AC pixels in the image being determined based on a combination of an illumination component and a diffuse component of the AC pixels; determining information of AC light included in the image based on a visual space, wherein the step of estimating the information of the AC light comprises: estimating an intersection line of the visual space, wherein the intersection line indicates specular chromaticity including information related to color of light; and estimating the information of the AC light based on the intersection line; and Processing images based on the information of the communication light; and The output unit is configured to display the processed image.

34. The image processing apparatus according to claim 33, further comprising: A non-transitory computer-readable storage medium stores AC pixels, images, and instructions. In response to a processor executing the instructions, AC pixels are extracted, information about AC light is determined, and images are processed.

Images