Device and method for image sensing

Abstract

A device for image sensing according to an example embodiment of the present disclosure may comprise: an RGB-IR image sensor comprising a pixel array with an RGB-IR pattern; an IR LED configured to be turned on and off at a preset interval; and an RGB-IR image processing unit which is configured to generate an IR image on the basis of a first RBG-IR image captured via the RGB-IR image sensor while the IR LED is turned on, and generate an RGB image on the basis of a second RGB-IR image captured via the RGB-IR image sensor while the IR LED is turned off.

Description

Image sensing device and method

[0001] The technical idea of ​​the present disclosure relates to image sensing, and more specifically, to an image sensing device and method.

[0002] Image sensing devices capture images by utilizing the light-responsive properties of semiconductors. With the recent development of the computer and communications industries, the demand for high-performance image sensing devices has increased in various fields such as smartphones, digital cameras, gaming devices, the Internet of Things (IoT), robots, security cameras, and medical micro-cameras. For example, as IR (Infrared) images are increasingly utilized in areas such as night surveillance, iris recognition, and facial recognition, the demand for RGB-IR image sensing devices is increasing. However, the R, G, and B pixels of an RGB-IR image sensing device can collect IR light as well as R, G, and B light, and thus color distortion can occur due to the IR component. Therefore, to accurately restore color using an RGB-IR image sensing device, it is necessary to calculate the IR component collected by the R, G, and B pixels and remove the IR component from the total collected light components, which are the visible light component and the IR component.

[0003] The technical problem of the present disclosure is to provide a method and device for generating an IR (Infrared) image and an RGB image using an LED that repeats on-off.

[0004] The technical problem of the present disclosure is to provide a method and device for effectively removing an IR signal entering an RGB pixel.

[0005] The technical problem of the present disclosure is to provide a method and device for generating an RGB image without color distortion even in an environment where IR exists.

[0006] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0007] An image sensing device according to an exemplary embodiment of the present disclosure may include an RGB-IR image sensor including a pixel array of an RGB-IR pattern, an IR LED configured to be turned on and off at a predetermined cycle, and an RGB-IR image processing unit configured to generate an IR image based on a first RGB-IR image captured through the RGB-IR image sensor when the IR LED is turned on, and to generate an RGB image based on a second RGB-IR image captured through the RGB-IR image sensor when the IR LED is turned off.

[0008] According to one embodiment, the image sensing device may further include an exposure value determination unit configured to set the RGB-IR image sensor to have a first exposure value when the IR LED is turned on, and to set the RGB-IR image sensor to have a second exposure value when the IR LED is turned off.

[0009] According to one embodiment, the exposure value determining unit may be configured to determine a first exposure value based on brightness information of an IR image and to determine a second exposure value based on brightness information of an RGB image.

[0010] According to one embodiment, the image sensing device may further include a memory configured to store an IR image and an RGB image, and a memory controller configured to store the IR image and the RGB image in the memory and to simultaneously output the stored IR image and the stored RGB image.

[0011] According to one embodiment, the memory controller may be configured to increase a frame rate of an output IR image by performing frame interpolation on an IR image sequence, and to increase a frame rate of an output RGB image by performing frame interpolation on an RGB image sequence.

[0012] According to one embodiment, the RGB-IR image processing unit may be configured to generate an IR image by performing a decolor operation on the first RGB-IR image.

[0013] According to one embodiment, the RGB-IR image processing unit may be configured to generate an IR image by performing denoising on the first RGB-IR image.

[0014] According to one embodiment, the RGB-IR image processing unit may be configured to generate a first RGB image including an IR component based on a second RGB-IR image, generate a second RGB image from which the IR component is removed based on the second RGB-IR image and the first RGB image, generate a third RGB image based on a brightness component of the first RGB image and a color component of the second RGB image, and generate an RGB image by converting the third RGB image into a Bayer pattern image.

[0015] According to one embodiment, the RGB-IR image sensor may be characterized as being a global shutter type image sensor.

[0016] According to one embodiment, the RGB-IR image processing unit may be configured to generate an IR image by performing denoising on the first RGB-IR image.

[0017] An image sensing method according to an exemplary embodiment of the present disclosure may include a step of turning an IR LED on and off at a predetermined cycle, a step of generating an IR image based on a first RGB-IR image captured through an RGB-IR image sensor when the IR LED is turned on, and a step of generating an RGB image based on a second RGB-IR image captured through the RGB-IR image sensor when the IR LED is turned off.

[0018] According to one embodiment, the image sensing method may further include the step of setting the RGB-IR image sensor to have a first exposure value when the IR LED is turned on and the step of setting the RGB-IR image sensor to have a second exposure value when the IR LED is turned off.

[0019] According to one embodiment, the step of setting the RGB-IR image sensor to have the first exposure value may include the step of determining the first exposure value based on brightness information of the IR image, and the step of setting the RGB-IR image sensor to have the second exposure value may include the step of determining the second exposure value based on brightness information of the RGB image.

[0020] According to one embodiment, an image sensing method may be characterized by including the steps of storing an IR image and an RGB image in a memory and simultaneously outputting the stored IR image and the stored RGB image.

[0021] According to one embodiment, the step of simultaneously outputting the stored IR image and the stored RGB image may include the step of increasing the frame rate of the output IR image by performing frame interpolation on the IR image sequence and the step of increasing the frame rate of the output RGB image by performing frame interpolation on the RGB image sequence.

[0022] According to one embodiment, the step of generating an IR image may include the step of generating the IR image by performing a decolor operation on the first RGB-IR image.

[0023] According to one embodiment, the step of generating an IR image may include the step of generating the IR image by performing a decolor operation on the first RGB-IR image.

[0024] According to one embodiment, the step of generating an RGB image may include the steps of generating a first RGB image including an IR component based on a second RGB-IR image, generating a second RGB image from which an IR component is removed based on the second RGB-IR image and the first RGB image, generating a third RGB image based on a brightness component of the first RGB image and a color component of the second RGB image, and generating an RGB image by converting the third RGB image into a Bayer pattern image.

[0025] According to one embodiment, the RGB-IR image sensor may be characterized as being a global shutter type image sensor.

[0026] According to one embodiment, the RGB-IR image sensor may be characterized as being a rolling shutter type image sensor.

[0027] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0028] According to an image sensing method according to an exemplary embodiment of the present disclosure, an IR image and an RGB image expressing color can be simultaneously output even when IR light is strongly irradiated through an IR (Infrared) LED.

[0029] According to the image sensing method according to the exemplary embodiment of the present disclosure, an IR image and an RGB image can be acquired simultaneously with a single RGB-IR sensor, so that a system can be implemented at low cost and low power.

[0030] According to the image sensing method according to the exemplary embodiment of the present disclosure, since there is no need to keep the IR LED on, the power consumption of the overall system can be reduced.

[0031] According to an image sensing method according to an exemplary embodiment of the present disclosure, an IR signal entering an RGB pixel can be effectively removed.

[0032] According to an image sensing method according to an exemplary embodiment of the present disclosure, an RGB image without color distortion can be generated even in an environment where IR is present.

[0033] According to an image sensing method according to an exemplary embodiment of the present disclosure, an ISP (Image Signal Processor) and an image processing system that process Bayer pattern images can be used without change by combining an RGB image and an IR image generated by an RGB-IR image sensing device into one.

[0034] According to an image sensing method according to an exemplary embodiment of the present disclosure, an RGB-IR image can be converted into a Bayer pattern image.

[0035] According to an image sensing method according to an exemplary embodiment of the present disclosure, an image including an IR component but without color distortion can be generated.

[0036] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0037] FIG. 1 is a block diagram illustrating an image sensing device according to an exemplary embodiment of the present disclosure.

[0038] FIG. 2 is a flowchart illustrating an image sensing method according to an exemplary embodiment of the present disclosure.

[0039] FIG. 3 is a drawing for explaining an RGB-IR pattern according to an exemplary embodiment of the present disclosure.

[0040] FIG. 4 is a block diagram of an image sensing device according to an exemplary embodiment of the present disclosure.

[0041] FIG. 5 is a drawing for explaining an image sensing method according to an exemplary embodiment of the present disclosure.

[0042] FIG. 6 is a drawing for explaining an image sensing method according to an exemplary embodiment of the present disclosure.

[0043] FIG. 7 is a diagram for explaining an image sensing method according to an exemplary embodiment of the present disclosure.

[0044] FIG. 8 is a drawing for explaining an image sensing method according to an exemplary embodiment of the present disclosure.

[0045] FIG. 9 is a drawing for explaining an image sensing method according to an exemplary embodiment of the present disclosure.

[0046] FIG. 10 is a block diagram illustrating an imaging sensing device according to an exemplary embodiment of the present disclosure.

[0047] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0048] In describing embodiments of the present disclosure, detailed descriptions of known configurations or functions will be omitted if they are deemed to obscure the gist of the present disclosure. Furthermore, portions unrelated to the description of the present disclosure in the drawings have been omitted, and similar portions have been designated with similar reference numerals.

[0049] In the present disclosure, when a component is said to be "connected," "coupled," or "connected" to another component, this may include not only a direct connection, but also an indirect connection in which another component exists in between. Furthermore, when a component is said to "include" or "have" another component, unless specifically stated otherwise, this does not exclude the other component, but rather implies that the other component may be included.

[0050] In this disclosure, terms such as first, second, etc. are used solely to distinguish one component from another, and do not limit the order or importance of components unless specifically stated otherwise. Accordingly, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0051] In this disclosure, distinct components are used only to clearly illustrate their respective characteristics and do not necessarily imply separation. That is, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed into multiple hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included within the scope of this disclosure.

[0052] In the present disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments comprising a subset of the components described in one embodiment are also within the scope of the present disclosure. Furthermore, embodiments that include other components in addition to the components described in various embodiments are also within the scope of the present disclosure.

[0053] In the present disclosure, expressions of positional relationships used in the present specification, such as top, bottom, left, right, etc., are described for convenience of explanation, and when the drawings illustrated in the present specification are viewed in reverse, the positional relationships described in the specification may be interpreted in the opposite way.

[0054] In the present disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0055] Hereinafter, exemplary embodiments of the present disclosure will be specifically described with reference to FIGS. 1 to 10.

[0056] FIG. 1 is a block diagram illustrating an image sensing device according to an exemplary embodiment of the present disclosure.

[0057] Referring to FIG. 1, an image sensing device (10) according to an exemplary embodiment of the present disclosure may include an RGB-IR image sensor (11), an IR LED (12), an RGB-IR image processing unit (13), an exposure value determination unit (14), a memory controller (15), and a memory (16).

[0058] The RGB-IR image sensor (11) can output an image by digitizing light collected on a photodiode, which is a silicon surface, through an ADC (Analog-to-Digital Converter). For example, the RGB-IR image sensor (11) can include a pixel array of an RGB-IR pattern, and can generate an RGB-IR image through the pixel array of the RGB-IR pattern.

[0059] The RGB-IR image sensor (11) may be a global shutter type or rolling shutter type image sensor, but is not limited thereto.

[0060] The R pixels, G pixels, and B pixels of the RGB-IR image sensor (11) may collect near-infrared (IR) light in addition to visible light, and the IR component may distort the RGB color components. In other words, when an RGB image is generated by receiving both RGB wavelengths and IR wavelengths in an environment where IR exists, the color of the RGB image may be distorted. Therefore, a method for generating an RGB image without color distortion even in an environment where IR exists may be needed by removing the IR signal entering the RGB pixels of the RGB-IR image sensor (11). According to an exemplary embodiment of the present disclosure, an image sensing device (10) can generate an RGB image without color distortion even in an environment where IR is strong by effectively removing the IR signal entering the RGB pixels. For example, the image sensing device (10) can generate a first RGB image including an IR component based on an RGB-IR image generated by the RGB-IR image sensor (11), and can generate a second RGB image with the IR component removed based on the RGB-IR image. Additionally, the image sensing device (10) can generate a third RGB image, which is an RGB image without color distortion, based on the brightness component of the first RGB image and the color component of the second RGB image. More specific details on generating an RGB image without color distortion using the image sensing device (10) will be described later.

[0061] The IR LED (12) may be a light source that irradiates IR light to an object. The IR LED (12) may be configured to be turned on and off at a predetermined cycle. Specifically, the IR LED (12) may be set to be turned on only when the image sensing device (10) captures an image for generating an IR image. Therefore, the image sensing device (10) according to an exemplary embodiment of the present disclosure may reduce power consumption by turning on the IR LED (12) only when necessary. For example, the IR LED (12) may be turned on only in one of two consecutive frames and turned off in the other frame, thereby turning the IR LED (12) on and off for each frame, but is not limited thereto.

[0062] The RGB-IR image processing unit (13) may be configured to generate an IR image and an RGB image based on RGB-IR images generated by the RGB-IR image sensor (11). For example, the RGB-IR image processing unit (13) may generate an IR image based on a first RGB-IR image captured through the RGB-IR image sensor (11) when the IR LED is turned on. Specifically, the RGB-IR image processing unit (13) may be configured to generate an IR image by performing a decolor operation on the first RGB-IR image. In addition, the RGB-IR image processing unit (13) may be configured to generate an IR image by performing denoising on the first RGB-IR image. In addition, the RGB-IR image processing unit (13) may generate an RGB image based on a second RGB-IR image captured through the RGB-IR image sensor (11) when the IR LED is turned off. More specific details of how the RGB-IR image processing unit (13) generates an RGB image will be described later.

[0063] The exposure value determination unit (14) may be configured to set the RGB-IR image sensor (11) to have a first exposure value when the IR LED (12) is turned on, and to set the RGB-IR image sensor (11) to have a second exposure value when the IR LED (12) is turned off. The RGB-IR image sensor (11) may adjust the amount of light received through exposure. Since the image sensing device (10) according to the exemplary embodiment of the present disclosure captures an image by turning the IR LED (12) on and off, it may be necessary to set the exposure value of the RGB-IR image sensor (11) differently depending on whether the IR LED (12) is turned on and off. For example, the exposure value determination unit (14) may determine the first exposure value based on brightness information of the IR image, and may determine the second exposure value based on brightness information of the RGB image, thereby setting the exposure value of the RGB-IR image sensor (11) differently.

[0064] The memory controller (15) can be configured to store an IR image and an RGB image in the memory (16) and to output the stored IR image and the stored RGB image simultaneously. Specifically, when the RGB-IR image processing unit (13) alternately outputs the IR image and the RGB image, the memory controller (15) can output two images by storing the alternately output IR image and the RGB image in the memory (16) and reading out information about the stored IR image and the RGB image at once.

[0065] The memory controller (15) may be configured to increase the frame rate of the output IR image by performing frame interpolation on the IR image sequence, and to increase the frame rate of the output RGB image by performing frame interpolation on the RGB image sequence. For example, assuming that the RGB-IR image sensor (11) outputs the RGB-IR image with a frame rate of 60 fps, and the RGB-IR image processing unit (13) alternately generates the IR image and the RGB image for each frame, the IR image and the RGB image may each be output with a frame rate of 30 fps. At this time, the memory controller (15) may increase the output frame rates of the IR image and the RGB image to 60 fps, respectively, by performing inter-frame interpolation. Frame interpolation may mean newly generating an intermediate frame between adjacent frames. Frame interpolation can be performed based on the direction of pixel movement by repeating the previous frame or by using pixel-by-pixel movement information between frames (e.g., motion vectors, optical flow).

[0066] The memory (16) can be configured to store IR images and RGB images.

[0067] FIG. 2 is a flowchart illustrating an image sensing method according to an exemplary embodiment of the present disclosure.

[0068] FIG. 3 is a drawing for explaining an RGB-IR pattern according to an exemplary embodiment of the present disclosure. Hereinafter, FIG. 2 will be explained with reference to FIG. 3.

[0069] Referring to FIG. 2, the image sensing method according to an exemplary embodiment of the present disclosure can turn the IR LED on and off at a predetermined cycle in step S210.

[0070] The image sensing method may generate an IR image based on a first RGB-IR image captured by an RGB-IR image sensor when the IR LED is turned on in step S220. The RGB-IR pattern of the RGB-IR image sensor may be the same as the pattern (310) of FIG. 3. Specifically, the RGB-IR pattern may be the same as a pattern in which some of the G pixels in the Bayer pattern (300) are changed into IR pixels. In addition, the RGB-IR pattern may be the pattern (320) of FIG. 3. Specifically, the RGB-IR pattern may be a pattern in which some of the R pixels and B pixels in the Bayer pattern (300) are changed into IR pixels. In addition, the RGB-IR pattern may be the same as the pattern (330) of FIG. 3. Specifically, the RGB-IR pattern may be a pattern in which some of the B pixels in the Bayer pattern (300) are changed into IR pixels. However, the RGB-IR pattern is not limited to the above-described pattern, and may have various patterns other than the pattern (310, 320, 330).

[0071] The image sensing method can generate an RGB image based on a second RGB-IR image captured through the RGB-IR image sensor when the IR LED is turned off in step S230.

[0072] Additionally, the image sensing method can set the RGB-IR image sensor to have a first exposure value when the IR LED is turned on, and can set the RGB-IR image sensor to have a second exposure value when the IR LED is turned off.

[0073] Additionally, the image sensing method can determine the first exposure value based on brightness information of the IR image, and can set the RGB-IR image sensor to have the second exposure value.

[0074] Additionally, the image sensing method can determine the first exposure value based on brightness information of the IR image and determine the second exposure value based on brightness information of the RGB image.

[0075] Additionally, the image sensing method can store IR images and RGB images in memory, and output the stored IR images and the stored RGB images simultaneously.

[0076] Additionally, the image sensing method can increase the frame rate of an output IR image by performing frame interpolation on an IR image sequence, and can increase the frame rate of an output RGB image by performing frame interpolation on an RGB image sequence.

[0077] Additionally, the image sensing method can generate an IR image by performing a decoloring operation on the first RGB-IR image. Additionally, the image sensing method can generate an IR image by performing denoising on the first RGB-IR image.

[0078] FIG. 4 is a block diagram of an image sensing device according to an exemplary embodiment of the present disclosure.

[0079] FIG. 5 is a diagram illustrating an image sensing method according to an exemplary embodiment of the present disclosure. Hereinafter, FIG. 4 will be described with reference to FIG. 5.

[0080] Referring to FIG. 4, the RGB-IR image processing unit described above may include an RGB interpolation unit (410), an IR interpolation unit (420), a brightness component processing unit (430), a color component processing unit (440), a merging unit (450), a pattern conversion unit (460), an IR processing unit (470), and a multiplexer (480).

[0081] For example, the RGB interpolation unit (410) can generate a first RGB image by performing interpolation on the R channel, the G channel, and the B channel based on the RGB-IR image captured when the IR LED is off. For example, the RGB interpolation unit (410) can perform a bilinear algorithm or perform interpolation based on the gradient of pixels surrounding the pixel to be interpolated. In addition, the RGB interpolation unit (410) can generate an estimated image by estimating an R pixel value, a G pixel value, or a B pixel value for an IR pixel position of the RGB-IR pattern so as to correspond to a Bayer pattern. For example, referring to (a) of FIG. 5, the G value for the positions of the IR pixels (511 to 514) of the RGB-IR pattern (510) can be estimated so that the RGB-IR pattern (510) corresponds to a Bayer pattern (515). A bilinear algorithm may be used to estimate the G value, and a method using the gradient of the surrounding RGB pixels of the IR pixels (511 to 514) may be used, but is not limited thereto. In addition, referring to FIG. 5 (b), the B value for the positions of the IR pixels (521 and 524) of the RGB-IR pattern (520) and the G value for the positions of the IR pixels (522 and 523) may be estimated so that the RGB-IR pattern (520) corresponds to the Bayer pattern (525). A bilinear algorithm may be used to estimate the B value and the G value, and a method using the gradient of the surrounding RGB pixels of the IR pixels (521 to 524) may be used, but is not limited thereto. In addition, referring to (c) of FIG. 5, the B value for the positions of the IR pixels (531 and 532) of the RGB-IR pattern (530) can be estimated so that the RGB-IR pattern (530) corresponds to the Bayer pattern (535).A bilinear algorithm may be used to estimate the B value, and a method utilizing the gradient of the surrounding RGB pixels of the IR pixels (531 and 532) may be used, but is not limited thereto. The RGB interpolation unit (410) may generate a first RGB image by performing demosaicing on an estimated image generated by estimating the R pixel value, G pixel value, or B pixel value at the IR pixel location.

[0082] The IR interpolation unit (420) can generate a first IR image by performing interpolation on an IR channel based on an RGB-IR image captured when the IR LED is turned off. For example, the IR interpolation unit (420) can estimate an IR value for each pixel by using a bilinear algorithm using IR pixel values ​​of the RGB-IR image. Additionally, the IR interpolation unit (420) can estimate an IR value for each pixel by using a gradient of RGB pixels surrounding the IR pixel.

[0083] The color component processing unit (440) can generate a second RGB image with the IR component removed based on the first RGB image and the first IR image. Since the RGB color of the first RGB image including the IR component is distorted by the IR component, if the first RGB image is directly used to extract the color component and the image is generated using the color component, the color of the generated image may be distorted. Therefore, the RGB image used for color component processing in the color component processing unit (440) may be a second RGB image that is an image obtained by removing the IR component from the first RGB image including the IR component. The color component processing unit (440) can generate the second RGB image by subtracting the first IR image from the first RGB image. However, since visible light and IR have different wavelengths, their focal lengths are different, which may cause crosstalk between pixels. Therefore, the second RGB image with the IR removed may not be generated simply by subtracting the first IR image from the first RGB image. Therefore, the following mathematical formula may be used to generate the second RGB image.

[0084]

[0085] Here, R, G, and B may be R values, G values, and B values ​​of the second RGB image, respectively. In addition, R', G', and B' may be R values, G values, and B values ​​of the first RGB image, respectively. In addition, IR' may be IR values ​​of the first IR image. In addition, the coefficient , and may be coefficients determined experimentally.

[0086] Additionally, a second RGB image can be generated through matrix operations, as simple first-order equations may not account for the influences (e.g., crosstalk) between R pixels, G pixels, B pixels, and IR pixels. For example, a second RGB image can be generated based on the following mathematical equation.

[0087]

[0088] Here, a00 to a23 are transformation matrix coefficients, and a second RGB image, which is a target image to be generated by the first RGB image and the first IR image, can be obtained through regression analysis. The method for generating the second RGB image is not limited to the above-described method, and the image sensing method can generate the second RGB image in various ways.

[0089] In addition, the brightness component processing unit (430) can calculate the brightness component value of the first RGB image. In other words, the brightness component processing unit (430) can calculate the brightness component value of the first RGB image, which is an image including an IR component. For example, the image sensing method can calculate the Y value of the first RGB image. Specifically, the image sensing method can calculate the Y value of the first RGB image using the following formula.

[0090]

[0091] However, the method of calculating the brightness component value in the image sensing method is not limited to the above-described method. For example, the image sensing method can obtain the brightness component value of the first RGB image (610) by converting the color domain of the first RGB image into the HSV (Hue, Saturation, Value) domain based on the R value, G value, and B value of the first RGB image and calculating the V value. In addition, the image sensing method can obtain the brightness component value of the first RGB image by converting the color domain of the first RGB image into the HSL (Hue, Saturation, Lightness) domain based on the R value, G value, and B value of the first RGB image and calculating the L value. In addition, the image sensing method can obtain the brightness component value of the first RGB image by converting the color domain of the first RGB image into the HSI (Hue, Saturation, Intensity) domain based on the R value, G value, and B value of the first RGB image and calculating the I value.

[0092] Additionally, the image sensing method can calculate the color component values ​​of the second RGB image. For example, the image sensing method can calculate the U value and the V value of the second RGB image. Specifically, the image sensing method can calculate the U value and the V value of the second RGB image using the following formula.

[0093]

[0094] However, the method for calculating the color component value in the image sensing method is not limited to the above-described method. For example, the image sensing method can obtain the color component value of the second RGB image by converting the color domain of the second RGB image into the HSV domain based on the R value, G value, and B value of the second RGB image and calculating the H value and the S value. In addition, the image sensing method can obtain the color component value of the second RGB image by converting the color domain of the second RGB image into the HSL domain based on the R value, G value, and B value of the second RGB image and calculating the H value and the S value. In addition, the image sensing method can obtain the color component value of the second RGB image by converting the color domain of the second RGB image into the HSI domain based on the R value, G value, and B value of the second RGB image and calculating the H value and the S value.

[0095] The image sensing method can generate a third RGB image by merging the brightness component of the first RGB image and the color component of the second RGB image. For example, the image sensing method can calculate the R value R'', the G value G'', and the B value B'' of the third RGB image using the Y value of the first RGB image, the U value, and the V value of the second RGB image using the following formula.

[0096]

[0097] However, the method for generating an RGB image based on brightness component values ​​and color component values ​​in the image sensing method is not limited to the above. For example, the image sensing method can generate R'' values, G'' values, and B'' values ​​by using the inverse matrix of the matrix used to obtain the Y value, U value, and V value.

[0098] The merging unit (450) can generate a third RGB image by merging the brightness component value of the first RGB image and the color component value of the second RGB image. For example, the merging unit (450) can generate a third RGB image by calculating the R value, G value, and B value using the Y value, which is the brightness component value of the first RGB image, and the U value and V value, which are the color component values ​​of the second RGB image.

[0099] The color component processing unit (440) can generate a fourth RGB image by removing the IR component from the first RGB image using the first IR image. In addition, the color component processing unit (440) can generate a second RGB image by increasing the color saturation of the fourth RGB image. In addition, the color component processing unit (440) can also generate the second RGB image by performing denoising on the fourth RGB image. In addition, the color component processing unit (440) can generate the second RGB image by using a matrix for removing the IR component from the first RGB image by considering crosstalk occurring between the R pixels, G pixels, B pixels, and IR pixels included in the pixel array of the RGB-IR pattern, pixel values ​​of the first RGB image, and pixel values ​​of the first IR image. In this case, the matrix may be a matrix obtained by performing regression analysis by setting the RGB image from which the IR component has been removed as a target RGB image.

[0100] The pattern conversion unit (460) can convert the third RGB image into a Bayer pattern image.

[0101] When the IR light from the IR LED is irradiated, the captured image is mostly composed of IR signals, and thus can be an IR image in itself. However, a small amount of RGB signals may be mixed in the captured image, which can be removed by a decoloring process. The IR processing unit (470) can generate a second IR image by performing a decoloring process or denoising on the captured image when the IR light from the IR LED is irradiated.

[0102] The multiplexer (480) can alternately output an RGB image of a Bayer pattern with the IR component removed and an IR image composed of an IR signal.

[0103] FIG. 6 is a drawing for explaining an image sensing method according to an exemplary embodiment of the present disclosure.

[0104] Referring to FIG. 6, according to an image sensing method according to an exemplary embodiment of the present disclosure, RGB-IR images may be generated by an RGB-IR sensor. Some of the RGB-IR images may be images captured while the IR LED is on, and the remaining portions may be images captured while the IR LED is off. For example, the IR LED may be turned on and off for each frame. In other words, one of two adjacent frames may be a frame captured while the IR LED is on, and the remaining frame may be a frame captured while the IR LED is off, but is not limited thereto.

[0105] The image sensing method can generate an IR image based on a first RGB-IR image captured by an RGB-IR image sensor when the IR LED is turned on, and can generate an RGB image based on a second RGB-IR image captured by the RGB-IR image sensor when the IR LED is turned off, through RGB / IR image processing.

[0106] The generated RGB image and the generated IR image can be stored in the frame memory under the control of the frame memory controller. The frame memory controller can read out the stored RGB image and IR image at the same time and output them simultaneously.

[0107] The RGB-IR image sensor can have two different exposure values ​​depending on whether the IR LED is turned on or off. The exposure value determination unit can perform an auto exposure function that determines the exposure level. Since the exposure value determination unit determines two exposure values, it can perform a dual auto exposure function.

[0108] Since the RGB-IR sensor must turn the IR LED on and off according to the image capture timing when capturing an image, the RGB-IR sensor can generate a synchronization signal (e.g., Vsync, Hsync). The IR LED can be controlled by the synchronization signal. The IR LED can be controlled to turn on only during the required frames and sections rather than being continuously on, and thus, power consumption can be reduced.

[0109] FIG. 7 is a diagram illustrating an image sensing method according to an exemplary embodiment of the present disclosure. Specifically, FIG. 7 is a diagram illustrating a frame-by-frame exposure process of an RGB-IR image sensor using a global shutter method. However, the RGB-IR sensor according to an exemplary embodiment of the present disclosure is not limited to the global shutter method.

[0110] Referring to FIG. 7, the IR LED can be turned on and off repeatedly for each frame. For example, the IR LED can be turned on during the exposure time of Frame N, which is the Nth frame, and the IR LED can be turned off after the exposure time for Frame N. In addition, the IR LED can be kept off during the exposure time of Frame N+1, which is the N+1th frame, and the IR LED can be turned on again during the exposure time of Frame N+2, which is the N+2nd frame, and then turned off after the exposure time has elapsed. The RGB-IR image sensor can convert the amount of light collected on the RGB-IR image sensor during the exposure time into an electrical signal and read it out in units of lines.

[0111] Additionally, the RGB-IR image sensor can have different exposure values ​​depending on whether the IR LED is turned on or off. For example, when the IR LED is turned on, the RGB-IR image sensor can have an exposure value of Exposure-0, and when the IR LED is turned off, the RGB-IR image sensor can have an exposure value of Exposure-1.

[0112] FIG. 8 is a drawing for explaining an image sensing method according to an exemplary embodiment of the present disclosure.

[0113] Referring to FIG. 8, the RGB-IR image processing unit can perform processing on an input RGB-IR image in accordance with a synchronization signal. For example, the RGB-IR image processing unit can perform IR processing and RGB processing based on a Vsync (Frame Sync) signal generated from an RGB-IR image sensor. Here, the IR processing can refer to a process of generating an IR image based on an image captured while the IR LED is on. Additionally, the RGB processing can refer to a process of generating an RGB image based on an image captured while the IR LED is off.

[0114] FIG. 9 is a drawing for explaining an image sensing method according to an exemplary embodiment of the present disclosure.

[0115] Referring to FIG. 9, the on period of the IR LED may need to match the exposure timing within the RGB-IR image sensor. The exposure timing within the RGB-IR image sensor can be determined through a synchronization signal output from the RGB-IR sensor. The exposure time is generally adjusted in units of lines. For example, when the exposure time is 1 line, as shown in FIG. 9, the synchronization signal (Hsync (Line sync)) and the signal regarding the IR LED ON may differ by tR and tF. Timing values ​​such as tR and tF can be finely adjusted so that the on timing of the IR LED and the internal exposure timing of the RGB-IR image sensor match.

[0116] Referring to FIG. 10, the image sensing device (100) may be a complementary metal oxide semiconductor image sensor (CIS) that converts an optical signal into an electrical signal. The image sensing device (100) may be controlled for on / off, operation mode, sensitivity, etc. by an image signal processor (ISP) (50).

[0117] The image sensing device (100) may include a pixel array (110), a row driver (120), a correlated double sampler (CDS) 130, an analog-to-digital converter (ADC) 140, an output buffer (150), a column driver (160), a conversion unit (170), and a timing controller (180). Here, each configuration of the image sensing device (100) is merely exemplary, and at least some of the configurations may be added or omitted as needed.

[0118] The pixel array (110) may include a plurality of pixels arranged in a plurality of rows and a plurality of columns. In one embodiment, the plurality of pixels may be arranged in a two-dimensional pixel array including rows and columns. For example, the pixel array (110) may have an RGB-IR pattern. In another embodiment, the plurality of unit image pixels may be arranged in a three-dimensional pixel array. The plurality of pixels may convert an optical signal into an electrical signal on a pixel-by-pixel basis or on a pixel-by-pixel group basis, and pixels within a pixel group may share at least a specific internal circuit. The pixel array (110) may receive a driving signal including a row selection signal, a pixel reset signal, and a transmission signal from the row driver (120), and a corresponding pixel of the pixel array (110) may be activated by the driving signal to perform an operation corresponding to the row selection signal, the pixel reset signal, and the transmission signal.

[0119] The row driver (120) can activate the pixel array (110) to perform specific operations on pixels included in a corresponding row based on commands and control signals supplied by the timing controller (180). In one embodiment, the row driver (120) can select at least one pixel arranged in at least one row of the pixel array (110). The row driver (120) can generate a row selection signal to select at least one row among a plurality of rows. The row driver (120) can sequentially enable a pixel reset signal and a transmission signal for pixels corresponding to at least one selected row. Accordingly, an analog reference signal and an image signal generated from each of the pixels of the selected row can be sequentially transmitted to the correlated double sampler (130). Here, the reference signal may be an electrical signal provided to the correlated double sampler (130) when the sensing node of the pixel (e.g., a floating diffusion node) is reset, and the image signal may be an electrical signal provided to the correlated double sampler (130) when the photocharge generated by the pixel is accumulated in the sensing node. The reference signal representing the reset noise unique to the pixel and the image signal representing the intensity of incident light may be collectively referred to as pixel signals.

[0120] CMOS image sensors can utilize correlated double sampling to remove unwanted offset values ​​of pixels, such as fixed pattern noise, by sampling the pixel signal twice to remove the difference between the two samples. For example, correlated double sampling removes unwanted offset values ​​by comparing pixel output voltages acquired before and after photocharges generated by incident light are accumulated at a sensing node, so that pixel output voltages based solely on incident light can be measured. In one embodiment, the correlated double sampler (130) can sequentially sample and hold a reference signal and an image signal provided to each of a plurality of column lines from the pixel array (110). That is, the correlated double sampler (130) can sample and hold the levels of the reference signal and the image signal corresponding to each of the columns of the pixel array (110).

[0121] The correlated double sampler (130) can transmit the reference signal and image signal of each column to the ADC (140) as a correlated double sampling signal based on a control signal from the timing controller (180).

[0122] The ADC (140) can convert the correlated double sampling signal for each column output from the correlated double sampler (130) into a digital signal and output it. In one embodiment, the ADC (140) can be implemented as a ramp-compare type ADC. The ramp-compare type ADC can include a comparison circuit that compares a ramp signal that rises or falls over time with an analog pixel signal, and a counter that performs a counting operation until the ramp signal matches the analog pixel signal. In one embodiment, the ADC (140) can convert the correlated double sampling signal generated by the correlated double sampler (130) for each column into a digital signal and output it.

[0123] The ADC (140) may include a plurality of column counters corresponding to each column of the pixel array (110). Each column of the pixel array (110) is connected to each column counter, and image data may be generated by converting a correlated double sampling signal corresponding to each column into a digital signal using the column counters. According to another embodiment, the ADC (140) may include one global counter, and may convert a correlated double sampling signal corresponding to each column into a digital signal using a global code provided by the global counter.

[0124] The output buffer (150) can temporarily hold and output image data of each column provided from the ADC (140). The output buffer (150) can temporarily store image data output from the ADC (140) based on a control signal of the timing controller (180). The output buffer (150) can operate as an interface that compensates for the difference in transmission (or processing) speed between the image sensing device (100) and other devices connected thereto.

[0125] The column driver (160) can select a column of the output buffer (150) based on a control signal of the timing controller (180), and control the output buffer (150) so that image data temporarily stored in the selected column of the output buffer (150) is sequentially output. In one embodiment, the column driver (160) can receive an address signal from the timing controller (180), and the column driver (160) can generate a column selection signal based on the address signal to select a column of the output buffer (150), thereby controlling image data to be output from the selected column of the output buffer (150) to an external source (e.g., an image signal processor (50)).

[0126] The conversion unit (170) can convert image data regarding an RGB-IR pattern into image data regarding a Bayer pattern and IR image data having an IR signal.

[0127] The timing controller (180) can control at least one of the row driver (120), the correlated double sampler (130), the ADC (140), the output buffer (150), and the column driver (160).

[0128] The timing controller (180) may provide clock signals required for the operation of each component of the image sensing device (100), control signals for timing control, and address signals for selecting a row or column to at least one of the row driver (120), the correlated double sampler (130), the ADC (140), the output buffer (150), and the column driver (160). According to one embodiment, the timing controller (180) may include a logic control circuit, a phase lock loop (PLL) circuit, a timing control circuit, and a communication interface circuit.

[0129] The image signal processor (50) processes image data input from the image sensing device (100) and can control each component of the photographing device (10) according to the processing result or according to an external input signal. The image signal processor (50) can reduce noise on the image data and perform image signal processing for improving image quality, such as gamma correction, color filter array interpolation, color matrix, color correction, and color enhancement. In addition, the image data generated by performing image signal processing for improving image quality can be compressed to generate an image file, or the image data can be restored from the image file. The compression format of the image can be a reversible format or an irreversible format. As an example of the compression format, in the case of a still image, the JPEG (Joint Photographic Experts Group) format or the JPEG 2000 format can be used. In addition, in the case of a video, a video file can be generated by compressing a plurality of frames according to the MPEG (Moving Picture Experts Group) standard. Image files can be created, for example, according to the Exif (Exchangeable image file format) standard.

[0130] Image data output from the image signal processor (50) can be stored in the internal memory or external memory of the image sensing system or displayed through a display according to a user's request or automatically.

[0131] In addition, the image signal processor (50) can perform blur processing, edge enhancement processing, image interpretation processing, image recognition processing, image effect processing, etc.

[0132] In addition, the image signal processor (50) can perform display image signal processing for display. For example, the image signal processor (50) can perform brightness level adjustment, color correction, contrast adjustment, outline emphasis adjustment, screen split processing, character image generation, and image synthesis processing.

[0133] The above description is merely an illustrative description of the technical idea of ​​the present disclosure, and those skilled in the art to which the present disclosure pertains will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present disclosure. Therefore, the embodiments disclosed in the present disclosure are not intended to limit the technical idea of ​​the present disclosure, but rather to explain it, and the scope of the technical idea of ​​the present disclosure is not limited by these embodiments. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present disclosure.

Claims

1. An RGB-IR image sensor comprising a pixel array of an RGB-IR pattern; an IR LED configured to turn on and off at a predetermined cycle; and An image sensing device comprising an RGB-IR image processing unit configured to generate an IR image based on a first RGB-IR image captured through the RGB-IR image sensor while the IR LED is turned on, and to generate an RGB image based on a second RGB-IR image captured through the RGB-IR image sensor while the IR LED is turned off.

2. In paragraph 1, An image sensing device characterized in that it further includes an exposure value determination unit configured to set the RGB-IR image sensor to have a first exposure value when the IR LED is turned on, and to set the RGB-IR image sensor to have a second exposure value when the IR LED is turned off.

3. In paragraph 2, The above exposure value determination unit, Determine the first exposure value based on the brightness information of the IR image, An image sensing device characterized in that it is configured to determine the second exposure value based on brightness information of the RGB image.

4. In paragraph 1, a memory configured to store the IR image and the RGB image; and An image sensing device characterized in that it further includes a memory controller configured to store the IR image and the RGB image in the memory and to simultaneously output the stored IR image and the stored RGB image.

5. In paragraph 4, The above memory controller, By performing frame interpolation on the IR image sequence, the frame rate of the output IR image is increased, An image sensing device characterized in that it is configured to increase the frame rate of the output RGB image by performing frame interpolation on the RGB image sequence.

6. In paragraph 1, The above RGB-IR image processing unit, An image sensing device characterized in that it is configured to generate the IR image by performing a decolor operation on the first RGB-IR image.

7. In paragraph 1, The above RGB-IR image processing unit, An image sensing device characterized in that it is configured to generate the IR image by performing denoising on the first RGB-IR image.

8. In paragraph 1, The above RGB-IR image processing unit, An image sensing device characterized in that it is configured to generate a first RGB image including an IR component based on the second RGB-IR image, generate a second RGB image from which the IR component is removed based on the second RGB-IR image and the first RGB image, generate a third RGB image based on a brightness component of the first RGB image and a color component of the second RGB image, and generate the RGB image by converting the third RGB image into a Bayer pattern image.

9. In paragraph 1, The above RGB-IR image sensor, An image sensing device characterized by being an image sensor of a global shutter type.

10. In paragraph 1, The above RGB-IR image sensor, An image sensing device characterized by being an image sensor of a rolling shutter type.

11. Step of turning the IR LED on and off at a predetermined cycle; A step of generating an IR image based on a first RGB-IR image captured through an RGB-IR image sensor while the IR LED is turned on; and An image sensing method comprising the step of generating an RGB image based on a second RGB-IR image captured through the RGB-IR image sensor while the IR LED is turned off.

12. In paragraph 11, A step of setting the RGB-IR image sensor to have a first exposure value when the IR LED is turned on; and An image sensing method further comprising a step of setting the RGB-IR image sensor to have a second exposure value when the IR LED is turned off.

13. In paragraph 12, The step of setting the RGB-IR image sensor to have the first exposure value is as follows: A step of determining the first exposure value based on brightness information of the IR image, The step of setting the RGB-IR image sensor to have the second exposure value is as follows: An image sensing method characterized by comprising a step of determining the second exposure value based on brightness information of the RGB image.

14. In paragraph 11, A step of storing the IR image and the RGB image in memory; and An image sensing method characterized by comprising a step of simultaneously outputting the stored IR image and the stored RGB image.

15. In paragraph 14, The step of simultaneously outputting the stored IR image and the stored RGB image is: A step of increasing the frame rate of the output IR image by performing frame interpolation on the IR image sequence; and An image sensing method, characterized by comprising a step of increasing the frame rate of the output RGB image by performing frame interpolation on the RGB image sequence.

16. In paragraph 11, The step of generating the above IR image is: An image sensing method characterized by comprising a step of generating the IR image by performing a decolor operation on the first RGB-IR image.

17. In paragraph 11, The step of generating the above IR image is: An image sensing method characterized by comprising a step of generating the IR image by performing a decolor operation on the first RGB-IR image.

18. In paragraph 11, The steps for generating the above RGB image are: A step of generating a first RGB image including an IR component based on the second RGB-IR image; A step of generating a second RGB image from which the IR component has been removed based on the second RGB-IR image and the first RGB image; A step of generating a third RGB image based on the brightness component of the first RGB image and the color component of the second RGB image; and An image sensing method characterized by comprising a step of generating the RGB image by converting the third RGB image into a Bayer pattern image.

19. In paragraph 11, The above RGB-IR image sensor, An image sensing method characterized by being an image sensor of a global shutter type.

20. In paragraph 11, The above RGB-IR image sensor, An image sensing method characterized by being an image sensor of a rolling shutter type.

Images