Infrared light imaging device for generating wide dynamic range image

Abstract

The present invention relates to an IR imaging device including: an image capturing unit which includes a pixel array having a spatially varying exposure (SVE) pixel pattern of a size of N x N in which pixels are classified into pixel groups so that pixels having an identical sensitivity belong to an identical group and pixels having multiple sensitivities are arranged so that pixels having an identical sensitivity are not adjacent to each other, wherein the multiple pixels having the multiple sensitivities concurrently detect light with the multiple sensitivities so as to capture an image; and an image restoring unit for acquiring multiple images having different sensitivities by generating an image having a low resolution for each pixel group by using only data of multiple pixels belonging to an identical pixel group among the data of the captured image and interpolating data of pixels belonging to different pixel groups in a demosaicing method to restore the resolution thereof.

Description

Infrared imaging device that produces wide dynamic range images

[0001] The present invention relates to an infrared imaging device for generating a wide dynamic range image.

[0002] The real world, as perceived visually by humans, contains a wide range of brightness levels. There can be significant differences in brightness between bright areas illuminated by light and dark areas in shadows. Conventional imaging devices have limitations in accurately capturing images with varying brightness levels.

[0003] HDR (high dynamic range) imaging devices address these issues by supporting a wide dynamic range. HDR imaging devices combine multiple images with different exposures to create HDR images.

[0004] Near-infrared (IR) imaging devices can capture information in the infrared region, invisible to the human eye, and are therefore useful in a variety of environments. Typically, NIR imaging devices have all pixels with the same structure and sensitivity. Furthermore, global shutter imaging devices collect light for the same amount of time from all pixels, making it impossible to capture multiple images with different exposures or sensitivities. Consequently, this makes it difficult to obtain infrared images with a wide dynamic range. However, as the applications of NIR imaging devices diversify, the need for NIR imaging devices capable of capturing images with a wider dynamic range is growing.

[0005] The present invention seeks to provide an infrared imaging device capable of generating HDR images even in a global shutter manner by changing a pixel pattern structure.

[0006] According to an embodiment of the present invention, an IR (infrared light) imaging device includes a pixel array having an N×N sized SVE (spatially varying exposure) pixel pattern in which a plurality of pixels having a plurality of sensitivities are arranged so that pixels having the same sensitivities are not adjacent to each other, an image capture unit for capturing an image by simultaneously detecting light with the plurality of sensitivities by the plurality of pixels having the plurality of sensitivities, and an image restoration unit for generating a low-resolution image for each pixel group using only data of a plurality of pixels belonging to the same pixel group among data of the captured image, and restoring the resolution by interpolating data of pixels belonging to different pixel groups in a demosaicing manner, thereby obtaining a plurality of images having different sensitivities, wherein the pixel groups are classified such that pixels having the same sensitivities belong to the same group.

[0007] The present invention can generate HDR (high dynamic range) images having a wider dynamic range through image processing technology, using the structure of an existing SDR (standard dynamic range) imaging device without significantly changing the hardware configuration.

[0008] In addition, since multiple images are acquired from images shot with the same exposure to create HDR images, motion artifacts may not occur, and it can also be applied to global shutter type imaging devices.

[0009] FIG. 1 is a block diagram of an IR imaging device according to one embodiment of the present invention.

[0010] FIG. 2A illustrates an image capture unit including a two-channel pixel pattern according to one embodiment of the present invention.

[0011] FIG. 2b illustrates an image capture unit including a three-channel pixel pattern according to one embodiment of the present invention.

[0012] FIG. 3 is a block diagram of an image restoration unit according to an embodiment of the present invention.

[0013] FIG. 4a illustrates an image restoration process in a two-channel pixel pattern according to an embodiment of the present invention.

[0014] FIG. 4b illustrates an image restoration process in a two-channel pixel pattern according to an embodiment of the present invention.

[0015] FIG. 5a is a block diagram of an image synthesis unit according to an embodiment of the present invention.

[0016] FIG. 5b is a block diagram of an image synthesis unit according to an embodiment of the present invention.

[0017] FIG. 6 illustrates a high-sensitivity image captured with a general imaging device, a low-sensitivity image captured with a general imaging device, and an HDR image result according to an embodiment of the present invention.

[0018] Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components will be given the same reference numerals, even if they appear in different drawings. Furthermore, when describing embodiments of the present invention, detailed descriptions of related known structures or functions will be omitted if they are deemed to hinder understanding of the embodiments of the present invention.

[0019] Conventional near-infrared IR imaging devices have all pixels with the same structure and sensitivity. Because all pixels collect light for the same amount of time, pixels easily reach saturation in bright areas. In dark areas, insufficient light is collected, resulting in increased noise. Furthermore, because all pixels begin and end exposure simultaneously, it is difficult to adjust the exposure for each individual pixel. Therefore, it is difficult to acquire HDR images using existing IR imaging devices. The present invention proposes an IR imaging device capable of acquiring images with multiple different sensitivities for HDR image generation without significantly changing the pixel structure of existing sensors.

[0020] FIG. 1 is a block diagram of an IR imaging device according to one embodiment of the present invention.

[0021] An IR (infrared light) imaging device (100) may include a configuration for generating multiple images with different sensitivities to generate an image capable of expressing a wider dynamic range. In addition, the IR imaging device (100) may include a configuration for restoring the resolution of the image to simultaneously maintain the quality of the resolution while generating multiple images with different sensitivities.

[0022] Referring to FIG. 1, the IR imaging device (100) may include an image capture unit (105), an image restoration unit (110), and an image synthesis unit (115).

[0023] In one example, the image capture unit (105) may include a pixel array, an exposure control circuit, an analog-to-digital converter (ADC), and / or an interface. For example, the image capture unit (105) may collect an optical signal through the pixel array and convert it into an electrical signal. The pixel array may have, for example, an N×N sized SVE (spatially varying exposure) pixel pattern in which a plurality of pixels having a plurality of sensitivities are arranged so that pixels having the same sensitivities do not adjoin each other. The SVE pixel pattern may be a pattern in which a plurality of pixels having different sensitivities are regularly arranged. The SVE pixel pattern may be N×N sized. For example, in the case of a 2×2 sized SVE pixel pattern, two pixels having different sensitivities may be arranged in one column, and two pixels having the same sensitivities as the pixels arranged in the first column may be arranged in the second column. N may include a natural number greater than or equal to 2. The image capture unit (105) can control the exposure time by controlling the time at which each pixel collects charges through an exposure control circuit when converting an optical signal into an electrical signal through a pixel array. The electrical signal of the image captured through the pixel array can be converted into a digital signal through an analog-to-digital signal converter. The converted digital signal can be transmitted to the image restoration unit (110) through an interface.

[0024] The image capture unit (105) can capture a single captured image (120) by using the plurality of pixels to allow the plurality of pixels to detect light at multiple sensitivities simultaneously. The single captured image (120) captured by the image capture unit (105) can include an image with different sensitivities depending on the pixel location. A pixel group can mean a set of pixels classified such that pixels having the same sensitivity among the plurality of pixels belong to the same group.

[0025] The image capture unit (105) can capture images using a global shutter method and / or a rolling shutter method. For example, the rolling shutter method may include a method of photographing an object by reading the image line by line. The rolling shutter method may cause distortion when photographing a fast-moving object. For example, the global shutter method may include a method of photographing an object by reading the entire image at once. Compared to the rolling shutter method, the global shutter method may have the advantage of preventing distortion caused by fast movement.

[0026] In one example, the image restoration unit (110) can obtain one image for each pixel group by filtering data of pixels belonging to other pixel groups except for pixels belonging to a specific pixel group using one captured image (120) captured by the image capture unit (105). For example, the image restoration unit (110) can obtain one image with a low resolution for each pixel group using only data of multiple pixels belonging to the same pixel group among the data of the captured captured image (120). Since the images obtained for each pixel group do not contain data of pixels belonging to other pixel groups, they may include images with a lower resolution than images captured using all pixels. Since the image restoration unit (110) can obtain one new image for each pixel group, it can obtain multiple images with a lower resolution as a result. The image restoration unit (110) can generate an image with a restored resolution by interpolating data at the positions of pixels belonging to other pixel groups where data does not exist using a demosaicing method using the images for each low-resolution pixel group. The image restoration unit (110) can transmit a plurality of restored images (125) to the image synthesis unit (115).

[0027] In one example, the image synthesis unit (115) can synthesize a plurality of restored images (125) to generate an HDR image (130). For example, the image synthesis unit (115) can synthesize a plurality of restored images (125) into a single synthesized image and apply tone mapping to the synthesized image to generate an HDR image (130). For example, the image synthesis unit (115) can synthesize an HDR image (130) using a plurality of restored images (125) in an exposure fusion manner.

[0028] In one example, the captured image (120), the restored images (125), and / or the HDR image (130) may include grayscale images. Each pixel constituting the grayscale image may include only a brightness value between white and black as data. In the present disclosure, a “channel” may refer to pixel data having a specific sensitivity. For example, pixel data of different channels may refer to pixel data having different sensitivities. For example, pixel data of the same channel may refer to pixel data having the same sensitivity. For example, a two-channel pixel pattern may include a pixel pattern configured by arranging pixels having two different sensitivities. For example, a three-channel pixel pattern may include a pixel pattern configured by arranging pixels having three different sensitivities.

[0029] FIG. 2A illustrates an image capture unit including a two-channel pixel pattern according to one embodiment of the present invention.

[0030] Referring to FIGS. 1 and 2A, the image capture unit (105) may include a pixel array having a two-channel pixel pattern. For example, the image capture unit (105) may include a pixel array having a 2×2 sized SVE pixel pattern in which two first pixels (210) and two second pixels (220) are alternately arranged so that pixels of the same type are not adjacent to each other.

[0031] In one example, the first pixel (210) may include pixels having a normal sensitivity. For example, the first pixel may include pixels that have not undergone additional processing to lower the sensitivity. A set of a plurality of first pixels (210) may be referred to as a first pixel group. Since the first pixel (210) has a relatively higher sensitivity compared to the second pixel (220), the image data captured by the first pixel (210) may include high-sensitivity image data. The high-sensitivity image may play a role similar to a long-exposure image in generating an HDR image.

[0032] In one example, the second pixel (220) may include a pixel with a lower sensitivity than the first pixel. For example, the second pixel (220) may include a pixel with a lower sensitivity by applying an ND (neutral density) filter or by adjusting the amount of light by covering a portion of the pixel with a layer (e.g., metal) that has undergone a specific process. A set of a plurality of second pixels (220) may be referred to as a second pixel group. Image data captured by the second pixel (220) may include low-sensitivity image data. The low-sensitivity image may play a role similar to a short-exposure image in generating an HDR image.

[0033] Although the two-channel pixel pattern is illustrated in FIG. 2A as having a 2×2 size, the embodiments in the present disclosure are not limited thereto. For example, the size of the two-channel pixel pattern may be 3×3 and / or 4×4. Although the two-channel pixel pattern in FIG. 2A is illustrated as having the first pixels (210) arranged at the upper right and lower left, and the second pixels (220) arranged at the lower right and upper left, the embodiments in the present disclosure are not limited thereto. For example, the two-channel pixel pattern may include a pattern in which the positions of the first pixels (210) and the second pixels (220) are arranged in a form where the positions are switched in the form illustrated in FIG. 2. For example, the two-channel pixel pattern may include a pattern in which two second pixels (220) are arranged on the left, and two first pixels (210) are arranged on the right. For example, a two-channel pixel pattern may include a pattern in which two first pixels (210) are placed on the left and two second pixels (220) are placed on the right.

[0034] FIG. 2b illustrates an image capture unit including a three-channel pixel pattern according to one embodiment of the present invention.

[0035] Referring to FIGS. 1 and 2B, the image capture unit (105) may include a pixel array having a 3-channel pixel pattern. For example, the image capture unit (105) may include a pixel array having a 2×2 sized SVE pixel pattern in which two first pixels (210), one second pixel (220), and / or one third pixel (230) are alternately arranged so that pixels of the same type are not adjacent to each other.

[0036] In one example, the first pixel (210) may include a pixel having a general sensitivity. Referring to FIG. 2A, the first pixel (210) may be referred to as described above in FIG. 2A.

[0037] In one example, the second pixel (220) may include a pixel with a lower sensitivity than the first pixel. Referring to FIG. 2A, the second pixel (220) may be referred to as described above in FIG. 2A.

[0038] In one example, the third pixel (230) may include a pixel with a lower sensitivity than the second pixel. For example, the third pixel (230) may include a pixel with a lower sensitivity by applying an ND (neutral density) filter or by adjusting the amount of light by covering a portion of the pixel with a layer (e.g., metal) that has undergone a specific process. A set of a plurality of third pixels (230) may be referred to as a third pixel group. Image data captured by the third pixel (230) may include ultra-low sensitivity image data.

[0039] Although the 3-channel pixel pattern in FIG. 2B is illustrated as having a 2×2 size, embodiments in the present disclosure are not limited thereto. For example, the size of the 3-channel pixel pattern may be 3×3 and / or 4×4. Although the 3-channel pixel pattern in FIG. 2B is illustrated as having two first pixels, one second pixel, and / or one third pixel arranged, embodiments in the present disclosure are not limited thereto. For example, the 3-channel pattern may include having two second pixels, one first pixel, and / or one third pixel arranged. For example, the 3-channel pixel pattern may include having two third pixels, one second pixel, and / or one first pixel arranged.

[0040] FIG. 3 is a block diagram of an image restoration unit according to an embodiment of the present invention.

[0041] Referring to FIGS. 1 and 3, the image restoration unit (110) can generate a plurality of restored images (125) using the captured image (120). For example, the image restoration unit (110) can obtain a low-resolution image by using only the data of pixels belonging to a portion of a pixel group of the captured captured image (120), and process and use the data of pixels belonging to the remaining pixel groups to restore the resolution of the low-resolution image. For example, the image restoration unit (110) can include a path for restoring images equal to the number of restored images (125) to be restored. The image restoration unit (110) can include a path for restoring images equal to the number of channels of the pixel pattern. For example, referring to FIG. 2A, in a two-channel pixel pattern, when the image restoration unit (110) restores the image with the sensitivity of the first pixel (210), the restored image can be generated using data of pixels belonging to the first pixel group and data of pixels belonging to the second pixel group processed through the gain application unit (310) and / or the filter application unit (320). For example, referring to FIG. 2A, in a two-channel pixel pattern, when the image restoration unit (110) restores the image with the sensitivity of the second pixel (210), the restored image can be generated using data of pixels belonging to the second pixel group and data of pixels belonging to the first pixel group processed through the gain application unit (310) and / or the filter application unit (320).

[0042] In one example, the gain application unit (310) may apply a gain to pixel data belonging to a certain pixel group to restore the resolution. The gain applied by the gain application unit (310) may include a digital gain. The digital gain may include amplifying or reducing digital data converted from an analog signal. For example, a method of applying the digital gain may include a method of applying a weight to the digital data converted from an analog signal. In one example, the gain application unit (310) may apply the gain by applying an upweight to the image data of low-sensitivity pixels when restoring a high-sensitivity image. In one example, the gain application unit (310) may apply the gain by applying a downweight to the image data of high-sensitivity pixels when restoring a low-sensitivity image.

[0043] In one example, the gain application unit (310) may apply a gain that is greater or less than the difference in sensitivity between the high-sensitivity pixels and the low-sensitivity pixels. The gain may affect the noise and resolution of the restored image. The gain may have a trade-off relationship between resolution restoration and noise. For example, if the gain value increases, the noise of the restored image may increase. For example, if the gain value decreases, if the ambient illumination is low, the pixel data value may not be large enough to restore the resolution even if the gain is applied, making it difficult to restore the resolution. For example, as the gain value approaches the difference in sensitivity between the high-sensitivity pixels and the low-sensitivity pixels, the resolution quality of the restored image may improve. For example, the gain application unit (310) may directly tune and set the gain value based on the ambient illumination, the level of noise, and / or the resolution quality of the restored image. For example, the gain application unit (310) may receive a preset gain value from an external source and apply the gain. For example, if there is no significant difference in resolution between an image restored by applying gain and an image restored without applying gain, the gain application unit (310) may not apply gain.

[0044] In one example, the image restoration unit (110) may generate restored images (125) using only the gain application unit (310). For example, if there is little noise and the resolution of the restored image is not affected, the image restoration unit (110) may obtain a high-sensitivity restored image by using the data processed by simply multiplying the low-sensitivity pixel data by a gain equal to the difference in sensitivity between the high-sensitivity pixels and the low-sensitivity pixels and the high-sensitivity pixel data. For example, if there is little noise and the resolution of the restored image is not affected, the image restoration unit (110) may obtain a low-sensitivity restored image by using the data processed by simply dividing the high-sensitivity pixel data by a gain equal to the difference in sensitivity between the high-sensitivity pixels and the low-sensitivity pixels and the low-sensitivity pixel data.

[0045] In one example, the filter application unit (320) may apply filtering to pixel data belonging to a certain pixel group to restore resolution. For example, the filter application unit (320) may apply a low band filter to remove noisy data to remove severe noise. For example, the filter application unit (320) may adjust pixel data by applying a filter (e.g., a Gaussian filter and / or a bilateral filter) to remove data clipped to zero.

[0046] In one example, the image restoration unit (110) may generate restored images (125) using only the gain application unit (310) and the filter application unit (320). For example, if there is little noise and the resolution of the restored image is not affected, the image restoration unit (110) may multiply the low-sensitivity pixel data by a gain equal to the difference in sensitivity between the high-sensitivity pixels and the low-sensitivity pixels using the gain application unit (310), apply filtering using the filter application unit (320), and use the processed data and the high-sensitivity pixel data to obtain a high-sensitivity restored image. For example, if there is little noise and the resolution of the restored image is not affected, the image restoration unit (110) may divide the high-sensitivity pixel data by a gain equal to the difference in sensitivity between the high-sensitivity pixels and the low-sensitivity pixels using the gain application unit (310), and use the processed data and the low-sensitivity pixel data to obtain a low-sensitivity restored image by applying filtering using the filter application unit (320).

[0047] Although FIG. 3 illustrates that the gain is applied first in the gain application unit (310) and then filtering is applied in the filter application unit (320), the embodiments in the present disclosure are not limited thereto. For example, the image restoration unit (110) may first apply filtering in the filter application unit (320) and then apply the gain in the gain application unit (310). For example, the image restoration unit (110) may omit the process of applying the gain through the gain application unit (310) when generating the restored images (125). For example, the image restoration unit (110) may omit the process of applying filtering through the filter application unit (320) when generating the restored images (125). For example, the image restoration unit (110) may omit the process of using the gain application unit (310) and / or the filter application unit (320) when generating the restored images (125).

[0048] In one example, the pixel data restoration unit (330) can restore the resolution of an image by interpolating data of other channels in which data does not exist. The pixel restoration unit (330) can generate restored images as many as the number of pixel groups of the pixel array included in the image capture unit (105). The pixel restoration unit (330) can use a demosaicing algorithm (e.g., a linear minimum mean square error (LMMSE) demosaicing algorithm and / or a gradient-based threshold free (GBTF) demosaicing algorithm) to interpolate data of pixels belonging to pixel groups in which data does not exist in an RGB Bayer filter. For example, the pixel restoration unit (330) can map an SVE pixel pattern to an RGB Bayer pattern and apply an RGB Bayer demosaicing algorithm. For example, in order to restore a high-sensitivity image from a 2-channel pixel pattern, the pixel restoration unit (330) may apply a demosaicing algorithm by mapping pixels of a pixel group having high sensitivity to pixels at the G channel position among RGB channels, and by mapping pixels of a pixel group having low sensitivity to pixels at the R channel and / or B channel positions among RGB channels. The pixel restoration unit (330) may interpolate high-sensitivity data at positions of pixels belonging to a pixel group having low sensitivity where high-sensitivity channel data did not exist by applying the demosaicing algorithm in the above manner. For example, in order to restore a low-sensitivity image from a 2-channel pixel pattern, the pixel restoration unit (330) may map pixels of a pixel group having low sensitivity to pixels at the G channel position among RGB channels, and by mapping pixels of a pixel group having high sensitivity to pixels at the R channel and / or B channel positions among RGB channels, and thereby apply a demosaicing algorithm.The pixel restoration unit (330) can interpolate low-sensitivity data at the positions of pixels belonging to a pixel group having high sensitivity where low-sensitivity channel data did not exist by applying the demosaicing algorithm in the above manner.

[0049] In one example, the noise reduction processing unit (340) may receive restored images (331) before noise reduction processing from the pixel data restoration unit (330) as input and perform noise reduction processing on the input images. The restored images (125) may be generated by performing noise reduction processing on the images restored by the pixel data restoration unit (330) through the noise reduction processing unit (340). For example, the noise reduction processing may include processing utilizing a denoising algorithm (e.g., Bilateral, Gaussian filtering, NLM, and / or 3DNR) that utilizes spatial and temporal characteristics. For example, the noise reduction processing unit (340) may perform noise reduction processing on all or part of the restored images (331) before noise reduction processing that were received. For example, the noise reduction processing unit (340) may not perform noise reduction processing on all or part of the restored images (331) before noise reduction processing that were received.

[0050] FIG. 4a illustrates an image restoration process in a two-channel pixel pattern according to an embodiment of the present invention.

[0051] The image restoration unit (110) may include paths for restoring images equal to the number of channels of the pixel pattern. FIGS. 4A and 4B each separately illustrate two examples of paths for restoring images in a two-channel pixel pattern.

[0052] Referring to FIGS. 1 and 4A, the image restoration unit (110) may interpolate data at the positions of pixels belonging to the second pixel group to restore the first pixel group image. The first pixel group image may refer to an image whose resolution is restored by interpolating pixel data of the remaining pixel groups that do not belong to the first pixel group with the sensitivities of the pixels belonging to the first pixel group.

[0053] The image restoration unit (110) can generate a first pixel group restoration image (440) by applying a gain to the second pixel group image data (420) through the gain application unit (310), applying filtering through the filter application unit (320), processing the data, applying a demosaicing algorithm to the first pixel group image data (310) through the pixel data restoration unit (330), and performing noise reduction processing through the noise reduction processing unit (340).

[0054] Referring to FIG. 3, the process in which the gain application unit (310) applies gain to the second pixel group image data (420), the process in which the filter application unit (320) applies filtering to the second pixel group image data (420), the process in which the pixel data restoration unit (330) applies a demosaicing algorithm using the processed second pixel group image data and the first pixel group image data (410), and / or the process in which the noise reduction processing unit (340) performs noise reduction processing on the first pixel group restored image (430) before noise reduction processing may be referred to as the contents described above in FIG. 3.

[0055] FIG. 4b illustrates an image restoration process in a two-channel pixel pattern according to an embodiment of the present invention.

[0056] The image restoration unit (110) may include paths for restoring images equal to the number of channels of the pixel pattern. FIGS. 4A and 4B each separately illustrate two examples of paths for restoring images in a two-channel pixel pattern.

[0057] Referring to FIGS. 1 and 4B, the image restoration unit (110) may interpolate data at the positions of pixels belonging to the first pixel group to restore a second pixel group image. The second pixel group image may refer to an image whose resolution is restored by interpolating pixel data of the remaining pixel groups that do not belong to the second pixel group with the sensitivities of the pixels belonging to the second pixel group.

[0058] The image restoration unit (110) can generate a second pixel group restoration image (445) by applying a gain to the first pixel group image data (410) through the gain application unit (310), applying filtering through the filter application unit (320), processing the data, applying a demosaicing algorithm to the second pixel group image data (310) through the pixel data restoration unit (330), and performing noise reduction processing through the noise reduction processing unit (340).

[0059] Referring to FIG. 3, the process in which the gain application unit (310) applies gain to the first pixel group image data (410), the process in which the filter application unit (320) applies filtering to the first pixel group image data (410), the process in which the pixel data restoration unit (330) applies a demosaicing algorithm using the processed first pixel group image data and the second pixel group image data (420), and / or the process in which the noise reduction processing unit (340) performs noise reduction processing on the second pixel group restored image (435) before noise reduction processing may be referred to as the contents described above in FIG. 3.

[0060] FIG. 5a is a block diagram of an image synthesis unit according to an embodiment of the present invention.

[0061] Referring to FIGS. 1 and 5A, the image synthesis unit (115) may include a synthesis unit (510) and / or a tone mapping application unit (520). The synthesis unit (510) may generate a synthesis image by combining restored images (125). For example, the synthesis unit (510) may generate a synthesis image by combining pixel data at the same location by applying a weight to the brightness of each pixel of each restored image (125). For example, the synthesis unit (510) may receive and synthesize a first restored image and a second restored image having different sensitivities. For example, when the difference in sensitivity between the first restored image and the second restored image is N (e.g., 16), the synthesis unit (510) may obtain a pixel value of the synthesis image by multiplying the pixel value of the second restored image by N (e.g., 16) and adding the pixel value of the first restored image. For example, if the first restored image and the second restored image each have a 10-bit depth and N is 16, the synthesized image generated by the synthesis unit (510) may have a 14-bit depth. The synthesized image having a 14-bit depth may have a higher dynamic range than the first restored image and the second restored image having a 10-bit depth. The synthesized image may include an image composed of data having a wider dynamic range than the first restored image or the second restored image.

[0062] The tone mapping application unit (520) can apply tone mapping to a synthetic image to generate an HDR image (130). “Tone mapping” may be referred to as adjusting brightness and color to output an image with a wide dynamic range on an output interface with a low dynamic range (e.g., an LDR (low dynamic range) display). Applying tone mapping may be referred to as applying a tone mapping algorithm (e.g., a tone mapping algorithm using sigmoid and / or histogram).

[0063] FIG. 5b is a block diagram of an image synthesis unit according to an embodiment of the present invention.

[0064] Referring to FIG. 1 and FIG. 5B, the image synthesis unit (115) may include an exposure fusion synthesis unit (530). The exposure fusion synthesis unit (530) may generate an HDR image (130) using an exposure fusion algorithm. For example, the exposure fusion synthesis unit (530) may receive restored images (125) as input, analyze the exposure of the input restored images (125), extract parts expressed with optimal exposure among each restored image (125), and continuously connect the extracted parts to generate an HDR image (130).

[0065] FIG. 6 illustrates a high-sensitivity image captured with a general imaging device, a low-sensitivity image captured with a general imaging device, and an HDR image result according to an embodiment of the present invention.

[0066] Referring to FIGS. 1 and 6, a high-sensitivity image (610) captured with a general imaging device may be an image that cannot express details in bright areas due to high sensitivity and saturated light in bright areas. A high-sensitivity image (610) captured with a general imaging device may be an image suitable for expressing details in dark areas, but unsuitable for expressing details in bright areas.

[0067] A low-sensitivity image (620) captured with a general imaging device may be an image that cannot express details in dark areas because the pixel data in dark areas is close to 0 due to low sensitivity. A low-sensitivity image (620) captured with a general imaging device may be an image that is suitable for expressing details in bright areas, but is unsuitable for expressing details in dark areas.

[0068] An HDR image (130) according to an embodiment of the present invention may include an image having a wider dynamic range compared to a high-sensitivity image (610) captured with a general imaging device and / or a low-sensitivity image (620) captured with a general imaging device. An HDR image (130) according to an embodiment of the present invention may include an image in which light in a bright part is not saturated, so that details in a bright part can be expressed, and at the same time, pixel data in a dark part is not close to 0, so that details in a dark part can also be expressed.

Claims

1. An image capture unit comprising a pixel array having a square-shaped pixel pattern in which a plurality of pixels belonging to a plurality of pixel groups are arranged so that pixels having different sensitivities are adjacent to each other, and which detects light with a plurality of sensitivities using the pixel array to capture an image; and An image restoration unit is included that creates an image for each pixel group with a lower resolution than the captured image by using data of pixels belonging to the same pixel group among the data of the captured image, and restores the resolution by demosaicing data of pixels belonging to different pixel groups, thereby obtaining multiple images with different sensitivities. The above plurality of pixel groups are IR (infrared light) imaging devices that are groups in which pixels included in the pixel array are classified by sensitivity.

2. In paragraph 1, The above image capture unit is an IR imaging device including a pixel array having a 2×2 sized pixel pattern in which first pixels having normal sensitivity and second pixels having lower sensitivity than the first pixels are alternately arranged adjacent to each other.

3. In paragraph 1, An IR imaging device comprising a pixel array having a 2×2 sized pixel pattern in which at least one first pixel having a normal sensitivity, at least one second pixel having a lower sensitivity than the first pixel, and at least one third pixel having a lower sensitivity than the second pixel are arranged.

4. In paragraph 1, The above image capture unit is an IR imaging device that captures images using a global shutter method or a rolling shutter method.

5. In paragraph 1, An IR imaging device further comprising an image synthesis unit that generates a composite image by synthesizing the plurality of images acquired from the image restoration unit and applies tone mapping to the composite image.

6. In paragraph 1, An IR imaging device further comprising an image synthesis unit that generates a composite image by synthesizing the plurality of images acquired from the image restoration unit using an exposure fusion method.

7. In paragraph 1, An IR imaging device, wherein the image restoration unit includes a gain application unit that can apply a gain to each pixel group, a filter application unit that filters out data unnecessary for image restoration, and a pixel data restoration unit that demosaices data of pixels in which data does not exist by applying a demosaicing algorithm.

8. In paragraph 7, The above plurality of pixel groups are composed of a first pixel group and a second pixel group, The image restoration unit performs at least one of the following operations: when restoring the resolution of an image generated from image data corresponding to pixels belonging to a first pixel group, applying a gain to data of pixels belonging to a second pixel group through a gain application unit, or filtering at least a portion of data of pixels belonging to a second pixel group through a filter application unit. An IR imaging device that interpolates pixel data at locations of pixels belonging to a second pixel group where data does not exist among pixel data of an image generated from pixels belonging to the first pixel group by applying a demosaicing algorithm.

9. In paragraph 7, The above plurality of pixel groups are composed of a first pixel group and a second pixel group, An IR imaging device in which, when the image restoration unit restores the resolution of an image generated by pixels belonging to the first pixel group, a gain application unit applies a gain equivalent to the difference in sensitivity between the first pixel group and the second pixel group to the data of pixels belonging to the second pixel group.

10. In paragraph 7, The above plurality of pixel groups are composed of a first pixel group and a second pixel group, The image restoration unit performs at least one of the following operations: when restoring the resolution of an image generated by pixels belonging to a second pixel group, applying a gain to data of pixels belonging to a first pixel group through a gain application unit, or filtering at least a portion of data of pixels belonging to the first pixel group through a filter application unit. An IR imaging device that interpolates pixel data at locations of pixels belonging to a first pixel group where data does not exist among pixel data of an image generated from pixels belonging to a second pixel group by applying a demosaicing algorithm.

11. In paragraph 7, The above plurality of pixel groups are composed of a first pixel group and a second pixel group, An IR imaging device in which the image restoration unit applies a gain equivalent to the difference in sensitivity between the first pixel group and the second pixel group to the data of pixels belonging to the first pixel group through a gain application unit when restoring the resolution of an image generated by pixels belonging to the second pixel group.

12. In paragraph 7, An IR imaging device, wherein the image restoration unit further includes a noise reduction processing unit that reduces noise in a plurality of restored images having different sensitivities by performing noise reduction processing on at least one pixel group data among the pixel groups.

Images