Video capturing device and electronic device including the same

Abstract

To provide a vide capturing device and an electronic device including the same.SOLUTION: A video capturing device includes a first image sensor having a first spatial resolution and capturing a first video based on a first wavelength band, a second image sensor having a second spatial resolution lower than the first spatial resolution and capturing a second video based on a second wavelength band wider than the first wavelength band, and a processor that actively uses relative position information between the first image sensor and the second image sensor to match the second video based on the first image, replace the hue components of the first video with the hue components of the second video to form the hue components of a third video, and combine the hue components of the first video and the hue components of the third image in an alpha blending manner to form the hue components of a fourth video.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image capture device that provides a wide color gamut image, and an electronic device including the same. [Background technology]

[0002] An image sensor is a device that receives light incident from a subject, photoelectrically converts the received light, and generates an electrical signal.

[0003] To represent color, the image sensor typically uses a color filter consisting of an array of filter elements that selectively transmit red, green, and blue light, and senses the amount of light transmitted by each filter element.

[0004] In such image acquisition, the wavelength band transmitted by the filter elements of the color filter is limited, so that the color gamut for expressing the subject is limited. Summary of the Invention [Problem to be solved by the invention]

[0005] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide an image acquisition device that provides an image with a wide color gamut, and an electronic device including the same. [Means for solving the problem]

[0006] According to one embodiment, an image acquisition device includes a first image sensor having a first spatial resolution and acquiring a first image based on a first wavelength band, a second image sensor having a second spatial resolution lower than the first spatial resolution and acquiring a second image based on a second wavelength band wider than the first wavelength band, and a processor that aligns the second image based on the first image using relative position information between the first image sensor and the second image sensor, replaces a hue component of the first image with a hue component of the second image to form a hue component of a third image, and combines the hue component of the first image and the hue component of the third image in an alpha blending manner to form a hue component of a fourth image.

[0007] The processor adds a value obtained by multiplying the hue component of the first image by an alpha blending coefficient α to a value obtained by multiplying the hue component of the third image by “1−α” to form the hue component of the fourth image, where the alpha blending coefficient α may have a value greater than or equal to 0 and less than or equal to 1.

[0008] The processor may calculate parameters for matching the first image and the second image based on at least one of relative position information, resolution, angle of view, and focal length of each of the first image sensor and the second image sensor.

[0009] The third image may have the first spatial resolution and a color gamut corresponding to the second image.

[0010] The processor may match hue statistics of the first image with hue statistics of the second image to form a hue component of the third image.

[0011] The hue statistic is also the average value for the surrounding pixels adjacent to the central pixel.

[0012] The hue statistic is also a standard deviation value for the surrounding pixels adjacent to the central pixel.

[0013] The processor may calculate the average value by weighting the surrounding pixels by the similarity between the central pixel and the surrounding pixels.

[0014] The processor may calculate the standard deviation value by weighting the similarity between the central pixel and the surrounding pixels as a square of the deviation.

[0015] The similarity may be calculated using luminance values ​​of the center pixel and the surrounding pixels as a feature vector.

[0016] The processor determines the alpha blending coefficient α value as a logical sum of a first coefficient and a second coefficient, where the first coefficient indicates information related to an occlusion region due to parallax between the first image and the second image, and the second coefficient indicates color error information of a hue component of the third image.

[0017] The processor may set a first coefficient associated with a pixel of interest of the first image to 1 if there is no pixel in the second image that corresponds to the pixel of interest of the first image, and may set the first coefficient to 0 if there is a pixel in the second image that corresponds to the pixel of interest of the first image.

[0018] The processor may determine a larger value of the first coefficient and the second coefficient as the alpha blending coefficient α.

[0019] The second coefficient is proportional to a color difference between the first image and the third image and may have a value between 0 and 1.

[0020] The second coefficient may have a value closer to 0 as the saturation of the first image increases, and may have a value closer to 1 as the saturation of the first image decreases.

[0021] The processor sets the second coefficient based on the difference in brightness values ​​between corresponding pixels of the first image and the second image, and may set the second coefficient closer to 1 when the difference in brightness values ​​is larger, and set the second coefficient closer to 0 when the difference in brightness values ​​is smaller.

[0022] The processor may separate the first image and the second image into a luminance component and a hue component, respectively, to form the third image and the fourth image.

[0023] The processor may extract edge features or corner features in the first image and the second image, and match at least one of the edge features or corner features between the first image and the second image, thereby aligning the first image and the second image.

[0024] An image acquisition control method according to one embodiment includes the steps of acquiring a first image and a second image from a first image sensor and a second image sensor, respectively; aligning the acquired first image and second image; replacing a hue component of the first image with a hue component of the second image to form a hue component of a third image; and combining the hue component of the first image and the hue component of the third image using an alpha blending method to form a hue component of a fourth image. [Brief description of the drawings]

[0025] [Figure 1] 1 is a block diagram illustrating a schematic structure of an image capture device according to an embodiment; [Diagram 2] 4 is a color coordinate system illustrating a color gamut of an image captured by an image capture device according to an embodiment; [Diagram 3] 1 is a conceptual diagram illustrating a schematic structure of an image capture device according to an embodiment; [Figure 4] 2 is a diagram illustrating a circuit configuration of a first image sensor and a second image sensor included in an image capturing device according to an embodiment. [Diagram 5]4 is a diagram illustrating a wavelength spectrum according to a first image sensor included in an image capturing device according to an embodiment. [Figure 6] 3 illustrates an exemplary pixel array of a first image sensor included in an image capturing device according to an embodiment. [Figure 7] 3 illustrates an exemplary pixel array of a first image sensor included in an image capturing device according to an embodiment. [Figure 8] 3 illustrates an exemplary pixel array of a first image sensor included in an image capturing device according to an embodiment. [Figure 9] 11 is a diagram illustrating a wavelength spectrum according to a second image sensor included in an image capturing device according to an embodiment. [Figure 10] 13 is a diagram illustrating an exemplary pixel array of a second image sensor included in another image capture device according to an embodiment. [Figure 11] 13 is a diagram illustrating an exemplary pixel array of a second image sensor included in another image capture device according to an embodiment. [Figure 12] 13 is a diagram illustrating an exemplary pixel array of a second image sensor included in another image capture device according to an embodiment. [Figure 13] 4 is a flowchart illustrating an image processing process of an image capture device according to an embodiment; [Figure 14] 1 is a block diagram showing a schematic structure of an electronic device according to an embodiment; [Figure 15] 15 is a block diagram illustrating a camera module provided in the electronic device of FIG. 14. [Figure 16A] 1A to 1C are diagrams illustrating various examples of an electronic device to which an image capturing device according to an embodiment is applied; [Figure 16B] 1A to 1C are diagrams illustrating various examples of an electronic device to which an image capturing device according to an embodiment is applied; [Figure 16C] 1A to 1C are diagrams illustrating various examples of an electronic device to which an image capturing device according to an embodiment is applied; [Figure 16D] 1A to 1C are diagrams illustrating various examples of an electronic device to which an image capturing device according to an embodiment is applied; [Figure 16E] 1A to 1C are diagrams illustrating various examples of an electronic device to which an image capturing device according to an embodiment is applied; [Figure 17A] 1A to 1C are diagrams illustrating various examples of an electronic device to which an image capturing device according to an embodiment is applied; [Figure 17B] 1A to 1C are diagrams illustrating various examples of an electronic device to which an image capturing device according to an embodiment is applied; [Figure 17C] 1A to 1C are diagrams illustrating various examples of an electronic device to which an image capturing device according to an embodiment is applied; [Figure 17D] 1A to 1C are diagrams illustrating various examples of an electronic device to which an image capturing device according to an embodiment is applied; [Figure 17E] 1A to 1C are diagrams illustrating various examples of an electronic device to which an image capturing device according to an embodiment is applied; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, the present embodiment will be described in detail with reference to the accompanying drawings. The described embodiment is merely exemplary, and various modifications are possible from such an embodiment. In the following drawings, the same reference numerals refer to the same components, and the size of each component in the drawings is exaggerated for the sake of clarity and convenience of description.

[0027] In the following description, the terms "upper" and "above" include not only what is directly above in contact with something, but also what is above without contacting something.

[0028] Terms such as first and second may be used to describe various components, but are used only to distinguish one component from another, and are not intended to limit the components to differing materials or structures.

[0029] The singular expression includes the plural expression unless otherwise clearly and distinctly meant in the context. Furthermore, when a part is described as "comprising" a certain element, it does not mean excluding other elements, but also means including other elements, unless otherwise specifically stated to the contrary.

[0030] Furthermore, terms such as "unit" and "module" used in the specification refer to a unit that processes at least one function or operation, and may be embodied in hardware or software, or a combination of hardware and software.

[0031] Use of the terms "said" and similar referents refers to both the singular and the plural.

[0032] The steps constituting the method may be performed in any suitable order unless there is an explicit reference that they must be performed in the order described. In addition, the use of all exemplary terms (e.g., "etc.", etc.) is merely for the purpose of explaining the technical idea in detail, and such terms do not limit the scope of the rights, except as limited by the claims.

[0033] FIG. 1 is a block diagram illustrating a schematic structure of an image capture device according to an embodiment, and FIG. 2 is a color coordinate system illustrating a color gamut of an image captured by the image capture device according to an embodiment.

[0034] The image acquisition device 1000 includes a first image sensor 100 that acquires a first image IMG1 based on M spectral bands, a second image sensor 200 that acquires a second image IMG2 based on N spectral bands, where N is greater than M, and a processor 500 that performs signal processing on the first image IMG1 and the second image IMG2 to form a third image IMG3 and combines the first image IMG1 and the third image IMG3 to form a fourth image IMG4.

[0035] The first image sensor 100 is a sensor used in a general RGB (red-green-blue) camera, and is also a complementary metal-oxide-semiconductor (CMOS) image sensor using a Bayer color filter array. The first image IMG1 acquired by the first image sensor 100 is also an RGB image based on red, green and blue colors, and may have a color gamut range acquired by a general RGB camera. For example, the first image IMG1 may have the s-RGB (standard RGB) color gamut, the BT.709 color gamut, the DCI-P3 color gamut, or the Adobe RGB color gamut. FIG. 2 exemplarily illustrates the BT.709 color gamut and the DCI-P3 color gamut.

[0036] The second image sensor 200 is a sensor that senses light of more wavelengths than the first image sensor 100. The second image sensor 200 may use, for example, 16 channels, 31 channels, or other number of channels. The bandwidth of each channel is set narrower than the R, G, and B bands, and the total bandwidth of all the channels is wider than the RGB bandwidth, i.e., the visible light bandwidth. For example, the bandwidth may be about 350 nm to 1,000 nm. The second image IMG2 acquired by the second image sensor 200 may be a hyperspectral image or a wavelength-based image in which a wavelength band wider than the RGB wavelength band, for example, an ultraviolet wavelength band or an infrared wavelength band, which is a wider wavelength band including the visible light band, is divided into 16 or more channels. The second image IMG2 may be an image acquired using all available channels of the second image sensor 200, or an image acquired by selecting a specific channel. The spatial resolution of the second image IMG2 is lower than the spatial resolution of the first image IMG1, although embodiments of the present disclosure are not limited thereto.

[0037] The first image sensor 100 and the second image sensor 200 may be formed on separate chips or on a single chip.

[0038] The first image IMG1 and the second image IMG2 acquired by the first image sensor 100 and the second image sensor 200 may be stored in the memory 300. The memory 300 may be a line memory that stores the first image IMG1 and the second image IMG2 in units of lines, or a frame buffer that stores the entire image. The memory 300 may be a static random access memory (SRAM) or a dynamic random access memory (DRAM). The memory 300 may be located outside the first image sensor 100 and the second image sensor 200, respectively, or may be integrated inside the first image sensor 100 and the second image sensor 200, respectively. When integrated inside the first image sensor 100 and the second image sensor 200, the memory 300 may be integrated together with a sensor circuit. In this case, the pixel unit and the circuit unit, which is the other part, and the memory 300 may be stacked and integrated into two stacks to be configured into one chip. Alternatively, depending on the configuration, 3D stacking having three layers of a pixel section, a circuit section, and a memory section is also possible.

[0039] The processor 500 uses the first image IMG1 as an input image, and the second image IMG2 as a reference image to form a third image IMG3. The third image IMG3 is an image having a spatial resolution corresponding to the first image IMG1 by the first image sensor 100 and a color gamut corresponding to the second image IMG2 by the second image sensor 200. The third image IMG3 is an image having the same spatial resolution as the first image IMG1 and an expanded color gamut, and may have a color gamut range equal to or greater than that of DCI-P3 in FIG. 2, for example.

[0040] The third image IMG3 may be obtained by replacing the hue components of the first image IMG1 with the hue components of the second image IMG2. To achieve this, the first image IMG1 and the second image IMG2 may be registered on a two-dimensional plane to grasp the correspondence between the pixels of the first image IMG1 and the pixels of the second image IMG2.

[0041] The processor 500 may combine the first image IMG1 and the third image IMG3 in an alpha blending manner to form a fourth image IMG4. The fourth image IMG4 may have the same spatial resolution as the first image IMG1 and an expanded color gamut, for example, a color gamut range equal to or greater than that of DCI-P3 in FIG. 2.

[0042] The image processing steps of the processor 500 are described in more detail below.

[0043] The processor 500 also includes an image acquiring unit 510, an image matching unit 520, a color transmitting unit 530, and an image synthesizing unit 540. For convenience of explanation, the image acquiring unit 510, the image matching unit 520, the color transmitting unit 530, and the image synthesizing unit 540 are divided according to the operation of the processor 500, but such division does not necessarily mean that the units are physically separated. The units correspond to any combination of hardware and / or software included in the processor 500, and may be physically the same as or different from each other.

[0044] The image acquiring unit 510 may perform basic image processing after acquiring images from the first image sensor 100 and the second image sensor 200 and before or after storing the images in the memory 300. For example, the image acquiring unit 510 may perform bad pixel correction, fixed pattern noise correction, crosstalk reduction, remosaicing, demosaicing, false color reduction, denoising, chromatic aberration correction, etc. It goes without saying that the image acquiring unit 510 may perform the same image processing on both the first image sensor 100 and the second image sensor 200 or may perform different image processing on each of the first image sensor 100 and the second image sensor 200.

[0045] The image acquiring unit 510 may apply demosaicing to interpolate raw images of the first image IMG1 and the second image IMG2, each having one channel, into an RGB image or a multi-channel spectral image. The image acquiring unit 510 may also convert the first image IMG1, which is an RGB image, and the second image IMG2, which is a multi-channel spectral image, into a three-channel XYZ color space, each through an image domain conversion process. The image acquiring unit 510 may normalize the luminance of the second image IMG2 based on the first image IMG1 in order to accurately transfer color information of the second image IMG2 to the first image IMG1.

[0046] The image registration unit 520 may register the first image IMG1 and the second image IMG2 using relative position information between the first image sensor 100 and the second image sensor 200. The image registration unit 520 may grasp the positional relationship between pixels of the images, taking into consideration the spatial resolution of the images acquired from the first image sensor 100 and the second image sensor 200, the field of view of the optical system used to acquire the images, the focal length, and the like. In this case, the image of one sensor may be used as a reference and the image of the other sensor may be overlayed thereon. For example, the first image IMG1 acquired by the first image sensor 100 may be used as a reference, and a pixel of the second image IMG2 corresponding to each pixel of the first image IMG1 may be obtained. To this end, the image registration unit 520 may perform scaling, translation, rotation, affine transform, perspective transform, and the like on the pixels of the second image IMG2.

[0047] The number of pixels in the second image IMG2 that correspond to a pixel in the first image IMG1 may be one or more. If there are several corresponding pixels, the pixel in the second image IMG2 that corresponds to the pixel in the first image IMG1 may be calculated through additive stacking for multiple pixels in the second image IMG2 according to their positions.

[0048] Furthermore, there is no pixel in the second image IMG2 that corresponds to a pixel in the first image IMG1. This is due to disparity between the first image sensor 100 and the second image sensor 200, and the image matching unit 520 may determine that a pixel is an occlusion area when there is no corresponding pixel. The image matching unit 520 may store information related to the pixel determined to be an occlusion area in the form of a map. Specifically, the image matching unit 520 may set the value of the pixel to 1 when there is no corresponding pixel, and set the value of the pixel to 0 when there is a corresponding pixel, thereby generating an occlusion map.

[0049] In order to improve the accuracy of the matching, the image matching unit 520 may perform matching in sub-pixel units. In the sub-pixel unit matching, the position of a pixel may be represented by a real number instead of an integer.

[0050] The image alignment unit 520 may also improve alignment efficiency by allowing the first image sensor 100 and the second image sensor 200 to focus on an object at the same position through focus control. In addition, the two sensors may have the same field of view, allowing image alignment to be performed quickly and accurately. For example, if the imaging optical systems for forming optical images on the first image sensor 100 and the second image sensor 200 have the same focal length and the same field of view, there is only a translation between the first image IMG1 and the second image IMG2, and related parameters may be calculated using the relative positions of the first image sensor 100 and the second image sensor 200 and the focal length of each optical system.

[0051] Before performing the alignment, aberrations included in the first image IMG1 and the second image IMG2 may be corrected. That is, the alignment may be performed after correcting the effects of distortion, geometric aberration, chromatic aberration, etc., caused by lenses included in the imaging optical system used to acquire the first image IMG1 and the second image IMG2.

[0052] The image matching unit 520 may extract edge feature information in the images and match the features between the two images using the edge feature information. Since image matching misalignment occurs in the boundary area of ​​the object, edge information may be extracted and matching may be performed to align the edges between the two images and to prevent distortion in the boundary area. In addition to edges, image matching may be performed using other image features such as corner points.

[0053] The hue transfer unit 530 uses the first image IMG1 as an input image and the second image IMG2 as a reference image, and changes the hue of the first image IMG1 to match the hue of the second image IMG2 to form a third image IMG3. Specifically, the hue transfer unit 530 matches the hue statistics of the first image IMG1 with the hue statistics of the second image IMG2 to form a hue component of the third image IMG3. The hue statistics may be based on an average value of the surrounding pixels adjacent to the central pixel and / or a standard deviation value of the surrounding pixels adjacent to the central pixel.

[0054] The color transfer unit 530 may separate the first image IMG1 and the second image IMG2 into a luminance component and a hue component, respectively, to form the third image IMG3. For example, in the CIE LAB color space in which the luminance component and the hue component are highly independent, the color transfer unit 530 may change the hue of the first image IMG1 to match the hue of the second image IMG2 to form the hue component of the third image IMG3, and may use the luminance component of the first image IMG1 as is to form the luminance component of the third image IMG3.

[0055] The color transfer unit 530 forms the third image IMG3 using a color transfer function expressed by the following Equations 1 and 2.

[0056] First, the luminance component of the third image IMG3

number

number

number

[0057] Next, the hue component of the third image IMG3 is the hue component at the center pixel p

number

[0058] Specifically, the hue component of the third image IMG3

number

[0059]

number

[0060] statistics {μ c (p),STD C (p)} is calculated according to the following Equations 3 and 4.

[0061]

number

[0062]

number

[0063] Where:

number

number

number

number

number

number

number

[0064] The mean (μ) and standard deviation (STD) used in the transfer function (Equation 2) that forms the hue component of the third image IMG3 are determined by the similarity between the center pixel p and the surrounding pixels q.

number

[0065] Indicates the similarity between the center pixel p and the surrounding pixel q

number

number

number

number

number

[0066] The image synthesis unit 540 synthesizes the color components of the first image IMG1 and the third image IMG3 using an alpha blending method to form the color components of the fourth image IMG4.

[0067] Luminance component of the fourth image IMG4

number

number

number

[0068] Hue component of the fourth image IMG4

number

number

[0069] The image synthesis unit 540 adds a value obtained by multiplying the hue component of the first image IMG1 by the alpha blending coefficient α and a value obtained by multiplying the hue component of the third image IMG3 by “1−α” to obtain a hue component of the fourth image IMG4.

number

[0070] The image synthesizer 540 calculates the first coefficient (α 1 ) and the second coefficient (α 2 Specifically, the image synthesizing unit 540 determines the alpha blending coefficient α value by performing a logical sum with the first coefficient (α 1 ) and the second coefficient (α 2 ) may be set as an alpha blending coefficient α. This may be expressed as Equation 7 below.

number

[0071] The first coefficient (α 1) is a parameter related to the occlusion area due to the parallax between the first image IMG1 and the second image IMG2.

[0072] The image synthesis unit 540 may receive information OC related to the occlusion region from the image matching unit 520. As described above, if there is no pixel in the second image IMG2 that corresponds to the first image IMG1, the image matching unit 520 may determine that the pixel is an occlusion region. The image matching unit 520 may store information related to the pixel determined to be the occlusion region in the form of a map. Specifically, the image matching unit 520 may set the value of the pixel to 1 if there is no corresponding pixel, and set the value of the pixel to 0 if there is a corresponding pixel, thereby generating an occlusion map. As a result, the first coefficient (α 1 ) may have a value of 1 if it falls within the occlusion area (if there are no corresponding pixels between the first and second images), and may have a value of 0 if it does not fall within the occlusion area (if there are corresponding pixels between the first and second images).

[0073] The second coefficient (α 2 ) is a parameter related to false color information of the third image IMG3. The second image sensor 200 senses relatively more wavelength channels than the first image sensor 100. Therefore, the distance between pixels of the same wavelength channel in the second image sensor 200 is longer than the distance between pixels of the same wavelength channel in the first image sensor 100 (see, for example, FIGS. 6 and 10). As a result, the second image sensor 200 may generate incorrect hue information, i.e., false color information, during demosaicing.

[0074] The image synthesizer 540 calculates the second coefficient (α 2 ) may be set to have a value between 0 and 1, but is proportional to the color difference between the first image IMG1 and the third image IMG3.

[0075] In addition, the image synthesizer 540 calculates the second coefficient (α 2) may be determined based on the difference in luminance values ​​between corresponding pixels in the first image IMG1 and the second image IMG2. This is based on the empirical fact that false color information is particularly noticeable in an area where the difference in luminance values ​​between corresponding pixels in the first image IMG1 and the second image IMG2 is large. Therefore, the image synthesis unit 540 may determine a second coefficient (α 2 ) may be set to a value between 0 and 1. Specifically, the image synthesizer 540 may set the second coefficient (α 2 ) value is set to be close to 1, and the smaller the difference, the better the second coefficient (α 2 ) value may be set close to 0.

[0076] In addition, the image synthesizer 540 calculates the second coefficient (α 2 ) may be set so that the higher the saturation of the first image IMG1, the closer to 0 the coefficient, and the lower the saturation of the first image IMG1, the closer to 1 the coefficient. This is based on the empirical fact that erroneous color information is particularly noticeable in an achromatic color region. 2 A specific method for setting ) is shown in Equation 8 below.

[0077]

number

[0078] The clip function checks whether the input value is between 0 and 1, and outputs 0 if the input value is less than 0, and outputs 1 if the input value is greater than 1. As a result, the clip function is 2 ) is set to have a value between 0 and 1. ΔC(p) is a color difference value between the first image IMG1 and the third image IMG3. S1(p) is a saturation value that indicates the vividness of the hue component of the first image IMG1. k 1 and k 2are heuristic hyperparameters, respectively.

[0079] In one embodiment, the image synthesizer 540 synthesizes the luminance component of the fourth image IMG4.

number

number

[0080] As described above, the image acquisition device 1000 according to one embodiment uses the first image IMG1 captured by the first image sensor 100 having high spatial resolution as an input image, uses the second image IMG2 captured by the second image sensor 200 having a wide color gamut as a reference image, and transmits hue information of the second image IMG2 to the first image IMG1 to form a final image having the spatial resolution of the first image IMG1 and the wide color gamut of the second image IMG2.

[0081] The hue information transfer is implemented by matching the hue statistics of the first image IMG1 with the hue statistics of the second image IMG2.

[0082] In addition, the first image (IMG1) and the intermediate image (IMG3) are synthesized by alpha blending to form a final image (IMG4). The alpha blending coefficient α is set to the first coefficient (α 1 ) and a second coefficient (α2), which is a parameter related to erroneous color information of the intermediate image (third image), to correct the erroneous hue information transmission and form the final image (fourth image IMG4).

[0083] FIG. 3 is a conceptual diagram illustrating a schematic structure of an image capturing device according to an embodiment, and FIG. 4 illustrates a circuit configuration of a first image sensor and a second image sensor included in the image capturing device according to an embodiment.

[0084] The image acquiring device 1000 includes a first image sensor 100 for acquiring a first image IMG1 based on a first wavelength band, a second image sensor 200 for acquiring a second image IMG2 based on a second wavelength band wider than the first wavelength band, and a processor 500 for processing the first image IMG1 and the second image IMG2 to form a fourth image IMG4. The image acquiring device 1000 further includes a memory 300 in which data related to the first image IMG1 and the second image IMG2 are stored.

[0085] The image acquisition device 1000 also includes a first imaging optical system 190 that forms an optical image of the object OBJ on the first image sensor 100, and a second imaging optical system 290 that forms an optical image of the object OBJ on the second image sensor 200. Although the first imaging optical system 190 and the second imaging optical system 290 are illustrated as including one lens each, this is for illustrative purposes only and is not limited thereto. The first imaging optical system 190 and the second imaging optical system 290 may be configured to have the same focal length and the same angle of view. If configured to have the same focal length and the same angle of view, the process of aligning the first image IMG1 and the second image IMG2 to form the third image IMG3 and the fourth image IMG4 is further facilitated. However, the embodiment of the present disclosure is not limited thereto.

[0086] The first image sensor 100 includes a first pixel array PA1, which includes a first sensor layer 110 in which a plurality of first sensing elements are arrayed, and a color filter 120 disposed on the first sensor layer 110. The color filter 120 also includes red filters, green filters, and blue filters arranged in an alternating manner. A first microlens array 130 may be disposed on the first pixel array PA1. Various examples of pixel arrangements applied to the first pixel array PA1 are described with reference to FIGS. 5 to 8.

[0087] The second image sensor includes a second pixel array PA2, which includes a second sensor layer 210 in which a plurality of second sensing elements are arrayed, and a spectral filter 220 disposed on the second sensor layer 210. The spectral filter 220 includes a plurality of filter groups, each of which includes a plurality of unit filters having different transmission wavelength bands. The spectral filter 220 may be configured to filter a wavelength band wider than that of the color filter 120, for example, a wavelength band from an ultraviolet wavelength range to an infrared wavelength range, by dividing the wavelength band into smaller parts than that of the color filter 120. A second microlens array 230 may be disposed on the second pixel array PA2. Examples of pixel arrangements applied to the second pixel array PA2 are described in FIG. 10 to FIG. 12.

[0088] The first sensor layer 110 and the second sensor layer 210 may include, but are not limited to, charge-coupled device (CCD) sensors or complementary metal-oxide-semiconductor (CMOS) sensors.

[0089] The first pixel array PA1 and the second pixel array PA2 may be disposed on the same circuit substrate SU so as to be spaced apart from each other in the horizontal, for example, X-direction.

[0090] The circuit board SU may include a first circuit element for processing a signal from the first sensor layer 110, and a second circuit element for processing a signal from the second sensor layer 210. However, the present invention is not limited thereto, and the first circuit element and the second circuit element may be provided on separate boards.

[0091] The memory 300 storing data related to the first image IMG1 and the second image IMG2 is illustrated separately from the circuit board SU, but this is merely an example and may be disposed on the same layer as the circuit elements or on a separate layer within the circuit board SU. The memory 300 may be a line memory that stores an image on a line-by-line basis, or a frame buffer that stores the entire image. The memory 300 may be an SRAM or a DRAM.

[0092] Various circuit elements required for the image capture device 1000 may be integrated and arranged on the circuit board SU. For example, a logic layer including various analog circuits and digital circuits may be provided, and a memory layer in which data is stored may be provided. The logic layer and the memory layer may be provided on different layers or on the same layer.

[0093] Referring to FIG. 4, a row decoder 102, an output circuit 103, and a timing controller (TC) 101 are connected to the first pixel array PA1. The row decoder 102 selects one row of the first pixel array PA1 in response to a row address signal output from the timing controller 101. The output circuit 103 outputs a photodetection signal in units of columns from a plurality of pixels arranged along the selected row. To this end, the output circuit 103 may include a column decoder and an analog-to-digital converter (ADC). For example, the output circuit 103 may include a plurality of analog-to-digital converters (ADCs) arranged for each column between the column decoder and the first pixel array PA1, or one analog-to-digital converter (ADC) arranged at an output end of the column decoder. The timing controller 101, the row decoder 102, and the output circuit 103 may be implemented on one chip or on separate chips. At least some of the illustrated circuit elements may be included on a circuit board SU of FIG. 3. A processor for processing the first image IMG1 output via the output circuit 103 may be implemented on one chip together with the timing controller 101, the row decoder 102 and the output circuit 103.

[0094] The second pixel array PA2 is also connected to a row decoder 202, an output circuit 203, and a timing controller (TC) 201, and signals from the second pixel array PA2 may be processed in a similar manner as described above. In addition, a processor for processing the second image IMG2 outputted through the output circuit 203 may be implemented on one chip together with the timing controller 201, the row decoder 202, and the output circuit 203.

[0095] Although the first pixel array PA1 and the second pixel array PA2 are illustrated as having different pixel sizes and numbers, this is merely an example and the embodiments of the present disclosure are not limited thereto.

[0096] When two different types of sensors are operated, timing control is required due to the different resolutions and output speeds, and the size of the area required for image matching. For example, when an image sequence is read based on the first image sensor 100, the image sequence of the second image sensor 200 corresponding to that area may already be stored in a buffer or must be newly read. It is necessary to correctly calculate and read such timing. Alternatively, the operations of the first image sensor 100 and the second image sensor 200 may be synchronized using the same synchronization signal. For example, a timing controller 400 may be further provided and may transmit a synchronization signal sync. to the first image sensor 100 and the second image sensor 200.

[0097] FIG. 5 illustrates a wavelength spectrum according to a first image sensor included in an image capturing device according to an embodiment, and FIGS. 6 to 8 illustrate exemplary pixel arrays of the first image sensor included in an image capturing device according to an embodiment.

[0098] Referring to FIG. 5, the first image sensor 100 can detect red light R in a wavelength band having a central wavelength of about 650 nm, detect green light G in a wavelength band having a central wavelength of about 540 nm, and detect blue light B in a wavelength band having a central wavelength of about 440 nm.

[0099] 6, in the color filter 120 provided in the first pixel array PA1, filters for filtering wavelength bands of red R, green G, and blue B are arranged in a Bayer pattern. That is, one unit pixel includes sub-pixels arranged in a 2×2 array, and the plurality of unit pixels are arranged two-dimensionally in a repeated manner. Red and green filters are arranged in the first row of the unit pixels, and green and blue filters are arranged in the second row. The pixel arrangement may be in other ways than the Bayer pattern.

[0100] For example, referring to FIG. 7, a CYGM type arrangement in which a magenta pixel M, a cyan pixel C, a yellow pixel Y and a green pixel G constitute one unit pixel is also possible.

[0101] 8, an RGBW type arrangement in which a green pixel G, a red pixel R, a blue pixel B, and a white pixel W constitute one unit pixel is also possible. Although not shown, the unit pixel may have a 3×2 array shape. In addition, the pixels of the first pixel array PA1 may be arranged in various ways according to the color characteristics of the first image sensor 100.

[0102] FIG. 9 illustrates a wavelength spectrum according to a second image sensor included in an image capturing device according to an embodiment, and FIGS. 10 to 12 illustrate exemplary pixel arrangements of a second image sensor of another image capturing device according to an embodiment.

[0103] 10, the spectral filter 220 included in the second pixel array PA2 includes a plurality of filter groups 221 arranged in a two-dimensional form. Here, each filter group 221 includes 16 unit filters F1 to F16 arranged in a 4×4 array form.

[0104] The first and second unit filters F1 and F2 may have central wavelengths UV1 and UV2 in the ultraviolet region, the third to fifth unit filters F3 to F5 may have central wavelengths B1 to B3 in the blue region, the sixth to eleventh unit filters F6 to F11 may have central wavelengths G1 to G6 in the green region, the twelfth to fourteenth unit filters F12 to F14 may have central wavelengths R1 to R3 in the red region, and the fifteenth and sixteenth unit filters F15 and F16 may have central wavelengths NIR1 and NIR2 in the near infrared region.

[0105] 11 is a plan view of another example of a filter group 222 included in the spectral filter 220. Referring to FIG. 11, the filter group 222 includes nine unit filters F1 to F9 arranged in a 3×3 array. Here, the first and second unit filters F1 and F2 may have central wavelengths UV1 and UV2 in the ultraviolet region, and the fourth, fifth, and seventh unit filters F4, F5, and F7 may have central wavelengths B1 to B3 in the blue region. The third and sixth unit filters F3 and F6 may have central wavelengths G1 and G2 in the green region, and the eighth and ninth unit filters F8 and F9 may have central wavelengths R1 and R2 in the red region.

[0106] Fig. 12 is a plan view of another example of one of the filter groups 223 included in the spectral filter 220. Referring to Fig. 12, the filter group 223 also includes 25 unit filters F1 to F25 arranged in a 5 x 5 array form. Here, the first unit filter F1 to the third unit filter F3 may have central wavelengths UV1 to UV3 in the ultraviolet light region, and the sixth unit filter F6, the seventh unit filter F7, the eighth unit filter F8, the eleventh unit filter F11, and the twelfth unit filter F12 may have central wavelengths B1 to B5 in the blue light region. The fourth unit filter F4, the fifth unit filter F5 and the ninth unit filter F9 may have central wavelengths G1 to G3 in the green light region, the tenth unit filter F10, the thirteenth unit filter F13, the fourteenth unit filter F14, the fifteenth unit filter F15, the eighteenth unit filter F18 and the nineteenth unit filter F19 may have central wavelengths R1 to R6 in the red light region, and the twentieth unit filter F20, the twenty-third unit filter F23, the twenty-fourth unit filter F24 and the twenty-fifth unit filter F25 may have central wavelengths NIR1 to NIR4 in the near infrared region.

[0107] The above-mentioned unit filter included in the spectral filter 220 has a resonant structure having two reflectors, and the transmitted wavelength band can be determined by the characteristics of the resonant structure. The transmitted wavelength band can be adjusted by the material of the reflector, the material of the dielectric material in the cavity, and the cavity thickness. In addition, a structure using a diffraction grating, a structure using a distributed Bragg reflector (DBR), etc. can be applied to the unit filter.

[0108] In addition, the pixels of the second pixel array PA2 may be arranged in various ways according to the color characteristics of the second image sensor 200.

[0109] FIG. 13 is a flow chart illustrating an image processing process of an image capture device according to an embodiment.

[0110] 1 and 13, a first image IMG1 is acquired from a first image sensor 100, and a second image IMG2 is acquired from a second image sensor 200 (S1410).

[0111] Basic image processing may be performed on the first image IMG1 and / or the second image IMG2 before or after they are stored in the memory 300. For example, bad pixel correction, fixed pattern noise correction, crosstalk reduction, remosaicing, demosaicing, false color reduction, denoising, chromatic aberration correction, etc. may be performed (S1410).

[0112] The first image IMG1 and the second image IMG2 are matched using relative position information between the first image sensor 100 and the second image sensor 200 (S1430). The positional relationship between each pixel of the image is grasped taking into consideration the spatial resolution of the images acquired from the first image sensor 100 and the second image sensor 200, the field of view of the optical system used to acquire the images, the focal length, etc. In this case, the image of one sensor can be used as a reference and the image of the other sensor can be overlayed thereon. For example, the first image IMG1 acquired by the first image sensor 100 can be used as a reference and the corresponding pixel of the second image IMG2 can be obtained for each pixel of the first image IMG1. To this end, the pixels of the second image IMG2 can be subjected to scaling, translation, rotation, affine transform, perspective transform, etc.

[0113] The hue component of the first image IMG1 is replaced with the hue component of the second image IMG2 to form the hue component of the third image IMG3 (S1440). The first image IMG1 is used as an input image, the second image IMG2 is used as a reference image, and the hue of the first image IMG1 is changed to match the hue of the second image IMG2 to form the third image IMG3. Specifically, the hue transfer unit 530 matches the hue statistics of the first image IMG1 with the hue statistics of the second image IMG2 to form the hue component of the third image IMG3. The hue statistics may be based on an average value of the surrounding pixels adjacent to the central pixel and / or a standard deviation value of the surrounding pixels adjacent to the central pixel.

[0114] The color components of the first image IMG1 and the third image IMG3 are mixed using alpha blending to form the color components of the fourth image IMG4 (S1450).

[0115] The luminance component of the fourth image IMG4 uses the luminance component value of the first image IMG1 as is, similar to the luminance component of the third image IMG3.

[0116] The hue component of the fourth image IMG4 is formed by adding a value obtained by multiplying the hue component of the first image IMG1 by the alpha blending coefficient α and a value obtained by multiplying the hue component of the third image IMG3 by "1-α". The alpha blending coefficient α may have a value between 0 and 1. The first coefficient (α 1 ) and the second coefficient (α 2 In one embodiment, the alpha blending factor α value is determined as a logical OR of the first factor (α 1 ) and the second coefficient (α 2 ), the larger value may be determined as the alpha blending coefficient α.

[0117] The first coefficient (α 1 ) is a parameter related to the occlusion area due to the parallax between the first image IMG1 and the second image IMG2, and the second coefficient (α 2 ) is a parameter related to false color information of the third image IMG3.

[0118] The image capture device 1000 may be employed in various high performance optical devices or high performance electronic devices, such as, but not limited to, smart phones, mobile phones, handphones, personal digital assistants (PDAs), laptops, personal computers (PCs), various portable devices, home appliances, security cameras, medical cameras, automobiles, Internet of things (IoT) devices, and other mobile or non-mobile computing devices.

[0119] In addition to the image capturing device 1000, the electronic device may further include a processor, for example, an application processor (AP), for controlling an image sensor provided therein, and may run an operating system (OS) or application programs via the processor, control a number of hardware or software components, and perform various data processing and calculations. The processor may further include a graphics processing unit (GPU) and / or an image signal processor. When the processor includes an image signal processor, an image (or video) captured by the image sensor may be stored and / or output using the processor.

[0120] Fig. 14 is a block diagram showing a schematic structure of an electronic device according to an embodiment. Referring to Fig. 14, in a network environment ED00, an electronic device ED01 may communicate with another electronic device ED02 via a first network ED98 (such as a short-range wireless communication network) or may communicate with another electronic device ED04 and / or a server ED08 via a second network ED99 (such as a long-range wireless communication network). The electronic device ED01 may communicate with the electronic device ED04 via the server ED08. The electronic device ED01 may also include a processor ED20, a memory ED30, an input device ED50, an audio output device ED55, a display device ED60, an audio module ED70, a sensor module ED76, an interface ED77, a haptic module ED79, a camera module ED80, a power management module ED88, a battery ED89, a communication module ED90, a subscriber identity module ED96, and / or an antenna module ED97. In the electronic device ED01, some of the components (such as the display device ED60) may be omitted, or other components may be added. Some of the components may be embodied as a single integrated circuit. For example, the sensor module ED76 (such as a fingerprint sensor, an iris sensor, or an illuminance sensor) may be embedded in the display device ED60 (such as a display). In addition, if the image sensor 1000 includes a spectroscopic function, some functions of the sensor module (such as a color sensor or an illuminance sensor) may be embodied as the image sensor 1000 itself, rather than as a separate sensor module.

[0121] The processor ED20 may execute software (such as the program ED40) and control one or more other components (such as hardware components, software components, etc.) of the electronic device ED01 coupled to the processor ED20 to perform various data processing or calculations. As part of the data processing or calculations, the processor ED20 may load instructions and / or data received from other components (such as the sensor module ED76, the communication module ED90, etc.) into the volatile memory ED32, process the instructions and / or data stored in the volatile memory ED32, and store the resulting data in the non-volatile memory ED34. The processor ED20 may also include a main processor ED21 (such as a central processing unit, an application processor, etc.) and an auxiliary processor ED23 (such as a graphics processing unit, an image signal processor, a sensor hub processor, a communication processor, etc.) that may operate independently of or in conjunction with the main processor ED21. The auxiliary processor ED23 may use less power than the main processor ED21 and perform specialized functions.

[0122] The auxiliary processor ED23 may take the place of the main processor ED21 while the main processor ED21 is in an inactive state (sleep state), or may control functions and / or states related to some of the components of the electronic device ED01 (such as the display device ED60, the sensor module ED76, and the communication module ED90) together with the main processor ED21 while the main processor ED21 is in an active state (application execution state). The auxiliary processor ED23 (such as an image signal processor or a communication processor) may be embodied as a part of other components (such as the camera module ED80 and the communication module ED90) that are functionally related to it.

[0123] The memory ED30 may store various data required by the components of the electronic device ED01 (processor ED20, sensor module ED76, etc.). The data may include, for example, software (program ED40, etc.) and input and / or output data for instructions associated therewith. The memory ED30 may include a volatile memory ED32 and / or a non-volatile memory ED34. The non-volatile memory ED34 may include an internal memory ED36 fixedly mounted in the electronic device ED01 and an external memory ED38 that is removable.

[0124] The program ED40 is stored as software in the memory ED30 and may also include an operating system (OS) ED42, a middleware ED44 and / or an application ED46.

[0125] The input device ED50 may receive instructions and / or data from outside (such as a user) of the electronic device ED01 for use by components (such as the processor ED20) of the electronic device ED01. The input device ED50 may also include a microphone, a mouse, a keyboard, and / or a digital pen (such as a stylus pen).

[0126] The audio output device ED55 may output an audio signal to the outside of the electronic device ED01. The audio output device ED55 may also include a speaker and / or a receiver. The speaker may be used for general purposes such as multimedia playback or recording playback, and the receiver may be used to receive an incoming call. The receiver may be combined as part of the speaker or may be embodied as a separate, independent device.

[0127] The display device ED60 may visually provide information external to the electronic device ED01. The display device ED60 may also include a display, a holographic device or a projector, and control circuitry for controlling the device. The display device ED60 may also include touch circuitry configured to sense a touch and / or sensor circuitry (such as a pressure sensor) configured to measure the strength of a force caused by the touch.

[0128] The audio module ED70 can convert sound into an electrical signal or vice versa, and can obtain sound via the input device ED50 or output sound via the sound output device ED55 and / or speakers and / or headphones of other electronic devices (such as the electronic device ED02) directly or wirelessly coupled to the electronic device ED01.

[0129] The sensor module ED76 may sense an operating state (power, temperature, etc.) of the electronic device ED01 or an external environmental state (user state, etc.) and generate an electrical signal and / or a data value corresponding to the sensed state. The sensor module ED76 may also include a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, and / or an illuminance sensor.

[0130] The interface ED77 may support one or more specified protocols that may be used for the electronic device ED01 to be directly or wirelessly coupled to other electronic devices (such as the electronic device ED02). The interface ED77 may also include a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, and / or an audio interface.

[0131] The connection terminal ED78 also includes a connector that allows the electronic device ED01 to be physically connected to another electronic device (such as the electronic device ED02). The connection terminal ED78 also includes an HDMI connector, a USB connector, an SD card connector, and / or an audio connector (such as a headphone connector).

[0132] The haptic module ED79 may convert electrical signals into mechanical stimuli (vibrations, movements, etc.) or electrical stimuli that the user may perceive via touch or kinesthetic sensations. The haptic module ED79 may also include motors, piezoelectric elements, and / or electrical stimulators.

[0133] The camera module ED80 may capture still and video images. The camera module ED80 may include the image capture device 1000 described above, and may further include a lens assembly, an image signal processor, and / or a flash. The lens assembly included in the camera module ED80 may collect light emitted from a subject from which an image is to be captured.

[0134] The power management module ED88 may manage the power supplied to the electronic device ED01. The power management module ED88 may be embodied as part of a power management integrated circuit (PMIC).

[0135] The battery ED89 may provide power to the components of the electronic device ED01. The battery ED89 may include a non-rechargeable primary battery, a rechargeable secondary battery, and / or a fuel cell.

[0136] The communication module ED90 may support the establishment of a direct (wired) communication channel and / or a wireless communication channel between the electronic device ED01 and other electronic devices (such as the electronic device ED02, the electronic device ED04, the server ED08, etc.) and the performance of communication through the established communication channel. The communication module ED90 may also include one or more communication processors that operate independently of the processor ED20 (such as an application processor) and support the direct communication and / or the wireless communication. The communication module ED90 may also include a wireless communication module ED92 (such as a cellular communication module, a short-range wireless communication module, a GNSS (global navigation satellite system, etc.) communication module) and / or a wired communication module ED94 (such as a LAN (local area network) communication module, a power line communication module, etc.). Among these communication modules, the communication module may communicate with other electronic devices via a first network ED98 (a short-range communication network such as Bluetooth, Wi-Fi Direct (WFD) or infrared data association (IrDA)) or a second network ED99 (a long-range communication network such as a cellular network, the Internet or a computer network (LAN (local area network), WAN (wide area network), etc.)). Such various communication modules may be integrated into one component (such as a single chip) or embodied by multiple components (multiple chips) that are separate from each other. The wireless communication module ED92 may identify and authenticate the electronic device ED01 in a communication network such as the first network ED98 and / or the second network ED99 using subscriber information (such as an international mobile subscriber identity (IMSI)) stored in the subscriber identity module ED96.

[0137] The antenna module ED97 may transmit signals and / or power to or receive signals from the outside (such as other electronic devices). The antenna may include a radiator formed by a conductive pattern formed on a substrate (such as a printed circuit board (PCB)). The antenna module ED97 may include one or more antennas. When multiple antennas are included, the communication module ED90 may select an antenna from the multiple antennas that is suitable for a communication method used in a communication network such as the first network ED98 and / or the second network ED99. Signals and / or power may be transmitted or received between the communication module ED90 and other electronic devices via the selected antenna. Components other than antennas (such as a radio frequency integrated circuit (RFIC)) may also be included as part of the antenna module ED97.

[0138] Some of the components may be connected to each other via a peripheral communication method (such as a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or Mobile Industry Processor Interface (MIPI)) and exchange signals (such as commands and data) with each other.

[0139] Commands or data may be transmitted or received between the electronic device ED01 and an external electronic device ED04 via a server ED08 connected to a second network ED99. The other electronic devices ED02, ED04 may be the same or different types of devices as the electronic device ED01. All or part of the operations performed by the electronic device ED01 may be performed by one or more of the other electronic devices ED02, ED04, ED08. For example, when the electronic device ED01 needs to perform a certain function or service, instead of performing the function or service itself, it may request one or more other electronic devices to perform the function or service in whole or in part. The one or more other electronic devices that receive the request may perform an additional function or service related to the request and transmit the results of the execution to the electronic device ED01. For this purpose, cloud computing, distributed computing and / or client-server computing technologies may be used.

[0140] Fig. 15 is a block diagram illustrating a camera module ED80 included in the electronic device of Fig. 14. The camera module ED80 may include the image capturing device 1000 described above or may have a structure modified therefrom. Referring to Fig. 15, the camera module ED80 may include a lens assembly CM10, a flash CM20, an image sensor CM30, an image stabilizer CM40, a memory CM50 (such as a buffer memory), and / or an image signal processor CM60.

[0141] The image sensor CM30 also includes the first image sensor 100 and the second image sensor 200 provided in the image capturing device 1000. The first image sensor 100 and the second image sensor 200 can capture an image corresponding to an object by converting light emitted or reflected from the object and transmitted through the lens assembly CM10 into an electrical signal. The first image sensor 100 can capture an RGB image, and the second image sensor 200 can capture a hyperspectral image in the ultraviolet wavelength range to the infrared wavelength range.

[0142] The image sensor CM30 may further include one or more sensors selected from image sensors with different attributes, such as an RGB sensor, a BW (black and white) sensor, an IR (infrared) sensor, or a UV (ultraviolet) sensor, in addition to the first image sensor 100 and the second image sensor 200. Each sensor included in the image sensor CM30 may be embodied as a CCD sensor and / or a CMOS sensor.

[0143] The lens assembly CM10 may collect light emitted from a subject to be imaged. The camera module ED80 may include multiple lens assemblies CM10, in which case the camera module ED80 may be a dual camera, a 360° camera, or a spherical camera. Some of the multiple lens assemblies CM10 may have the same lens attributes (angle of view, focal length, autofocus, F number, optical zoom, etc.) or may have different lens attributes. The lens assembly CM10 may include a wide-angle lens or a telephoto lens.

[0144] The lens assembly CM10 may be configured and / or focus controlled such that two image sensors included in the image sensor CM30 form optical images of a co-located object.

[0145] The flash CM 20 may emit light that is used to enhance the light emitted or reflected from a subject, and may also include one or more light-emitting diodes (such as RGB (red-green-blue) LEDs (light-emitting diodes), white LEDs, IR LEDs, UV LEDs, etc.) and / or xenon lamps.

[0146] The image stabilizer CM40 responds to the movement of the camera module ED80 or the electronic device ED01 including the camera module ED80, and moves one or more lenses included in the lens assembly CM10 or the image sensor CM30 in a specific direction, or controls the operating characteristics of the image sensor CM30 (such as adjusting the read-out timing) so that negative effects due to the movement are compensated for. The image stabilizer CM40 may sense the movement of the camera module ED80 or the electronic device ED01 using a gyro sensor (not shown) or an acceleration sensor (not shown) arranged inside or outside the camera module ED80. The image stabilizer CM40 may also be embodied optically.

[0147] The memory CM50 may store some or all of the data of an image acquired via the image sensor 1000 for the next image processing operation. For example, when multiple images are acquired at high speed, the acquired original data (Bayer-patterned data, high-resolution data, etc.) may be stored in the memory CM50 and used to display only the low-resolution image and then transmit the original data of the selected (e.g., user-selected) image to the image signal processor CM60. The memory CM50 may be integrated with the memory ED30 of the electronic device ED01 or may be configured as a separate memory operated independently.

[0148] The image signal processor CM60 may perform image processing on an image acquired through the image sensor CM30 or image data stored in the memory CM50. As described with reference to Figures 1 to 13, the image signal processor CM60 may process a first image (e.g., an RGB image) and a second image (e.g., a hyperspectral image) acquired by two image sensors included in the image sensor CM30 to form a third image with an expanded color gamut. The configuration of the processor 500 for this purpose is also included in the image signal processor CM60.

[0149] In addition to the above, image processing may also include depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, and / or image compensation (noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, softening, etc.). The image signal processor CM60 may perform control (exposure time control or readout timing control, etc.) for components (such as the image sensor CM30) included in the camera module ED80. The image processed by the image signal processor CM60 may be stored back in the memory CM50 for further processing or provided to external components (such as the memory ED30, the display device ED60, the electronic device ED02, the electronic device ED04, the server ED08, etc.) of the camera module ED80. The image signal processor CM60 may be integrated into the processor ED20 or configured as a separate processor operated independently of the processor ED20. If the image signal processor CM60 is configured as a processor separate from the processor ED20, the image processed by the image signal processor CM60 may be displayed via the display device ED60 after undergoing additional image processing by the processor ED20.

[0150] The electronic device ED01 may also include multiple camera modules ED80 each having different attributes or functions. In such a case, one of the multiple camera modules ED80 may be a wide-angle camera and another may be a telephoto camera. Similarly, one of the multiple camera modules ED80 may be a front camera and another may be a rear camera.

[0151] The image acquisition device 1000 according to the present embodiment may be applied to a mobile phone or smartphone 5100m shown in FIG. 16A, a tablet or smart tablet 5200 shown in FIG. 16B, a digital camera or camcorder 5300 shown in FIG. 16C, a notebook computer 5400 shown in FIG. 16D, or a television or smart television 5500 shown in FIG. 16E. For example, the smartphone 5100m or the smart tablet 5200 may include a plurality of high-resolution cameras each equipped with a high-resolution image sensor. The high-resolution cameras may be used to extract depth information of an object in an image, adjust out-focusing of the image, or automatically identify an object in an image.

[0152] In addition, the image acquisition device 1000 may be applied to a smart refrigerator 5600 shown in FIG. 17A, a security camera 5700 shown in FIG. 17B, a robot 5800 shown in FIG. 17C, a medical camera 5900 shown in FIG. 17D, and the like. For example, the smart refrigerator 5600 may automatically recognize food and drink in the refrigerator using the image acquisition device 1000, and may notify a user through a smartphone of the presence or absence of a specific food and drink, the type of food and drink that has been stored or removed, and the like. The security camera 5700 may provide a super high-resolution image and recognize objects or people in the image even in a dark environment using high sensitivity. The robot 5800 may be introduced into a disaster site or an industrial site that cannot be directly accessed by humans, and may provide a high-resolution image. The medical camera 5900 may provide a high-resolution image for diagnosis or surgery, and dynamically adjust the field of view.

[0153] Also, the image acquisition device 1000 may be applied to a vehicle 6000 as shown in Fig. 17E. The vehicle 6000 may include a plurality of vehicle cameras 6010, 6020, 6030, and 6040 arranged at various positions, and each of the vehicle cameras 6010, 6020, 6030, and 6040 may include the image acquisition device 1000 according to an embodiment. The vehicle 6000 may provide a driver with various information related to the inside or surroundings of the vehicle 6000 by using the plurality of vehicle cameras 6010, 6020, 6030, and 6040, and may automatically recognize objects or people in the image and provide information required for autonomous driving.

[0154] Although the image acquisition device and the electronic device including the same have been described with reference to the embodiments shown in the drawings, they are merely illustrative, and a person having ordinary skill in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the disclosed embodiments should be considered from an explanatory perspective, not a limiting one. The scope of the rights is set forth in the claims, not in the above description, and all differences within the scope of the rights should be interpreted as being within the scope of the rights. [Explanation of symbols]

[0155] 100 First image sensor 101,201,400 Timing Controller (TC) 102,202 Row Decoder 110 First sensor layer 120 Color Filter 130 First microlens array 190 First Imaging Optical System 200 Second image sensor 210 Second Sensor Layer 220 Spectral Filter 230 2nd Microlens Array 290 Second Imaging Optical System 300 Memory 500 processors 510 Video Acquisition Department 520 Video Matching Unit 530 Hue Transmission Unit 540 Video synthesis section 1000 Image Acquisition Device IMG1 1st video IMG2 2nd video IMG3 3rd video IMG4 4th video OBJ Subject PA1 1st pixel array PA2 2nd pixel array SU Circuit Board

Claims

1. a first image sensor having a first spatial resolution and configured to capture a first image based on a first wavelength band; a second image sensor having a second spatial resolution lower than the first spatial resolution and configured to acquire a second image based on a second wavelength band wider than the first wavelength band; a processor for aligning the second image based on the first image using relative position information between the first image sensor and the second image sensor, replacing a hue component of the first image with a hue component of the second image to form a hue component of a third image, and combining the hue component of the first image and the hue component of the third image in an alpha blending manner to form a hue component of a fourth image; 2. An image acquisition device comprising:

2. the processor multiplies the hue component of the first image by an alpha blending coefficient α, and adds the hue component of the third image by 1-α to form the hue component of the fourth image; The image capture device of claim 1 , wherein the alpha blending coefficient α has a value between 0 and 1.

3. The image acquisition device of claim 1 , wherein the processor calculates parameters for matching the first image and the second image based on at least one of relative position information, resolution, angle of view, and focal length of each of the first image sensor and the second image sensor.

4. The image acquisition device of claim 1 , wherein the third image has the first spatial resolution and a color gamut corresponding to the second image.

5. The image acquisition device of claim 1 , wherein the processor matches hue statistics of the first image with hue statistics of the second image to form a hue component of the third image.

6. The image acquisition device of claim 5 , wherein the hue statistics is an average value of surrounding pixels adjacent to a central pixel.

7. The image acquisition device of claim 5 , wherein the hue statistics is a standard deviation value of neighboring pixels adjacent to a central pixel.

8. The image acquisition device according to claim 6 , wherein the processor calculates the average value by weighting the surrounding pixels with a similarity between the center pixel and the surrounding pixels.

9. The image capture device of claim 7 , wherein the processor calculates the standard deviation value by weighting the similarity between the central pixel and the surrounding pixels as a square of the deviation.

10. The image acquisition device of claim 8 , wherein the similarity is calculated using luminance values ​​of the central pixel and the surrounding pixels as a feature vector.

11. The processor determines the alpha blending coefficient α by a logical sum of a first coefficient and a second coefficient; The first coefficient indicates information related to an occlusion area due to a disparity between the first image and the second image, The image acquisition device of claim 2 , wherein the second coefficient indicates false color information of a hue component of the third image.

12. The processor, If there is no pixel corresponding to the pixel of interest in the first image in the second image, a first coefficient associated with the pixel of interest is set to 1; The image capture device of claim 11 , wherein the first coefficient is set to 0 when a pixel in the second image corresponds to a pixel of interest in the first image.

13. The image capture device of claim 11 , wherein the processor determines a larger value of the first coefficient and the second coefficient as the alpha blending coefficient α.

14. The image acquisition device of claim 11 , wherein the second coefficient is proportional to a color difference between the first image and the third image and has a value between 0 and 1.

15. The image acquisition device of claim 11 , wherein the second coefficient has a value closer to 0 as the saturation of the first image increases and has a value closer to 1 as the saturation of the first image decreases.

16. The image acquisition device of claim 1 , wherein the processor separates the first image and the second image into a luminance component and a hue component to form the third image and the fourth image, respectively.

17. The image acquisition device of claim 1, wherein the processor extracts edge features or corner features in the first image and the second image, and matches at least one of the edge features or the corner features between the first image and the second image, thereby aligning the first image and the second image.

18. The processor sets the second coefficient based on a difference in luminance values ​​between corresponding pixels of the first image and the second image, The larger the difference in the luminance values ​​is, the closer the second coefficient is to 1 is set; The image acquisition device of claim 11 , wherein the second coefficient is set to be closer to 0 as the difference between the luminance values ​​is smaller.

19. An electronic device comprising an image capture device according to any one of claims 1 to 18.

20. acquiring a first image and a second image from a first image sensor and a second image sensor, respectively; aligning the acquired first and second images; replacing the hue components of the first image with the hue components of the second image to form a third image; forming a color component of a fourth image by combining the color components of the first image and the third image in an alpha blending manner; A method for controlling an image capture device, comprising:

Images