Image acquisition apparatus and electronic apparatus including the same

Abstract

An image acquisition apparatus includes: a first image sensor having a first spatial resolution and configured to acquire a first image based on a first wavelength band; a second image sensor having a second spatial resolution lower than the first spatial resolution and configured to acquire a second image based on a second wavelength band wider than the first wavelength band; and a processor configured to: register the first image and the second image by using relative position information between the first image sensor and the second image sensor; generating a third image, the color component of which is determined by replacing the color component of the first image with the color component of the second image; and generating a fourth image, in which a color component of the fourth image is determined by synthesizing a color component of the first image and a color component of the third image using alpha mixing.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2023-0143171 filed on October 24, 2023, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0003] Example embodiments of the present disclosure relate to an image acquisition device that provides an image with a wide color gamut and an electronic device including the image acquisition device. Background Art

[0004] The image sensor is a device that receives light incident from a subject and photoelectrically converts the received light to generate an electric signal.

[0005] For color expression, an image sensor generally uses a color filter in the form of a filter element array configured to selectively transmit red light, green light, and blue light, and senses the amount of light transmitted through each filter element.

[0006] In such image acquisition, the wavelength band transmitted by the filter element provided in the color filter is limited, which restricts the color gamut for expressing the object. Summary of the invention

[0007] One or more example embodiments of the present disclosure provide an image acquisition device providing an image having a wide color gamut, and an electronic device including the image acquisition device.

[0008] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the disclosed embodiments.

[0009] According to one aspect of an example embodiment, an image acquisition device includes: a first image sensor having a first spatial resolution and configured to acquire a first image based on a first band; a second image sensor having a second spatial resolution lower than the first spatial resolution and configured to acquire a second image based on a second band wider than the first band; and a processor configured to: align the first image and the second image in association with each other by using relative position information between the first image sensor and the second image sensor; generate a third image, wherein color components of the third image are determined by replacing color components of the first image with color components of the second image; and generate a fourth image, wherein color components of the fourth image are determined by synthesizing color components of the first image and color components of the third image using alpha blending.

[0010] The processor may determine the color components of the fourth image by summing a value obtained by multiplying the color components of the first image by an alpha blending coefficient α and a value obtained by multiplying the color components of the third image by (1-α), and the alpha blending coefficient α may have a value in the range of approximately 0 or greater to approximately 1 or less.

[0011] The processor may calculate parameters for registering the first image and the second image based on at least one of relative position information, resolution, field of view, or focal length of each of the first image sensor and the second image sensor.

[0012] The third image may have the first spatial resolution and a color gamut corresponding to the color gamut of the second image.

[0013] The processor may determine color components of the third image by matching color statistics of the first image to color statistics of the second image.

[0014] The color statistic may correspond to an average value for peripheral pixels adjacent to the center pixel.

[0015] The color statistics may correspond to standard deviation values ​​for peripheral pixels adjacent to the central pixel.

[0016] The processor may calculate the average by weighting by the similarity between the center pixel and the peripheral pixels.

[0017] The processor may calculate the standard deviation value by weighting the standard deviation using the similarity between the central pixel and the peripheral pixels.

[0018] The similarity can be calculated by using the brightness value of the central pixel and the brightness values ​​of the peripheral pixels as feature vectors.

[0019] The processor may determine the alpha blending coefficient α as a logical sum of a first coefficient and a second coefficient, the first coefficient may represent information about an occlusion area according to a disparity between the first image and the second image, and the second coefficient may represent false color information of a color component of the third image.

[0020] When there is no pixel in the second image corresponding to the pixel of interest of the first image, the processor may set the first coefficient of the pixel of interest of the first image to 1, and when at least one pixel of the second image corresponds to the pixel of interest of the first image, the processor may set the first coefficient of the pixel of interest to 0.

[0021] The processor may determine a larger value between the first coefficient and the second coefficient as the value of the alpha mixing coefficient α.

[0022] The second coefficient may be proportional to a color difference between the first image and the third image and have a value in the range of about 0 or more to about 1 or less.

[0023] The second coefficient may have a value closer to 0 as the saturation of the first image increases, or a value closer to 1 as the saturation of the first image decreases.

[0024] The processor may set the second coefficient based on a difference in brightness values ​​between corresponding pixels of the first image and the second image such that the second coefficient is closer to 1 as the difference increases or closer to 0 as the difference decreases.

[0025] The processor may separate the first image and the second image into a brightness component and a color component, respectively, and generate a third image and a fourth image by using the separated first image and the separated second image.

[0026] The processor may register the first image and the second image by extracting at least one of edge features or corner features from the first image and the second image, and matching the at least one of the extracted edge features or corner features between the first image and the second image.

[0027] According to an aspect of an exemplary embodiment, an electronic device includes the aforementioned image acquisition device.

[0028] According to one aspect of an example embodiment, a method of controlling an image acquisition device includes: acquiring a first image from a first image sensor, and acquiring a second image from a second image sensor; aligning the acquired first image and the acquired second image in association with each other; generating a third image, wherein color components of the third image are determined by replacing color components of the first image with color components of the second image; and generating a fourth image, wherein color components of the fourth image are determined by synthesizing color components of the first image and color components of the third image using alpha blending. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other aspects, features, and advantages of certain example embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0030] Figure 1 is a block diagram showing a schematic structure of an image acquisition device according to an example embodiment;

[0031] Figure 2 is a color coordinate that interprets a color gamut represented by an image acquired by an image acquisition device according to an example embodiment;

[0032] Figure 3 is a conceptual diagram showing a schematic structure of an image acquisition device according to an exemplary embodiment;

[0033] Figure 4 shows a circuit configuration of a first image sensor and a second image sensor provided in an image acquisition device according to an example embodiment;

[0034] Figure 5 shows a wavelength spectrum of a first image sensor provided in an image acquisition device according to an example embodiment;

[0035] Figures 6 to 8 shows a pixel arrangement of a first image sensor provided in an image acquisition device according to an example embodiment;

[0036] Fig. 9 shows a wavelength spectrum of a second image sensor provided in an image acquisition device according to an example embodiment;

[0037] Figures 10 to 12 shows a pixel arrangement of a second image sensor provided in an image acquisition device according to an example embodiment;

[0038] Fig.13 is a flowchart schematically explaining an image processing process of an image acquisition device according to an example embodiment;

[0039] Fig.14 is a block diagram showing a schematic structure of an electronic device according to an example embodiment;

[0040] Fig.15 is a schematic diagram showing the arrangement of Fig.14 A block diagram of a camera module in an electronic device; and

[0041] FIG. 16A to FIG. 16E as well as FIG. 17A to FIG. 17E Various examples of electronic devices to which the image acquisition device according to the example embodiments is applied are shown. DETAILED DESCRIPTION

[0042] Reference will now be made in detail to embodiments, examples of which are shown in the accompanying drawings, wherein the same reference numerals refer to the same elements throughout. In this respect, example embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein. Accordingly, embodiments are described below with reference to the accompanying drawings to explain various aspects of the present disclosure. The term "and / or" as used herein includes any and all combinations of one or more of the items listed in association. Statements such as "at least one of..." modify the entire element list when following an element list, rather than modifying a separate element in the list.

[0043] Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings. The example embodiments of the present disclosure may be modified in various ways and may be embodied in many different forms. In the following drawings, the same reference numerals refer to the same components, and the size of each component in the drawings may be exaggerated for clarity and convenience of description.

[0044] Hereinafter, the description as “on” or “on” may include above in direct contact as well as above in non-contact.

[0045] Terms such as "first" or "second" used herein may be used to describe various components, but may be used for the purpose of distinguishing one component from another component. These terms do not limit the difference in materials or structures of the components.

[0046] Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. When a component “includes” or “comprising” a component in the specification, unless otherwise defined, it does not exclude other components but may further include other components.

[0047] Furthermore, in the specification, the term "unit" or "module" means a unit or module that processes at least one function or operation and may be implemented by hardware, software, or a combination of hardware and software.

[0048] The term "above" and similar directional terms may apply to both the singular and the plural.

[0049] In this disclosure, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." When only one item is intended, the term "a" or similar language is used. For example, the term "a processor" may refer to a single processor or multiple processors. When a processor is described as performing an operation, and the processor is referred to as performing additional operations, the multiple operations may be performed by a single processor, or any one or combination of multiple processors.

[0050] Throughout this disclosure, references to "one embodiment," "(one) embodiment," "example embodiment," or similar language may indicate that a particular feature, structure, or characteristic described in conjunction with the indicated embodiment is included in at least one embodiment of the inventive solution. Therefore, the phrases "in one embodiment," "in (one) embodiment," "in an example embodiment," and similar language throughout this disclosure may, but do not necessarily, all refer to the same embodiment. The embodiments described herein are example embodiments, and therefore, the disclosure is not limited thereto, and may be implemented in various other forms. It will also be understood that one or more features included in one embodiment may be combined with one or more features included in other embodiments.

[0051] The operations of the methods described herein may be performed in any suitable order, unless it is explicitly stated that the operations must be performed in the order described. In addition, the use of all exemplary terms (e.g., "etc.") is only used to describe the technical ideas in detail, and unless the claims are explicitly limited, the scope of rights is not limited by these terms.

[0052] Figure 1 is a block diagram illustrating a schematic structure of an image acquisition apparatus 1000 according to an example embodiment. Figure 2 is an example of color coordinates explaining a color gamut represented by an image acquired by the image acquisition apparatus 1000 according to an exemplary embodiment.

[0053] The image acquisition device 1000 includes a first image sensor 100 for acquiring a first image IMG1 based on M (M is an integer) spectral bands, a second image sensor 200 for acquiring a second image IMG2 based on N (N is an integer greater than M) spectral bands, and a processor 500, which performs signal processing on the first image IMG1 and the second image IMG2 to form (or generate) a third image IMG3, and synthesizes the first image IMG1 and the third image IMG3 to form (or generate) a fourth image IMG4.

[0054] The first image sensor 100 is a sensor used in a general RGB camera, and may be a complementary metal oxide semiconductor (CMOS) image sensor using a Bayer color filter array. The first image IMG1 acquired by the first image sensor 100 may be an image based on red, green, and blue (RGB), and may have a color gamut range acquired by a general RGB camera. For example, the first image IMG1 may have a standard RGB (s-RGB) color gamut, a BT.709 color gamut, a DCI-P3 color gamut, or an Adobe RGB color gamut. Figure 2 The BT.709 color gamut and the DCI-P3 color gamut are shown as examples.

[0055] The second image sensor 200 is a sensor that senses more types of wavelengths of light than the first image sensor 100. The second image sensor 200 may use, for example, 16 channels, 31 channels, or another number of channels. The bandwidth of each channel may be set to be narrower than each of the R band, the G band, and the B band, and the total bandwidth (the sum of the bandwidths of all channels) may include an RGB bandwidth (i.e., a visible light bandwidth) and be wider than the RGB bandwidth. For example, the sum of the bandwidths of all channels of the second image sensor 200 may correspond to a bandwidth of about 350 nm to about 1000 nm (see, for example, Fig. 9). The second image IMG2 acquired by the second image sensor 200 may be a hyperspectral image, and may be a wavelength-based image including a band wider than an RGB band (e.g., a visible light band), and may also include an ultraviolet band or an infrared band wider than the visible light band by using 16 or more channels. The second image IMG2 may be an image acquired using all available channels of the second image sensor 200, or may be an image acquired by selecting a specific channel. The spatial resolution of the second image IMG2 may be lower than that of the first image IMG1, but embodiments of the present disclosure are not limited thereto.

[0056] The first image sensor 100 and the second image sensor 200 may be configured as separate chips or a single chip.

[0057] The first image IMG1 and the second image IMG2 acquired by the first image sensor 100 and the second image sensor 200, respectively, may be stored in the memory 300. The memory 300 may be a line memory storing the first image IMG1 and the second image IMG2 in units of lines, or may be a frame buffer storing the first image IMG1 and the second image IMG2 as a whole. The memory 300 may use a static random access memory (SRAM) or a dynamic random access memory (DRAM). The memory 300 may be located outside each of the first image sensor 100 and the second image sensor 200, or may be integrated in each of the first image sensor 100 and the second image sensor 200. When integrated in each of the first image sensor 100 and the second image sensor 200, the memory 300 may be integrated with the sensor circuit. In this regard, the pixel portion as one part, and the circuit portion and the memory 300 as another part may be configured as separate stacks, and integrated into 2 stacks to form one chip. Alternatively, according to the configuration, it may also be a 3-D stack having three layers of a pixel portion, a circuit portion, and a memory 300.

[0058] The processor 500 uses the first image IMG1 as an input image and the second image IMG2 as a reference image to form a third image IMG3. The third image IMG3 has a spatial resolution corresponding to the first image IMG1 of the first image sensor 100 and a color gamut corresponding to the second image IMG2 of the second image sensor 200. The third image IMG3 may have the same spatial resolution as the first image IMG1 and also have an extended color gamut (e.g., Figure 2 DCI-P3 or larger color gamut in the UHD).

[0059] The third image IMG3 can be obtained by replacing the color components of the first image IMG1 with the color components of the second image IMG2. To this end, the correspondence between the pixels of the first image IMG1 and the pixels of the second image IMG2 can be identified by performing image registration on the first image IMG1 and the second image IMG2 on a two-dimensional plane.

[0060] The processor 500 may form a fourth image IMG4 by combining the first image IMG1 and the third image IMG3 using alpha blending. The fourth image IMG4 may have the same spatial resolution as the first image IMG1 and also have an extended color gamut (eg, Figure 2 DCI-P3 or larger color gamut in the UHD).

[0061] The image processing process of processor 500 is described in more detail below.

[0062] Back to Figure 1 , the processor 500 may include an image acquisition unit 510, an image registration unit 520, a color transfer unit 530, and an image synthesis unit 540. For ease of explanation, the image acquisition unit 510, the image registration unit 520, the color transfer unit 530, and the image synthesis unit 540 are described according to the operation of the processor 500, but this division does not necessarily mean that the image acquisition unit 510, the image registration unit 520, the color transfer unit 530, and the image synthesis unit 540 are physically separated. The image acquisition unit 510, the image registration unit 520, the color transfer unit 530, and the image synthesis unit 540 may correspond to any combination of hardware and / or software included in the processor 500, and may be physically the same as or different from each other.

[0063] The image acquisition unit 510 may acquire an image from each of the first image sensor 100 and the second image sensor 200, and then perform basic image processing before or after storing the image in the memory 300. For example, the image acquisition unit 510 may perform bad pixel correction, fixed pattern noise correction, crosstalk reduction, remosaicing, demosaicing, false color reduction, denoising, chromatic aberration correction, etc. In addition, the image acquisition unit 510 may perform the same image processing on the images from the first image sensor 100 and the second image sensor 200, or may perform different image processing on the images from the first image sensor 100 and the second image sensor 200.

[0064] The image acquisition unit 510 may apply de-mosaicing to interpolate the original images of the first image IMG1 and the second image IMG2, each having one channel, into an RGB image and a multi-channel spectral image, respectively. In addition, the image acquisition unit 510 may convert the first image IMG1, which is an RGB image, and the second image IMG2, which is a multi-channel spectral image, into a three-channel XYZ color space through an image domain conversion process. In order to accurately transfer the color information of the second image IMG2 to the first image IMG1, the image acquisition unit 510 may normalize the brightness of the second image IMG2 relative to the first image IMG1.

[0065] The image registration unit 520 may use the relative position information between the first image sensor 100 and the second image sensor 200 to register the first image IMG1 and the second image IMG2. The image registration unit 520 may determine the positional relationship between the pixels of the images respectively acquired from the first image sensor 100 and the second image sensor 200 by considering the spatial resolution of the image, the field of view of the optical system used to acquire the image, the focal length, etc. At this time, the image registration unit 520 may superimpose the image of one sensor (e.g., the first image of the first image sensor) on the image of the other sensor (e.g., the second image of the second image sensor). For example, the image registration unit 520 may retrieve the pixel of the second image IMG2 corresponding to each pixel of the first image IMG1 with respect to the first image IMG1 acquired by the first image sensor 100. To this end, the image registration unit 520 may perform processing such as scaling, translation, rotation, affine transformation, perspective transformation, etc. on the pixels of the second image IMG2.

[0066] One or more pixels in the second image IMG2 may correspond to a pixel of the first image IMG1. When a plurality of pixels in the second image IMG2 correspond to a pixel in the first image IMG1, the image registration unit 520 may calculate a pixel value of the second image IMG2 corresponding to the pixel of the first image IMG1 by performing a weighted sum on the plurality of pixels in the second image IMG2 according to the positions of the plurality of pixels.

[0067] There may be no pixels in the second image IMG2 corresponding to pixels of the first image IMG1. This may be due to the parallax between the first image sensor 100 and the second image sensor 200, and when there is no corresponding pixel in the second image IMG2 for a pixel in the first image IMG1, the image registration unit 520 may determine such pixel in the first image IMG1 as an occlusion area. The image registration unit 520 may store information about pixels determined as occlusion areas in the form of a map. In an embodiment, the image registration unit 520 may generate an occlusion map by setting the value of the corresponding pixel to 1 when there is no corresponding pixel between the first image IMG1 and the second image IMG2, and setting the value of the corresponding pixel to 0 when there is a corresponding pixel between the first image IMG1 and the second image IMG2.

[0068] The image registration unit 520 may perform registration in sub-pixel units to increase registration accuracy. In an embodiment, in the registration in sub-pixel units, the position of a pixel may be represented as a real number instead of an integer.

[0069] The image registration unit 520 can also improve the registration efficiency by focusing the first image sensor 100 and the second image sensor 200 on the object at the same position through focus control. In addition, the image registration unit 520 can quickly and accurately perform image registration by ensuring that the two sensors have the same field of view. For example, when the imaging optical systems used to form optical images on the first image sensor 100 and the second image sensor 200 have the same focal length and the same field of view, there is only a translation between the first image IMG1 and the second image IMG2, and the relative position between the first image sensor 100 and the second image sensor 200 and the focal length of the imaging optical system can be used to calculate the relevant parameters.

[0070] Before performing the registration, the image registration unit 520 may correct the aberrations included in the first image IMG1 and the second image IMG2. That is, the image registration unit 520 may perform the registration after correcting the effects of distortion, geometric aberration, chromatic aberration, etc. caused by the lens included in the imaging optical system used when acquiring the first image IMG1 and the second image IMG2.

[0071] The image registration unit 520 may extract edge feature information within the image, and perform feature matching between the two images by using the edge feature information. When the image registration is misaligned in the edge region of the object, color distortion may occur. Therefore, the image registration unit 520 may extract edge information of the two images, and perform registration on the two images based on the match between their edge information, so that the edges of the two images are aligned to prevent distortion in the edge region. In an embodiment, the image registration unit 520 may perform image registration using other image features (e.g., corner features (e.g., corner points)) in addition to or as an alternative to the edge.

[0072] The color transfer unit 530 may use the first image IMG1 as an input image and the second image IMG2 as a reference image to change the color of the first image IMG1 to match the color of the second image IMG2 (i.e., replace the color components of the first image IMG1 with the color components of the second image IMG2), and form a third image IMG3 based thereon. In an embodiment, the color transfer unit 530 may form (or determine) the color components of the third image IMG3 by matching the color statistics of the first image IMG1 with the color statistics of the second image IMG2. The color statistics may be based on an average value for peripheral pixels adjacent to a central pixel and / or a standard deviation value for peripheral pixels adjacent to a central pixel.

[0073] The color transfer unit 530 may form a third image IMG3 by separating the first image IMG1 and the second image IMG2 into a brightness component and a color component, respectively. For example, the color transfer unit 530 may form a color component of the third image IMG3 by changing the color of the first image IMG1 to match the color of the second image IMG2 in a CIE LAB color space having a brightness component and a color component with higher independence, and form (or determine) a brightness component of the third image IMG3 by using the brightness component of the first image IMG1 as it is.

[0074] In an embodiment, the color transfer unit 530 may form the third image IMG3 using a color conversion function represented by Formula 1 and Formula 2 below.

[0075] First, the brightness component of the third image IMG3 The brightness component value of the first image IMG1 can be used , as shown in Formula 1 below.

[0076] [Formula 1]

[0077] Next, the color components of the peripheral pixels q adjacent to the central pixel p relative to the color components of the central pixel p can be used. The statistics are defined independently for the color components of the third image IMG3 as follows.

[0078] In an embodiment, the color components of the third image IMG3 may be calculated by the following formula 2: .

[0079] [Formula 2]

[0080] Wherein, "μ" is the average weighted by the similarity between the central pixel "p" and the peripheral pixels "q" around the central pixel p, and "STD" is the standard deviation weighted by the similarity between the central pixel "p" and the peripheral pixels "q" around the central pixel p. Formula 2 can be understood as the statistic {μ2 C (p), STD2 C (p)} replaces the statistics {μ1 C (p), STD1 C (p)} process.

[0081] Statistics {μ C (p), STD C (p)} can be calculated as shown in the following Formula 3 and Formula 4.

[0082] [Formula 3]

[0083] [Formula 4]

[0084] here, is a peripheral area set relative to the central pixel p and may have a square window. Alternatively, the peripheral area may have other area shapes. It may be a peripheral area disposed adjacent to the central pixel p relative to the central pixel p. Indicates the peripheral area of ​​the setting The number of pixels, and in the embodiment, when is a peripheral area disposed adjacent to the central pixel p and having a square window, It can be 8. is a Gaussian kernel representing the similarity between the central pixel p and the peripheral pixels q.

[0085] The mean value μ and the standard deviation STD used in the transfer function (e.g., as represented by Formula 2) for determining the color components of the third image IMG3 can be calculated by weighting according to the similarity of the central pixel p and the peripheral pixels q, and therefore, the transfer function (e.g., Formula 2) for determining the color components of the third image IMG3 can depend on the similar pixels between the first image IMG1 and the second image IMG2.

[0086] Represents the similarity between the central pixel p and the peripheral pixels q It can be calculated as shown in the following formula 5.

[0087] [Formula 5]

[0088] in, is a hyperparameter representing the smoothness of the kernel. In the exemplary embodiment, the feature vector The brightness value of a simple vector can be used as a feature vector. When the brightness value is used as a feature vector, the similarity calculation amount can be reduced. However, the embodiments of the present disclosure are not limited thereto, and any value can be used as long as the feature vector is a real number vector corresponding to the similarity of pixels.

[0089] The image synthesis unit 540 may form (or determine) color components of the fourth image IMG4 by synthesizing color components of the first image IMG1 and color components of the third image IMG3 using alpha blending.

[0090] The brightness component of the fourth image IMG4 and the brightness component of the third image IMG3 Similarly, the brightness component value of the first image IMG1 can be used as is (See Formula 1).

[0091] Used to determine the color components of the fourth image IMG4 The transfer function can be expressed as the following formula 6.

[0092] [Formula 6]

[0093] The image synthesis unit 540 may determine the color component of the fourth image IMG4 by summing a value obtained by multiplying the color component of the first image IMG1 by the alpha blending coefficient α and a value obtained by multiplying the color component of the third image IMG3 by (1−α). The alpha blending coefficient α may have a value in a range from about 0 or more to about 1 or less.

[0094] The image synthesis unit 540 can use the first coefficient α 1 and the second coefficient α 2 The value of the alpha blending coefficient α is determined by the logical sum of α and α. In an embodiment, the image synthesis unit 540 can use the first coefficient α 1 and the second coefficient α 2 The larger value is set as the value of the alpha mixing coefficient α. This can be expressed as shown in the following formula 7.

[0095] [Formula 7]

[0096] The first coefficient α 1 The may be a parameter related to the occlusion area determined due to the parallax between the first image IMG1 and the second image IMG2.

[0097] The image synthesis unit 540 may receive information OC related to the occlusion region from the image registration unit 520. As described above, when no pixel in the second image IMG2 corresponds to a pixel of the first image IMG1, the image registration unit 520 may determine that the corresponding pixel of the first image IMG1 is the occlusion region. The image registration unit 520 may store information related to the pixels determined as the occlusion region in the form of a map. In an embodiment, the image registration unit 520 may generate an occlusion map by setting the value of the corresponding pixel to 1 when the corresponding pixel does not exist in the second image IMG2, and setting the value of the corresponding pixel to 0 when the corresponding pixel exists in the second image IMG2. As a result, the first coefficient α 1 It may have a value of 1 when the pixel corresponds to an occlusion area (ie, when there is no corresponding pixel between the first image IMG1 and the second image IMG2), and have a value of 0 when the pixel does not correspond to an occlusion area (ie, when there is a corresponding pixel between the first image IMG1 and the second image IMG2).

[0098] The second coefficient α 2 The second image sensor 200 may sense relatively more wavelength channels than the first image sensor 100. Accordingly, the distance between pixels of the same wavelength channel in the second image sensor 200 may be greater than the distance between pixels of the same wavelength channel in the first image sensor 100 (see, for example, Figure 6 and Fig.10 As a result, the second image sensor 200 may generate incorrect color information (ie, false color information) during demosaicing.

[0099] The image synthesis unit 540 can convert the second coefficient α 2is set to be proportional to the color difference between the first image IMG1 and the third image IMG3 and has a value in the range of about 0 or more to about 1 or less.

[0100] In addition, the image synthesis unit 540 may determine the second coefficient α based on the brightness value difference between corresponding pixels of the first image IMG1 and the second image IMG2. 2 This is based on the empirical fact that false color information is particularly prominent in the area where the brightness value difference between the corresponding pixels of the first image IMG1 and the second image IMG2 is large. Therefore, the image synthesis unit 540 can set the second coefficient α 2 is set to be proportional to the difference in brightness values ​​between corresponding pixels of the first image IMG1 and the second image IMG2, and has a value in the range of about 0 or more to about 1 or less. In an embodiment, the image synthesis unit 540 may 2 is set to be closer to 1 as the difference in brightness values ​​between corresponding pixels of the first image IMG1 and the second image IMG2 increases, and closer to 0 as the difference decreases.

[0101] In addition, the image synthesis unit 540 can convert the second coefficient α 2 The second coefficient α is set to be closer to 0 as the saturation of the first image IMG1 increases, and closer to 1 as the saturation of the first image IMG1 decreases. This is based on the empirical fact that false color information is particularly prominent in a colorless area. The setting of the second coefficient α performed by the image synthesis unit 540 2 An example of the method is shown in Formula 8 below.

[0102] [Formula 8]

[0103] The clip function checks whether the input value is within the range of 0 to 1, and outputs 0 when the input value is less than 0, and outputs 1 when the input value is greater than 1. As a result, the clip function increases the second coefficient α by 2 is set to have a value in the range of about 0 or more to about 1 or less. ΔC(p) is a color difference value between the first image IMG1 and the third image IMG3. S1(p) is a saturation value indicating the clarity of the color component of the first image IMG1. k 1 and k 2 Each is a heuristic hyperparameter.

[0104] In an example embodiment, the image synthesis unit 540 may convert the brightness component of the fourth image IMG4 into and the color components of the fourth image IMG4 Convert to RGB image to form (or generate) the final output image.

[0105] As described above, the image acquisition device 1000 according to the embodiment can use the first image IMG1 with high spatial resolution captured by the first image sensor 100 as an input image, and use the second image IMG2 with wide color gamut captured by the second image sensor 200 as a reference image to transfer the color information of the second image IMG2 to the first image IMG1, thereby forming a final image with the high spatial resolution of the first image IMG1 and the wide color gamut of the second image IMG2.

[0106] The color information transfer is achieved by matching the color statistics of the first image IMG1 to the color statistics of the second image IMG2 (or replacing the color statistics of the first image IMG1 with the color statistics of the second image IMG2).

[0107] In addition, the image acquisition device 1000 may synthesize the first image and the intermediate image (third image) by alpha blending to form a final image (fourth image). In order to perform alpha blending, the image acquisition device 1000 may determine the alpha blending coefficient α as the first coefficient α 1 (parameters related to the occlusion area) and the second coefficient α 2 The logical sum of (parameters related to false color information of the intermediate image (third image)) is performed to correct the incorrect color information transfer and form a final image (fourth image).

[0108] Figure 3 1 is a conceptual diagram illustrating a schematic structure of an image acquisition apparatus 1000 according to an exemplary embodiment. Figure 4 A circuit configuration of a first image sensor IMG1 and a second image sensor IMG2 provided in an image acquisition device 1000 according to example embodiments is shown.

[0109] The image acquisition device 1000 may include a first image sensor 100 that acquires a first image IMG1 based on a first wavelength band, a second image sensor 200 that acquires a second image IMG2 based on a second wavelength band wider than the first wavelength band, and a processor 500 that performs signal processing on the first image IMG1 and the second image IMG2 to form a fourth image IMG4. The image acquisition device 1000 may also include a memory 300 that stores data related to the first image IMG1 and the second image IMG2.

[0110] The image acquisition apparatus 1000 may further include a first imaging optical system 190 that forms an optical image of the object OBJ on the first image sensor 100 , and a second imaging optical system 290 that forms an optical image of the object OBJ on the second image sensor 200 . Figure 3 and Figure 4It is shown that the first imaging optical system 190 and the second imaging optical system 290 each include one lens, but this is an example and is not limited thereto. The first imaging optical system 190 and the second imaging optical system 290 may be configured to have the same focal length and the same field of view. When the first imaging optical system 190 and the second imaging optical system 290 are configured to have the same focal length and the same field of view, the process of aligning the first image IMG1 and the second image IMG2 may be easier to form the third image IMG3 and the fourth image IMG4. However, embodiments of the present disclosure are not limited thereto.

[0111] The first image sensor 100 may include a first pixel array PA1. The first pixel array PA1 may include a first sensor layer 110 on which a plurality of first sensing elements are arranged, and a color filter 120 disposed on the first sensor layer 110. The color filter 120 may include a red filter, a green filter, and a blue filter that are alternately arranged. A first microlens array 130 may be disposed on the first pixel array PA1. Figures 5 to 8 Various examples of pixel arrangement applied to the first pixel array PA1 are described.

[0112] The second image sensor 200 may include a second pixel array PA2. The second pixel array PA2 may include a second sensor layer 210 in which a plurality of second sensing elements are arranged, and a spectral filter 220 disposed on the second sensor layer 210. The spectral filter 220 may include a plurality of filter groups. Each of the plurality of filter groups may include a plurality of unit filters having different transmission bands. The spectral filter 220 may be configured to subdivide and filter a band wider than the band of the color filter 120, for example, filtering a band including a band in the ultraviolet to infrared wavelength range. A second microlens array 230 may be disposed on the second pixel array PA2. Reference will be made to Figures 10 to 12 Various examples of pixel arrangement applied to the second pixel array PA2 are described.

[0113] The first sensor layer 110 and the second sensor layer 210 may each include, but are not limited to, a charge coupled device (CCD) sensor or a CMOS sensor.

[0114] The first pixel array PA1 and the second pixel array PA2 may be disposed to be horizontally (eg, in the X direction) spaced apart on the same circuit substrate SU.

[0115] A first circuit element processing a signal from the first sensor layer 110 and a second circuit element processing a signal from the second sensor layer 210 may be provided in the circuit substrate SU. However, the first circuit element and the second circuit element are not limited thereto and may be provided on separate substrates, respectively.

[0116] The memory 300 storing the data of the first image IMG1 and the second image IMG2 may be provided separately from the circuit substrate SU, but this is an example, and the memory 300 may be provided within the circuit substrate SU and in the same layer as the circuit element or in a separate layer. The memory 300 may be a line memory storing the first image IMG1 and the second image IMG2 in units of lines, or may be a frame buffer storing the first image IMG1 and the second image IMG2 as a whole. The memory 300 may use, for example, an SRAM or a DRAM.

[0117] Various circuit elements used in the image acquisition device 1000 may be integrated and arranged in the circuit substrate SU. For example, a logic layer including various analog circuits and digital circuits and a memory layer storing data may be arranged in the circuit substrate SU. The logic layer and the memory layer may each include different layers or the same layer.

[0118] refer to Figure 4 , a row decoder 102, an output circuit 103, and a timing controller (TC) 101 may be connected to the first pixel array PA1. The row decoder 102 may select one of the rows of the first pixel array PA1 in response to a row address signal output from the TC 101. The output circuit 103 may output a photosensitive signal in columns from a plurality of pixels arranged along the selected row. To this end, the output circuit 103 may include a column decoder and an analog-to-digital converter (ADC). For example, the output circuit 103 may include a plurality of ADCs arranged between the column decoder and the first pixel array PA1 for each column, or one ADC arranged at an output terminal of the column decoder. The TC 101, the row decoder 102, and the output circuit 103 may be implemented as one chip or separate chips. At least some of the illustrated circuit elements may be arranged in Figure 3 The processor for processing the first image IMG1 output through the output circuit 103 may be implemented as a single chip together with the TC 101, the row decoder 102, and the output circuit 103.

[0119] The row decoder 202, the output circuit 203, and the TC 201 may be connected to the second pixel array PA2, and may process a signal from the second pixel array PA2 similarly to the above. In addition, a processor for processing the second image IMG2 output through the output circuit 203 may be implemented as a single chip together with the TC 201, the row decoder 202, and the output circuit 203.

[0120] The first pixel array PA1 and the second pixel array PA2 are illustrated as having different pixel sizes and numbers, but this is an example and embodiments of the present disclosure are not limited thereto.

[0121] When operating two different types of sensors, timing control may be required according to different resolutions and output speeds and the size of the area where image registration is performed. For example, when an image string corresponding to an area is read out for the first image sensor 100, the image string corresponding to the area of ​​the second image sensor 200 may have been stored in the buffer, or may need to be re-read. Such timing needs to be calculated correctly, and the readout is performed according to the calculated timing. Alternatively, the operation of the first image sensor 100 and the second image sensor 200 can be synchronized by using the same synchronization signal. For example, TC 400 can also be set to send a synchronization signal sync to each of the first image sensor 100 and the second image sensor 200.

[0122] Figure 5 A wavelength spectrum of the first image sensor 100 provided in the image acquisition apparatus 1000 according to example embodiments is shown. Figures 6 to 8 A pixel arrangement of a first image sensor 100 provided in an image acquisition device 1000 according to example embodiments is shown.

[0123] refer to Figure 5 , the first image sensor 100 may detect red light in a band having a central wavelength of about 650 nanometers (nm), detect green light in a band having a central wavelength of about 540 nm, and detect blue light in a band having a central wavelength of about 440 nm.

[0124] refer to Figure 6 , filters for filtering the red R band, the green G band, and the blue B band are arranged in a Bayer pattern in the color filter 120, and the color filter 120 is arranged in the first pixel array PA1. In an embodiment, one unit pixel includes sub-pixels arranged in a 2×2 array, and a plurality of unit pixels are arranged two-dimensionally and repeatedly. A red filter and a green filter are arranged in a first row of unit pixels, and a green filter and a blue filter are arranged in a second row. Pixel arrangement may be in other ways besides the Bayer pattern.

[0125] For example, refer to Figure 7 , a CYGM arrangement in which a magenta pixel M, a cyan pixel C, a yellow pixel Y, and a green pixel G form one unit pixel is also possible.

[0126] In addition, refer to Figure 8 , an RGBW arrangement in which a green pixel G, a red pixel R, a blue pixel B, and a white pixel W form one unit pixel is also possible. In addition, although not shown, the unit pixel may have a 3×2 array form. In addition, the pixels of the first pixel array PA1 may be arranged in various ways according to the color characteristics of the first image sensor 100.

[0127] Fig. 9 A wavelength spectrum of the second image sensor 200 provided in the image acquisition apparatus 1000 according to example embodiments is shown. Figures 10 to 12 A pixel arrangement of the second image sensor 200 provided in the image acquisition device 1000 according to example embodiments is shown.

[0128] refer to Fig. 9 , the second image sensor 200 may have 16 bands (or channels), wherein the central wavelength is in the range of about 300 nm to about 950 nm. Fig.10 , the spectral filter 220 provided in the second pixel array PA2 may include a plurality of filter groups 221 arranged in a two-dimensional form. Here, each filter group 221 may include 16 unit filters F1 to F16 arranged in a 4×4 array.

[0129] The first unit filter F1 and the second unit filter F2 may have central wavelengths UV1 and UV2 in the ultraviolet region, respectively, and the third to fifth unit filters F3 to F5 may have central wavelengths B1 to B3 in the blue light region, respectively. The sixth to eleventh unit filters F6 to F11 may have central wavelengths G1 to G6 in the green light region, respectively, and the twelfth to fourteenth unit filters F12 to F14 may have central wavelengths R1 to R3 in the red light region, respectively. In addition, the fifteenth and sixteenth unit filters F15 and F16 may have central wavelengths NIR1 and NIR2 in the near infrared region, respectively.

[0130] Fig.11 FIG. 2 is a plan view showing another example of a filter group 222 provided in the spectral filter 220. Fig.11 , the filter group 222 may include nine unit filters F1 to F9 arranged in a 3×3 array. Here, the first unit filter F1 and the second unit filter F2 may have central wavelengths UV1, UV2 in the ultraviolet region, respectively, and the fourth unit filter F4, the fifth unit filter F5, and the seventh unit filter F7 may have central wavelengths B1 to B3 in the blue light region, respectively. The third unit filter F3 and the sixth unit filter F6 may have central wavelengths G1 and G2 in the green light region, respectively, and the eighth unit filter F8 and the ninth unit filter F9 may have central wavelengths R1 and R2 in the red light region, respectively.

[0131] Fig.12 FIG. 2 is a plan view showing another example of the filter group 223 provided in the spectral filter 220. Fig.12, the filter group 223 may include 25 unit filters F1 to F25 arranged in a 5×5 array. Among them, the first unit filter F1 to the third unit filter F3 may have central wavelengths UV1 to UV3 in the ultraviolet region, respectively, and the sixth unit filter F6, the seventh unit filter F7, the eighth unit filter F8, the eleventh unit filter F11 and the twelfth unit filter F12 may have central wavelengths B1 to B5 in the blue light region, respectively. The fourth unit filter F4, the fifth unit filter F5 and the ninth unit filter F9 may have central wavelengths G1 to G3 in the green light region, respectively, and the tenth unit filter F10, the thirteenth unit filter F13, the fourteenth unit filter F14, the fifteenth unit filter F15, the eighteenth unit filter F18 and the nineteenth unit filter F19 may have central wavelengths R1 to R6 in the red light region, respectively. In addition, the twentieth unit filter F20, the twenty-third unit filter F23, the twenty-fourth unit filter F24 and the twenty-fifth unit filter F25 may have central wavelengths NIR1 to NIR4 in the near infrared region, respectively.

[0132] The unit filters F1 to F25 provided in the spectrum filter 220 may have a resonant structure including two reflectors, and the transmission band may be determined according to the characteristics of the resonant structure. The transmission band may be adjusted according to the material of the reflector, the material of the dielectric material in the cavity, and the thickness of the cavity. In addition, a structure using a grating and a structure using a distributed Bragg reflector (DBR) may be applied to the unit filters F1 to F25.

[0133] In addition, the pixels of the second pixel array PA2 may be arranged in various ways according to color characteristics of the second image sensor 200 .

[0134] Fig.13 is a flowchart schematically explaining an image processing procedure of the image acquisition apparatus 1000 according to an example embodiment.

[0135] refer to Figure 1 and Fig.13 , the image acquisition apparatus 1000 acquires a first image IMG1 from the first image sensor 100 , and acquires a second image IMG2 from the second image sensor 200 ( S1410 ).

[0136] The image acquisition device 1000 may perform basic image processing on the first image IMG1 and / or the second image IMG2 before or after storing the first image IMG1 and / or the second image IMG2 in the memory 300. For example, the image acquisition device 1000 may perform bad pixel correction, fixed pattern noise correction, crosstalk reduction, re-mosaicing, demosaicing, false color reduction, denoising, chromatic aberration correction, etc. (S1420).

[0137] The image acquisition device 1000 registers the first image IMG1 and the second image IMG2 using the relative position information between the first image sensor 100 and the second image sensor 200 (S1430). The image acquisition device 1000 determines the positional relationship between the pixels of the first image IMG1 and the second image IMG2 respectively acquired from the first image sensor 100 and the second image sensor 200 by considering, for example, the spatial resolution of each of the first image IMG1 and the second image IMG2, the field of view of the optical system used to acquire each of the first image IMG1 and the second image IMG2, the focal length, etc. At this time, the image acquisition device 1000 may superimpose the image of one sensor (e.g., the first image of the first image sensor) on the image of the other sensor (e.g., the second image of the second image sensor). For example, the image acquisition device 1000 may retrieve the pixel corresponding to each pixel of the first image IMG1 in the second image IMG2 for the first image IMG1 acquired by the first image sensor 100. To this end, the image acquisition device 1000 may perform scaling, translation, rotation, affine transformation, perspective transformation, etc. on the pixels of the second image IMG2.

[0138] The image acquisition device 1000 may form color components of the third image IMG3 by replacing color components of the first image IMG1 with color components of the second image IMG2 in the first image IMG1 (S1440). The image acquisition device 1000 uses the first image IMG1 as an input image and the second image IMG2 as a reference image, and changes the color of the first image IMG1 to match the color of the second image IMG2, and forms the third image IMG3. In an embodiment, the color transfer unit 530 forms the color components of the third image IMG3 by matching the color statistics of the first image IMG1 with the color statistics of the second image IMG2 (or replacing the color statistics of the first image IMG1 with the color statistics of the second image IMG2). The color statistics may be based on an average value for peripheral pixels adjacent to a central pixel and / or a standard deviation value for peripheral pixels adjacent to a central pixel.

[0139] The image acquisition apparatus 1000 forms color components of the fourth image IMG4 by synthesizing color components of the first image IMG1 and color components of the third image IMG3 using alpha blending ( S1450 ).

[0140] The brightness component of the fourth image IMG4 is the same as the brightness component of the third image IMG3 , and uses the brightness component value of the first image IMG1 as it is.

[0141] The image acquisition device 1000 forms the color component of the fourth image IMG4 by summing the value obtained by multiplying the color component of the first image IMG1 by the alpha blending coefficient α and the value obtained by multiplying the color component of the third image IMG3 by "1-α". The alpha blending coefficient α may have a value in the range of about 0 or more to about 1 or less. The image acquisition device 1000 obtains the color component of the fourth image IMG4 by summing the value obtained by multiplying the color component of the first image IMG1 by the alpha blending coefficient α and the value obtained by multiplying the color component of the third image IMG3 by "1-α". 1 and the second coefficient α 2 In an example embodiment, the image acquisition device 1000 may determine the value of the alpha blending coefficient α by performing a logical sum of the first coefficient α and the second coefficient α. 1 and the second coefficient α 2 The larger value among them is determined as the value of the alpha mixing coefficient α.

[0142] The first coefficient α 1 is a parameter related to the occlusion area determined according to the parallax between the first image IMG1 and the second image IMG2. 2 is a parameter related to the false color information of the third image IMG3.

[0143] The image acquisition device 1000 described above can be employed in various high-performance optical devices or high-performance electronic devices. The electronic devices may include, for example, smart phones, mobile phones, cellular phones, personal digital assistants (PDAs), laptop computers, PCs, various portable devices, home appliances, security cameras, medical cameras, automobiles, Internet of Things (IoT) devices, other mobile or non-mobile computing devices, but are not limited thereto.

[0144] In addition to the image acquisition device 1000, the electronic device may further include a processor (e.g., an application processor (AP)) configured to control an image sensor disposed therein, and may control a plurality of hardware or software components by running an operating system or an application program using the processor, and perform various data processing and computing operations. The processor may further include, for example, a graphics processing unit (GPU) and / or an image signal processor. When the processor includes an image signal processor, images (or videos) acquired by the image sensor may be stored and / or output by using the processor.

[0145] Fig.14 is a block diagram showing a schematic structure of an electronic device according to an example embodiment. Fig.14In the network environment ED00, the electronic device ED01 can communicate with another electronic device ED02 via a first network ED98 (for example, a short-range wireless communication network, etc.), and / or can communicate with another electronic device ED04 and / or a server ED08 via a second network ED99 (for example, a long-range wireless communication network, etc.). The electronic device ED01 can communicate with the electronic device ED04 via the server ED08. The electronic device ED01 may include a processor ED20, a memory ED30, an input device ED50, a sound output device ED55, a display device ED60, an audio module ED70, a sensor module ED76, an interface ED77, a tactile module ED79, a camera module ED80, a power management module ED88, a battery ED89, a communication module ED90, a subscriber identification module ED96 and / or an antenna module ED97. Provide Fig.14 The number and arrangement of components of the network environment ED00 shown in FIG. are taken as examples. Fig.14 The components shown in the figure may have additional components, fewer components, different components, or components arranged differently. In addition, in the electronic device ED01, some components (e.g., the display device ED60, etc.) may be omitted and / or another component may be added. Some components may be configured as an integrated circuit. For example, the sensor module ED76 (e.g., a fingerprint sensor, an iris sensor, an illuminance sensor, etc.) may be embedded and implemented in the display device ED60 (e.g., a display, etc.). In addition, when the image acquisition device 1000 includes a spectral function, some functions of the sensor module (e.g., a color sensor, an illuminance sensor) may be implemented in the image acquisition device 1000 itself, rather than in a separate sensor module.

[0146] The processor ED20 can control one or more components (e.g., hardware, software components, etc.) of the electronic device ED01 connected to the processor ED20 by executing software (e.g., program ED40, etc.), and can perform various data processing or operations. As part of the data processing or operation, the processor ED20 can load commands and / or data received from another component (e.g., sensor module ED76, communication module ED90, etc.) to the volatile memory ED32, can process the commands and / or data stored in the volatile memory ED32, and can store the result data in the non-volatile memory ED34. The processor ED20 may include a main processor ED21 (e.g., a central processing unit, an application processor, etc.) and an auxiliary processor ED23 (e.g., a graphics processing unit, an image signal processor, a sensor hub processor, a communication processor, etc.), and the auxiliary processor ED23 can operate independently of the main processor ED21 or operate together with the main processor ED21. The auxiliary processor ED23 can use less power than the main processor ED21 and can perform a specified function.

[0147] The auxiliary processor ED23 can control functions and / or states related to some components (e.g., display device ED60, sensor module ED76, communication module ED90, etc.) in the electronic device ED01 on behalf of the main processor ED21 when the main processor ED21 is in an inactive state (e.g., a sleep state), or together with the main processor ED21 when the main processor ED21 is in an active state (e.g., an application execution state). The auxiliary processor ED23 (e.g., an image signal processor, a communication processor, etc.) can be implemented as a part of another component (e.g., a camera module ED80, a communication module ED90, etc.) that is functionally related thereto.

[0148] The memory ED30 can store various data used by the components of the electronic device ED01 (e.g., the processor ED20, the sensor module ED76, etc.). The data may include, for example, input data and / or output data related to software (e.g., the program ED40, etc.) and its related commands. The memory ED30 may include a volatile memory ED32 and / or a non-volatile memory ED34. The non-volatile memory ED34 may include an internal memory ED36 fixedly installed in the electronic device ED01 and a removable external memory ED38.

[0149] The program ED40 may be stored as software in the memory ED30 , and may include an operating system ED42 , middleware ED44 , and / or applications ED46 .

[0150] The input device ED50 may receive commands and / or data to be used in components (e.g., processor ED20, etc.) of the electronic device ED01 from outside (e.g., user, surrounding environment, etc.). The input device ED50 may include a microphone, a mouse, a keyboard, and / or a digital pen (e.g., a stylus), a button, a switch, a camera, a virtual reality (VR) headset, tactile gloves, etc.

[0151] The sound output device ED55 can output sound signals to the outside of the electronic device ED01. The sound output device ED55 may include a speaker and / or a receiver, a speaker, a buzzer, an alarm, etc. The speaker may be used for general purposes such as multimedia reproduction or recording playback, and the receiver may be used to receive calls. The receiver may be coupled as part of the speaker, or may be implemented as an independent device.

[0152] The display device ED60 can provide visual information to the outside of the electronic device ED01. The display device ED60 may include a display (e.g., a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), etc.), a holographic device or a projector, and a control circuit for controlling a corresponding device. The display device ED60 may include a touch circuit configured to sense a touch and / or a sensor circuit configured to measure the strength of a force generated by a touch (e.g., a pressure sensor, etc.).

[0153] The audio module ED70 can convert sound into an electrical signal, and vice versa. The audio module ED70 can obtain sound through the input device ED50, or can output sound via the sound output device ED55, and / or a speaker and / or earphone of another electronic device (e.g., electronic device ED02, etc.) directly or wirelessly connected to the electronic device ED01.

[0154] The sensor module ED76 can sense the operating state (e.g., power, temperature, etc.) of the electronic device ED01, or the external environment state (e.g., user state, brightness level, time of day, geographic location, etc.), and can generate an electrical signal and / or data value corresponding to the sensed state. The sensor module ED76 can include a gesture sensor, a gyroscope sensor, an accelerometer, a pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) ray sensor, an in-vivo sensor, a temperature sensor, a humidity sensor, an illumination sensor, an actuator, a transducer, a contact sensor, a distance measuring device, a global positioning system (GPS) sensor, etc.

[0155] Interface ED77 may support one or more specified protocols, which may be used to connect electronic device ED01 to another electronic device (e.g., electronic device ED02, etc.) directly or wirelessly in sequence. Interface ED77 may include a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, and / or an audio interface.

[0156] The connection terminal ED78 may include a connector through which the electronic device ED01 may be physically connected to another electronic device (eg, the electronic device ED02, etc.). The connection terminal ED78 may include an HDMI connector, a USB connector, an SD card connector, and / or an audio connector (eg, a headphone connector, etc.).

[0157] The haptic module ED79 may convert the electrical signal into mechanical stimulation (eg, vibration, motion, etc.) or electrical stimulation that the user may sense through tactile or kinesthetic sensation. The haptic module ED79 may include a motor, a piezoelectric device, and / or an electrical stimulation device.

[0158] The camera module ED80 can capture still images and videos. The camera module ED80 may include the above-mentioned image acquisition device 10000, an additional lens assembly, an image signal processor and / or a flash. The lens assembly included in the camera module ED80 may collect light emitted from an object as an object to be captured.

[0159] The power management module ED88 may manage power supplied to the electronic device ED01. The power management module ED88 may be implemented as a part of a power management integrated circuit (PMIC).

[0160] The battery ED89 can supply power to the components of the electronic device ED01. The battery ED89 can include a non-rechargeable primary battery, a rechargeable secondary battery, and / or a fuel cell.

[0161] The communication module ED90 can support the establishment of a direct (e.g., wired) communication channel and / or a wireless communication channel between the electronic device ED01 and another electronic device (e.g., electronic device ED02, electronic device ED04, server ED08, etc.), and perform communication through the established communication channel. The communication module ED90 can operate independently of the processor ED20 (application processor, etc.), and can include one or more communication processors that support direct communication and / or wireless communication. The communication module ED90 can include a wireless communication module ED92 (e.g., a cellular communication module (e.g., fifth generation (5G), long term evolution (LTE), code division multiple access (CDMA), etc.), a short-range wireless communication module (e.g., FlashLinQ, WiMedia, Bluetooth™, Bluetooth™ Low Energy (BLE), ZigBee, Institute of Electrical and Electronics Engineers (IEEE) 802.11x (Wi-Fi), etc.), a global navigation satellite system (GNSS) communication module) and / or a wired communication module ED94 (e.g., a local area network (LAN) communication module, a power line communication module, an IEEE 1094 (FireWire) module, etc.). In the communication module, the corresponding communication module can communicate with another electronic device via a first network ED09 (for example, a short-range communication network (for example, Bluetooth™, Wi-Fi Direct, or Infrared Data Association (IrDA))) or a second network ED99 (for example, a long-range communication network (for example, a cellular network, the Internet, or a computer network (for example, LAN, WAN, etc.))). The above-mentioned various communication modules can be integrated into one component (for example, a single chip, etc.), or can be implemented as multiple components independent of each other (for example, multiple chips). The wireless communication module ED92 can identify and authenticate the electronic device ED01 in the communication network (for example, the first network ED98 and / or the second network ED99) by using the subscriber information (for example, the International Mobile Subscriber Identifier (IMSI), etc.) stored in the subscriber identification module ED96.

[0162] The antenna module ED97 can send signals and / or power to the outside (for example, another electronic device, etc.), or receive signals and / or power from the outside. The antenna may include a radiator formed as a conductive pattern formed on a substrate (for example, a printed circuit board (PCB), etc.). The antenna module ED97 may include one or more antennas. When the antenna module ED97 includes multiple antennas, an antenna suitable for the communication type used in the communication network (for example, the first network ED98 and / or the second network ED99) can be selected from the multiple antennas by the communication module ED90. Signals and / or power can be transmitted between the communication module ED90 and another electronic device via the selected antenna. Other components other than the antenna (for example, a radio frequency integrated circuit (RFIC), etc.) may be included as part of the antenna module ED97.

[0163] Some components may be connected to each other via a communication method between peripheral devices (eg, bus, general purpose input and output (GPIO), serial peripheral interface (SPI), mobile industry processor interface (MIPI), etc.) and may exchange signals (eg, commands, data, etc.).

[0164] Commands and / or data can be sent and / or received between the electronic device ED01 and the external electronic device ED04 via the server ED08 connected to the second network ED99. The other electronic devices ED02 and ED04 can be devices of the same or different types as the electronic device ED01. All or some operations performed in the electronic device ED01 can be performed in one or more of the other electronic devices ED02, ED04 and ED08. For example, when the electronic device ED01 wants to perform a certain function or service, the electronic device ED01 can request one or more other electronic devices to perform some or all functions or services instead of performing the functions or services itself. The one or more electronic devices that receive the request perform additional functions or services related to the request, and can transmit the execution results to the electronic device ED01. To this end, for example, cloud computing, distributed computing, or client-server computing technology can be used.

[0165] Fig.15 is a schematic diagram showing the arrangement of Fig.14 The camera module ED80 in the electronic device ED01 is a block diagram of the camera module ED80. The camera module ED80 may include the above-mentioned image acquisition device 1000, or may have a structure modified therefrom. Fig.15 The camera module ED80 may include a lens assembly CM10 , a flash CM20 , an image sensor CM30 , an image stabilizer CM40 , a memory CM50 (eg, a buffer memory, etc.) and / or an image signal processor CM60 .

[0166] The image sensor CM30 may include the first image sensor 100 and the second image sensor 200 provided in the image acquisition device 1000. The first image sensor 100 and the second image sensor 200 may convert light emitted or reflected from an object and transmitted through the lens assembly CM10 into an electrical signal, thereby acquiring an image corresponding to the object. The first image sensor 100 may acquire an RGB image, and the second image sensor 200 may acquire a hyperspectral image in the ultraviolet to infrared wavelength range.

[0167] The image sensor CM30 may further include one or more sensors (e.g., an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor) selected from image sensors having different properties in addition to the first image sensor 100 and the second image sensor 200. Each sensor included in the image sensor CM30 may be implemented by a CCD sensor and / or a CMOS sensor.

[0168] The lens assembly CM10 may collect light emitted from an object for image capture. The camera module ED80 may include a plurality of lens assemblies CM10, and in this case, the camera module ED80 may include, for example, a dual camera, a 360-degree camera, or a spherical camera. Some lens assemblies CM10 may have the same lens properties (e.g., viewing angle, focal length, autofocus, F number, optical zoom, etc.), or different lens properties. The lens assembly CM10 may include a wide-angle lens or a telephoto lens.

[0169] The lens assembly CM10 may be configured and / or focus-controlled so that two image sensors included in the image sensor CM30 may form an optical image of a subject at the same position.

[0170] The flash CM20 may emit light for enhancing light emitted or reflected from an object and may include one or more light emitting diodes (eg, red, green, and blue (RGB) LEDs, white light LEDs, infrared LEDs, ultraviolet LEDs, etc.) and / or a xenon lamp.

[0171] In response to the movement of the camera module ED80 or the electronic device ED01 including the camera module ED80, the image stabilizer CM40 moves one or more lenses included in the lens assembly CM10 and / or the image sensor CM30 in a specific direction, or controls the operating characteristics of the image sensor CM30 (e.g., adjusting the readout timing, etc.) to compensate for the negative effects of the movement. The image stabilizer CM40 can sense the movement of the camera module ED80 or the electronic device ED01 by using a gyro sensor (not shown) or an acceleration sensor (not shown) arranged inside or outside the camera module ED80. The image stabilizer CM40 can be implemented as an optical type.

[0172] The memory CM50 can store part or all of the data of the image acquired by the image sensor 20 for subsequent image processing operations. For example, when acquiring multiple images at high speed, only the low-resolution image is displayed while the acquired raw data (e.g., Bayer pattern data, high-resolution data, etc.) is stored in the memory CM50. The memory CM50 can then be used to send the raw data of the selected (e.g., user selected, etc.) image to the image signal processor CM60. The memory CM50 can be incorporated into the memory ED30 of the electronic device ED01, or configured as a separate memory that operates independently.

[0173] The image signal processor CM60 may perform image processing on an image acquired by the image sensor CM30 or image data stored in the memory CM50. Figures 1 to 13 As described, a third image having an extended color gamut may be formed by processing a first image (eg, an RGB image) and a second image (eg, a hyperspectral image) acquired by two image sensors included in the image sensor CM30. Figure 1 The described configuration of the processor 500 may be included in the image signal processor CM60 .

[0174] The image processing performed by the image signal processor CM60 may include, for example, but not limited to, depth map generation, three-dimensional modeling, panorama generation, feature point extraction, image synthesis, and / or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, softening, etc.). The image signal processor CM60 may perform control (e.g., exposure time control, or readout timing control, etc.) on the constituent elements (e.g., image sensor CM30, etc.) included in the camera module ED80. The image processed by the image signal processor CM60 may be stored again in the memory CM50 for additional processing, and / or provided to external constituent elements (e.g., memory ED30, display device ED60, electronic device ED02, electronic device ED04, server ED08, etc.) of the camera module ED80. The image signal processor CM60 may be incorporated into the processor ED20, or may be configured as a separate processor that operates independently of the processor ED20. When the image signal processor CM60 is configured as a processor separate from the processor ED20 , the image processed by the image signal processor CM60 may be subjected to additional image processing by the processor ED20 and then displayed by the display device ED60 .

[0175] The electronic device ED01 may include a plurality of camera modules ED80 having different properties or functions. In this case, one of the camera modules ED80 may be a wide-angle camera, and another camera module may be a telephoto camera. Similarly, one of the camera modules ED80 may be a front camera, and another camera module may be a rear camera.

[0176] FIG. 16A to FIG. 16E as well as FIG. 17A to FIG. 17E Various examples of electronic devices to which the image acquisition device according to the example embodiments is applied are shown.

[0177] The image acquisition device 1000 according to some embodiments may be applied to Fig.16A The mobile phone or smart phone 5100m shown, Fig. 16B The tablet computer or smart tablet computer 5200 shown, Fig. 16C The digital camera or video camera 5300 shown, Fig.16D The notebook computer 5400 shown, Fig.16E The television or smart TV 5500 shown in the figure. For example, the smart phone 5100m or the smart tablet computer 5200 may include a plurality of high-resolution cameras, each of which has a high-resolution image sensor mounted thereon. The depth information of the object in the image can be extracted by using the high-resolution camera, the defocus of the image can be adjusted, or the object in the image can be automatically recognized.

[0178] In addition, the image acquisition device 1000 can be applied to Fig.17A The smart refrigerator 5600 shown, Fig. 17B The security camera 5700 shown, Fig. 17C The robot 5800 shown, Fig.17D Medical camera 5900 shown, etc. For example, smart refrigerator 5600 can automatically identify food in the refrigerator by using image acquisition device 1000, and notify the user through a smart phone of the presence of specific food, the type of food put in or taken out, etc. Security camera 5700 can provide ultra-high resolution images, and can identify objects or people in images in dark environments by using high sensitivity. Robot 5800 can be set up at a disaster site or industrial site that people cannot directly enter, and can provide high-resolution images. Medical camera 5900 can provide high-resolution images for diagnosis or surgery, and thus the field of view can be dynamically adjusted.

[0179] In addition, the image acquisition device 1000 can be applied to Fig.17EThe vehicle 6000 shown. The vehicle 6000 may include a plurality of vehicle-mounted cameras 6010, 6020, 6030, and 6040 arranged at various locations. Each of the vehicle-mounted cameras 6010, 6020, 6030, and 6040 may include the image acquisition device 1000 according to the example embodiment. The vehicle 6000 may provide the driver with various information related to the interior or periphery of the vehicle 6000 by using the vehicle-mounted cameras 6010, 6020, 6030, and 6040. For example, the vehicle 6000 may provide information related to an object or person in an image, which may be automatically recognized and used for autonomous driving.

[0180] It should be understood that the embodiments described herein should be considered in a descriptive sense only and not for restrictive purposes. The description of features or aspects in each embodiment should be considered as other similar features or aspects that can be used in other embodiments. Although one or more example embodiments have been described with reference to the accompanying drawings, it should be understood by those skilled in the art that various changes in form and details can be made without departing from the spirit and scope defined by the appended claims.

Claims

1. An image acquisition device, comprising: A first image sensor having a first spatial resolution and configured to acquire a first image based on a first wavelength band; a second image sensor having a second spatial resolution lower than the first spatial resolution and configured to acquire a second image based on a second wavelength band wider than the first wavelength band; as well as The processor is configured as: registering the first image and the second image in association with each other by using relative position information between the first image sensor and the second image sensor; generating a third image, wherein color components of the third image are determined by replacing color components of the first image with color components of the second image; and A fourth image is generated, wherein color components of the fourth image are determined by synthesizing color components of the first image and color components of the third image using alpha blending.

2. The image acquisition device according to claim 1, wherein: The processor is further configured to determine the color components of the fourth image by summing a value obtained by multiplying the color components of the first image by an alpha blending coefficient α and a value obtained by multiplying the color components of the third image by (1−α), and The alpha blending coefficient α has a value greater than or equal to 0 and less than or equal to 1.

3. The image acquisition device according to claim 1, wherein: The processor is further configured to calculate parameters for registering the first image and the second image based on at least one of relative position information, resolution, field of view, and focal length of each of the first image sensor and the second image sensor.

4. The image acquisition device according to claim 1, wherein: The third image has the first spatial resolution and a color gamut corresponding to the color gamut of the second image.

5. The image acquisition device according to claim 1, wherein: The processor is also configured to determine color components of the third image by matching color statistics of the first image to color statistics of the second image.

6. The image acquisition device according to claim 5, wherein: The color statistics correspond to an average value for peripheral pixels adjacent to the central pixel.

7. The image acquisition device according to claim 5, wherein: The color statistics correspond to standard deviation values ​​for peripheral pixels adjacent to a central pixel.

8. The image acquisition device according to claim 6, wherein: The processor is further configured to calculate the average value by weighting by similarity between the central pixel and the peripheral pixels.

9. The image acquisition device according to claim 7, wherein: The processor is further configured to calculate the standard deviation value by weighting the standard deviation using a similarity between the central pixel and the peripheral pixels.

10. The image acquisition device according to claim 8 or 9, wherein: The similarity is calculated by using the brightness value of the central pixel and the brightness values ​​of the peripheral pixels as feature vectors.

11. The image acquisition device according to claim 2, wherein: The processor is further configured to: determine the alpha blending coefficient α as a logical sum of a first coefficient and a second coefficient, The first coefficient represents information about an occlusion area according to a disparity between the first image and the second image, and The second coefficient represents false color information of a color component of the third image.

12. The image acquisition device according to claim 11, wherein: The processor is further configured to: When there is no pixel in the second image corresponding to the pixel of interest of the first image, setting a first coefficient of the pixel of interest of the first image to 1; as well as When at least one pixel in the second image corresponds to the pixel of interest of the first image, a first coefficient of the pixel of interest is set to zero.

13. The image acquisition device according to claim 11, wherein: The processor is further configured to determine a larger value between the first coefficient and the second coefficient as a value of the alpha mixing coefficient α.

14. The image acquisition device according to claim 11, wherein: The second coefficient is proportional to a color difference between the first image and the third image, and has a value greater than or equal to 0 and less than or equal to 1.

15. The image acquisition device according to claim 11, wherein: The second coefficient has a value that is closer to 0 as the saturation of the first image increases, or a value that is closer to 1 as the saturation of the first image decreases.

16. The image acquisition device according to claim 1, wherein: The processor is further configured to: separate the first image and the second image into a brightness component and a color component, respectively, and The third image and the fourth image are generated by using the separated first image and the separated second image.

17. The image acquisition device according to claim 1, wherein: The processor is also configured to align the first image and the second image by extracting at least one of edge features or corner features from the first image and the second image, and matching at least one of the extracted edge features or corner features between the first image and the second image.

18. The image acquisition device according to claim 11, wherein: The processor is further configured to: The second coefficient is set based on a difference in brightness values ​​between corresponding pixels of the first image and the second image so that the second coefficient becomes closer to 1 as the difference increases or becomes closer to 0 as the difference decreases.

19. An electronic device comprising the image acquisition device according to claim 1.

20. A method for controlling an image acquisition device, the method comprising: acquiring a first image from a first image sensor and acquiring a second image from a second image sensor; registering the acquired first image and the acquired second image in association with each other; generating a third image, wherein color components of the third image are determined by replacing color components of the first image with color components of the second image; and A fourth image is generated, wherein color components of the fourth image are determined by synthesizing color components of the first image and color components of the third image using alpha blending.

Images