An image acquisition device including a plurality of image sensors and an electronic device including the image acquisition device

Abstract

An image acquisition device, comprising: a first image sensor configured to acquire a first image based on a first band; a second image sensor configured to acquire a second image based on a second band of 10 nm to 1000 nm; and a processor configured to register the first image and the second image respectively output from the first image sensor and the second image sensor to obtain a registered image based on the first image and the second image, and perform color conversion on the registered image by using an illuminance value estimated from the second image.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority based on and claims the benefit of Korean Patent Application No. 10 - 2021 - 0113981, filed on August 27, 2021, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical field

[0003] The present disclosure relates to an image acquisition device including a plurality of image sensors, and an electronic device including the image acquisition device. Background art

[0004] An image sensor receives light incident from an object and performs photoelectric conversion on the received light to generate an electrical signal.

[0005] Such an image sensor uses a color filter including an array of filter elements that can selectively transmit red light, green light, and blue light for color representation, senses the amount of light passing through each filter element, and then forms a color image of the object through image processing.

[0006] Since the values sensed by the image sensor are affected by illumination, the colors of the images captured by a camera are also affected by illumination. A technique for eliminating this influence and capturing the unique color of an object as much as possible is called "white balance".

[0007] In the related art, an RGB image is first captured, and then white balance is performed by analyzing the information included in the RGB image. However, since this method is based on the gray - world assumption (i.e., assuming that the average values of the R, G, and B channel values are equal to each other) or has other limiting conditions, when the assumptions or limiting conditions are not met, this method may not work correctly. Summary of the invention

[0008] An image acquisition device including a plurality of image sensors and an electronic device including the image acquisition device are provided.

[0009] Additional aspects will be set forth in part in the following description, and in part will become apparent from the description, or may be learned by practice of the embodiments of the present disclosure.

[0010] According to one aspect of the present disclosure, an image acquisition device includes: a first image sensor configured to acquire a first image based on a first wavelength band; a second image sensor configured to acquire a second image based on a second wavelength band of 10 nm to 1000 nm; and a processor configured to perform registration on the first image and the second image respectively output from the first image sensor and the second image sensor to obtain a registered image; and perform color conversion on the registered image using the illuminance value estimated from the second image.

[0011] The processor may further be configured to: divide the first image into one or more first regions, and divide the second image into one or more second regions respectively corresponding to the one or more first regions; estimate a corresponding illuminance value for each of the one or more second regions; and perform color conversion on each of the one or more first regions using the estimated illuminance value.

[0012] The processor may further be configured to: when the difference between the illuminance values respectively estimated for adjacent second regions among the one or more second regions is greater than or equal to a first threshold, adjust any one of the illuminance values of the adjacent second regions to adjust the difference to be less than the first threshold.

[0013] The processor may further be configured to: perform post-processing on a boundary portion between adjacent second regions after performing color conversion.

[0014] The processor may further be configured to calculate parameters for registering the first image and the second image based on at least one of the resolution, field of view, and focal length of each of the first image sensor and the second image sensor.

[0015] The processor may further be configured to estimate the illuminance value by using spectral information obtained from a plurality of channels output from the second image sensor.

[0016] The processor may further be configured to estimate the illuminance value by using a neural network trained with a plurality of second images associated with a predetermined illuminance value.

[0017] The processor may further be configured to register the first image and the second image by respectively extracting a first feature from the first image and a second feature from the second image, and matching the extracted first feature with the extracted second feature.

[0018] The processor may further be configured to register the first image and the second image in units of pixel groups, pixels, or sub-pixels of the first image and the second image.

[0019] The first image sensor may include: a first pixel array, the first array including: a first sensor layer in which a plurality of first sensing elements are arranged; and a color filter on the first sensor layer and including alternately arranged red, green, and blue color filters, wherein the second image sensor includes a second pixel array, the second pixel array including: a second sensor layer in which a plurality of second sensing elements are arranged; and a spectral filter on the second sensor layer and in which a filter bank is repeatedly arranged, the filter bank including a plurality of unit filters, each unit filter having a transmission band different from every other unit filter of the filter bank.

[0020] The combined transmission band of the plurality of unit filters of the filter bank may include the visible light band and be wider than the visible light band, and the plurality of unit filters may include 16 unit filters arranged in a 4×4 array.

[0021] The first pixel array and the second pixel array may be horizontally separated from each other on the circuit board.

[0022] A first circuit element configured to process signals from the first sensor layer and a second circuit element configured to process signals from the second sensor layer may be provided on the circuit board.

[0023] The image acquisition device may further include a timing controller configured to synchronize the operation of the first circuit element with the operation of the second circuit element.

[0024] The image acquisition device may further include: a first memory storing data about the first image; and a second memory storing data about the second image.

[0025] The first memory and the second memory may be provided inside the circuit board.

[0026] The image acquisition device may further include: a first imaging optical system configured to form a first optical image of an object on the first image sensor, the first imaging optical system including at least one first lens; and a second imaging optical system configured to form a second optical image of the object on the second image sensor, the second imaging optical system including at least one second lens.

[0027] The first imaging optical system and the second imaging optical system may have the same focal length and the same field of view.

[0028] An electronic device may include the image acquisition device of the above aspects of the present disclosure.

[0029] According to one aspect of the present disclosure, a method of controlling an image acquisition device including a plurality of image sensors includes: acquiring a first image and a second image from a first image sensor and a second image sensor, respectively; registering the acquired first image and second image to obtain a registered image; and performing color conversion on the registered image using an illuminance value estimated from the second image.

[0030] According to one aspect of the present disclosure, an image acquisition device includes: a first image sensor including a first filter; a second image sensor including a second filter different from the first filter; and a processor configured to receive a first image of an object from the first image sensor and a second image of the object from the second image sensor; and generate a white-balanced first image of the object by eliminating the influence of illumination reflected by the object based on the received first image and the received second image.

[0031] The first image sensor may include a plurality of first pixels, wherein the first filter may include a plurality of first filter groups arranged repeatedly, each first filter group including a plurality of first unit filters, each first unit filter corresponding to a corresponding first pixel, wherein the second image sensor may include a plurality of second pixels, the second filter may include a plurality of second filter groups arranged repeatedly, each second filter group including a plurality of second unit filters, each second unit filter corresponding to a corresponding second pixel, wherein each first filter group of the first filter corresponds to a corresponding second filter group of the second filter, and the wavelength band of each first unit filter may be larger than the wavelength band of each second unit filter.

[0032] The combined wavelength band of the first filter groups may be smaller than the combined wavelength band of the second filter groups.

[0033] The processor may also be configured to: generate a conversion matrix based on the second image for eliminating the influence of illumination reflected by the object; and generate a white-balanced first image based on the conversion matrix and the first image.

[0034] The processor may also be configured to: divide the first image into portions corresponding to the plurality of first filter groups, and divide the second image into portions corresponding to the plurality of second filter groups; generate a corresponding conversion matrix for each portion of the second image; and generate portions of the white-balanced first image based on the respective conversion matrices and the respective portions of the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and other aspects, features, and advantages of some embodiments of the present disclosure will become clearer from the following description in conjunction with the accompanying drawings, in which:

[0036] Figure 1is a block diagram showing a schematic structure of an image acquisition device according to an embodiment;

[0037] Figure 2 is Figure 1 a detailed block diagram of the image acquisition device shown;

[0038] Figure 3 is a conceptual diagram showing Figure 1 the schematic structure of the image acquisition device shown;

[0039] Figure 4 is a view showing a circuit configuration of a first image sensor and a second image sensor provided in the image acquisition device according to an embodiment;

[0040] Figure 5 is a graph showing a wavelength spectrum obtained by using the first image sensor provided in the image acquisition device according to an embodiment;

[0041] Figures 6 to 8 is a view showing an example of a pixel arrangement of the first image sensor provided in the image acquisition device according to an embodiment;

[0042] Figure 9 is a graph showing a wavelength spectrum obtained by using the second image sensor provided in the image acquisition device according to an embodiment;

[0043] Figures 10 to 12 is a view showing an example of a pixel arrangement of the second image sensor provided in the image acquisition device according to an embodiment;

[0044] Figure 13 is a flowchart showing a method of controlling an image acquisition device according to an embodiment;

[0045] Figure 14 is a block diagram showing a schematic structure of an electronic device according to an embodiment;

[0046] Figure 15 is a view showing Figure 14 a block diagram of a camera module included in the electronic device; and

[0047] Figures 16 to 25 is a view showing various examples of an electronic device including an image acquisition device according to an embodiment. Detailed Description

[0048] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. In this regard, the embodiments may have different forms and should not be construed as limited to the description set forth herein. Accordingly, the embodiments are described only by way of reference to the drawings to explain the various aspects. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of..." modify the entire list of elements when following the list of elements, rather than modifying individual elements in the list.

[0049] Hereinafter, embodiments will be described with reference to the accompanying drawings. The embodiments described below are merely examples, and thus it should be understood that the embodiments may be modified in various forms. In the drawings, like reference numerals always refer to like elements, and the dimensions of the elements may be exaggerated for clarity.

[0050] In the following description, when an element is referred to as being "on" or "above" another element, it may be directly on the other element while in contact with the other element, or it may be above the other element without contacting the other element.

[0051] Although terms such as "first" and "second" are used to describe various elements, these terms are only used to distinguish one element from another. These terms do not limit the elements to having different materials or structures.

[0052] As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. Further, it should be understood that when a unit is referred to as "including" another element, there is a possibility that one or more other elements may exist or may be added.

[0053] In the present disclosure, terms such as "unit" or "module" may be used to represent a unit having at least one function or operation, and may be implemented by hardware, software, or a combination of hardware and software.

[0054] An element referred to by a definite article or a demonstrative pronoun may be construed as one or more elements, even if it is in the singular form.

[0055] Unless explicitly described in terms of order or described to the contrary, the operations of the method may be performed in an appropriate order. Additionally, example or exemplary terms (e.g., "such as" and "etc.") are used for descriptive purposes and are not intended to limit the scope of the present disclosure unless defined by the claims.

[0056] Generally, the value sensed by the camera can be expressed as the product of illuminance, object color, and the response of the camera, as shown in Equation 1 below.

[0057] [Equation 1]

[0058]

[0059] Here, ρ represents the sensed value, and E(λ), S(λ), and R(λ) represent illuminance, object surface reflectance, and camera response as functions of the spectrum λ, respectively. Since the value sensed by the camera is affected by illumination, the color of the image captured by the camera is also affected by illumination. White balance is a technique for eliminating such effects and capturing the unique color of the object as maximally as possible.

[0060] A photographing device such as a smartphone senses light in three spectra, namely R, G, and B spectra. Then, the photographing device uses the three sensed values to perform conversion to represent colors as expected. In the method of the related art, first, an RGB image is captured, and then white balance is performed by analyzing the information included in the RGB image. However, such methods are based on the gray world assumption or have other limiting conditions, and thus may not work correctly when the gray world assumption or limiting conditions are not satisfied.

[0061] In addition, in the method of the related art, first, an image is obtained using an RGB camera, and then white balance is performed using the information about the image. Since an RGB camera can basically sense the spectra of three colors and uses broadband filters for each color, there are limitations in obtaining accurate spectral information by using an RGB camera. Therefore, there are limitations in performing accurate white balance. This limitation occurs especially when an object is illuminated by more than one light source, and the techniques of the related art have limitations in separating the illuminations from each other. For example, when a person is photographed with natural light coming in from a left window and illumination light coming in from a right indoor light, the left and right sides of the face may have different colors. In addition, errors may occur when a specific color dominates in the scene. For example, in a photograph of a ski resort with snow as the background, the color of the snow may look different from its actual color.

[0062] For accurate white balance, the image acquisition device according to an embodiment can separate illumination from the color of the object by using a multi-spectral image (MSI) sensor to find the accurate color of the object, and then can perform color conversion or mapping on the RGB image by using the information about the illumination (i.e., illumination value) obtained by using the MSI sensor to perform white balance.

[0063] Figure 1 is a block diagram showing a schematic structure of an image acquisition device according to an embodiment.

[0064] Refer to Figure 1, the image acquisition device includes a first image sensor 100, a second image sensor 200, and a processor 500. The image acquisition device in this embodiment accurately performs white balance on the images captured by multiple image sensors. The first image sensor 100 acquires a first image in a first band. The second image sensor 200 acquires a second image in a second band. The second band may include the first band and may be wider than the first band. Here, the first image sensor 100 may include an RGB image sensor, and the second image sensor 200 may include an MSI sensor. The RGB image sensor has an R channel, a G channel, and a B channel. The MSI sensor has more channels than the RGB image sensor, so it senses light in more bands than the RGB image sensor.

[0065] The processor 500 registers the first image and the second image respectively output from the first image sensor 100 and the second image sensor 200 to obtain a registered image, and performs color conversion on the registered image by using the illuminance value estimated from the second image.

[0066] In addition, the processor 500 may divide the first image into one or more regions, estimate the illuminance values for the regions of the second image corresponding to the regions of the first image respectively, and may perform color conversion on the regions of the first image by using the illuminance values estimated for the regions of the second image respectively.

[0067] In addition, when the difference between the illuminance values estimated for adjacent regions of the second image is equal to or greater than a first threshold, the processor 500 may adjust one of the illuminance values of the adjacent regions to reduce the difference to a value smaller than the first threshold. In this case, the processor 500 may perform post-processing on the boundary portion between the adjacent regions after color conversion.

[0068] Figure 2 is Figure 1 a detailed block diagram of the image acquisition device shown.

[0069] See Figure 2 , the image acquisition device includes: a first image sensor 100 configured to acquire a first image IM1 based on a first band; a second image sensor 200 configured to acquire a second image IM2 based on a second band; a processor 500 configured to generate a third image IM3 by performing signal processing on the first image IM1 and the second image IM2. Here, the third image IM3 is generated by performing white balance on the first image IM1 acquired using the first image sensor 100, or by performing white balance on the image obtained by registering the first image IM1 acquired from the first image sensor 100 and the second image IM2 acquired from the second image sensor 200.

[0070] The first image sensor 100 may be a sensor used in a normal RGB camera, such as a complementary metal oxide semiconductor (CMOS) image sensor including a Bayer color filter array. The first image IM1 obtained using the first image sensor 100 may be an RGB image based on red, green, and blue. The first image sensor 100 typically has a bandwidth of 380 nm to 780 nm.

[0071] The second image sensor 200 may be a sensor capable of sensing light of more wavelengths than the first image sensor 100. The second image sensor 200 may use, for example, 16 channels, 31 channels, or other numbers of channels. The second image sensor 200 may have more channels than the first image sensor 100. The bandwidth of each channel may be set to be narrower than the R, G, and B bandwidths, and the total bandwidth as the sum of the bandwidths of all channels may include the RGB bandwidth and may be wider than the RGB bandwidth, that is, the total bandwidth (e.g., the combined transmission band) may include the visible light band and may be wider than the visible light band. For example, the second image sensor 200 may have a bandwidth of 10 nm to 1000 nm. In addition, the second image sensor may have a bandwidth of approximately 350 nm to 1000 nm. The second image IM2 obtained using the second image sensor 200 may be a multispectral or hyperspectral image, and may be an image based on wavelengths in 16 or more channels divided by a band wider than the RGB band (e.g., a band including the visible light band and ranging from the ultraviolet band to the infrared band). All available channels of the second image sensor 200 may be used to obtain the second image IM2, or selected channels of the second image sensor 200 may be used to obtain the second image IM2. The spatial resolution of the second image IM2 may be lower than the spatial resolution of the first image IM1, but is not limited thereto.

[0072] In an embodiment, the first image sensor 100 may include an RGB image sensor, and the second image sensor 200 may include an MSI sensor. In this case, the RGB image sensor may be a CMOS image sensor. The RGB image sensor may use a Bayer color filter array to generate a three-channel image by sensing spectra respectively representing R, G, and B. However, the RGB sensor may use other types of color filter arrays. The MSI sensor may sense and represent light having wavelengths different from those sensed and represented by the RGB image sensor. The MSI sensor has more channels than the RGB image sensor and thus can sense more wavelengths. For example, the MSI sensor may have 16 channels. In another example, the MSI sensor may have 31 channels. The transmission band, transmission amount, and transmission bandwidth of each channel may be adjusted to sense light in a desired band. The total bandwidth, which is the sum of the bandwidths of all channels, may include the bandwidth of a general RGB image sensor and may be wider than the bandwidth of a general RGB image sensor. The sensing spectra or bands of the RGB image sensor and the MSI sensor will be described later with reference to Figure 5 and Figure 9 describe the sensing spectra or bands of the RGB image sensor and the MSI sensor.

[0073] The first image sensor 100 and the second image sensor 200 may be provided as separate chips or a single chip.

[0074] In an embodiment, timing control may be performed according to the different resolutions and output speeds of different types of sensors and the size of the region required for image registration. For example, when reading an RGB image column in an operation based on the RGB image sensor, the image column corresponding to the RGB image column of the MSI sensor may be pre-stored in a buffer or may be re-read. Signals sensed by calculating such timing may be read out. Alternatively, the same synchronization signal may be used to synchronize the operations of the two sensors. In addition, focus control may be performed to focus the two sensors on the same position of an object.

[0075] In an embodiment, the MSI sensor may acquire an image through all channels (e.g., 16 channels) or through specific channels. Only specific channels may be used by combining sensor pixels, or by selecting or synthesizing specific channels.

[0076] Referring to Figure 2 , the first memory 300 stores a first image IM1 read out from the first image sensor 100. The second memory 310 stores a second image IM2 read out from the second image sensor 200.

[0077] Images are read out line by line from the first image sensor 100 and the second image sensor 200 and stored sequentially. The first memory 300 and the second memory 310 can be line memories for storing images line by line or frame buffers for storing entire images.

[0078] In an embodiment, when outputting an image, only an RGB image can be output, and the RGB image can be stored in the frame buffer. In this case, the MSI image can be stored in the line buffer and processed line by line, and then the RGB image in the frame buffer can be updated. The first memory 300 and the second memory 310 can be static random access memories (SRAMs) or dynamic random access memories (DRAMs). However, the types of the first memory 300 and the second memory 310 are not limited.

[0079] The first memory 300 and the second memory 310 can be provided outside the first image sensor 100 and the second image sensor 200, or can be integrated into the first image sensor 100 and the second image sensor 200. In the latter case, a method of integrating the memory into the sensor can be used by stacking each with pixel units, circuit units, and memories and integrating two stacks into one chip. Alternatively, three layers respectively including pixel units, circuit units, and memories can be formed into three stacks.

[0080] In the above embodiment, the first image IM1 and the second image IM2 respectively acquired using the first image sensor 100 and the second image sensor 200 are stored in different memories (i.e., the first memory 300 and the second memory 310). However, the embodiment is not limited thereto, and the first image IM1 and the second image IM2 can be stored in one memory.

[0081] The processor 500 includes an image registration unit 510, an illuminance estimation unit 520, and a color conversion unit 530. Although Figure 2 not shown, the processor 500 may further include an image signal processor (hereinafter referred to as ISP). Before or after the images are respectively stored in the first memory 300 and the second memory 310, the ISP can perform basic image processing operations on the images respectively acquired from the first image sensor 100 and the second image sensor 200. For example, the ISP can perform bad pixel correction, fixed pattern noise correction, crosstalk reduction, re-mosaicking, demosaicking, false color reduction, denoising, chromatic aberration correction, etc. In addition, the processor 500 or the ISP can perform the same or different image processing operations on the first image sensor 100 and the second image sensor 200.

[0082] In an embodiment, the processor 500 can perform accurate white balance by separating illumination from the color of an object using an MSI sensor to find the exact color of the object, and then convert the color of the image obtained from the RGB image sensor or the color of the registered image using the illuminance value. Now, the functions of the processor 500 will be described.

[0083] The image registration unit 5 registers the first image IM1 and the second image IM2 output from the first image sensor 100 and the second image sensor 200, respectively.

[0084] The image registration unit 510 can register the first image IM1 and the second image IM2 by using information about the relative positions of the first image sensor 100 and the second image sensor 200. The image registration unit 510 can find the positional relationship between image pixels by considering the spatial resolution of the images obtained using the first image sensor 100 and the second image sensor 200, the field of view and focal length of the optical system used to capture the images, and the like. In this case, the image obtained using one sensor (e.g., the first or second image sensor) can be set as a reference image, and the other image obtained using the other sensor can be superimposed on the reference image. For example, the first image IM1 obtained using the first image sensor 100 can be set as a reference image, and the pixels corresponding to the pixels of the first image IM1 can be found from the pixels of the second image IM2 obtained using the second image sensor 200. To this end, operations such as scaling, translation, rotation, affine transformation, and perspective transformation can be performed on the pixels of the second image IM2.

[0085] In addition, one or more pixels of the second image IM2 can correspond to the pixels of the first image IM1, and the pixel value of the second image IM2 corresponding to the pixel of the first image IM1 can be obtained by mixing the pixels of the second image IM2 at a certain ratio according to the positions of the pixels of the second image IM2. The channel-based image of the second image IM2 can be used for image registration. Image registration can be performed based on sub-pixels to improve the accuracy of image registration. In sub-pixel-based image registration, the position of each pixel can be represented using real numbers instead of integers.

[0086] The image registration unit 510 can also improve the efficiency of image registration by controlling the first image sensor 100 and the second image sensor 200 to focus on the same position of the object. In addition, when the first image sensor 100 and the second image sensor 200 have the same field of view, image registration can be performed quickly and accurately. For example, when the imaging optical systems for forming optical images on the first image sensor 100 and the second image sensor 200 have the same focal length and the same field of view, only translation may occur between the first image IM1 and the second image IM2, and the relative positions of the first image sensor 100 and the second image sensor 200 and the focal length of the imaging optical system can be used to calculate the relevant parameters.

[0087] When the spatial resolution of the second image IM2 is greater than that of the first image IM1, image registration can be performed by downsampling the second image IM2. In this case, filtering considering edge information (such as bilateral filtering or guided filtering) can be used for downsampling to improve the accuracy of image registration.

[0088] When the spatial resolution of the second image IM2 is less than that of the first image IM1, second image samples corresponding to the positions of each pixel of the first image IM1 can be generated for each channel by interpolation. Similarly, interpolation can be performed using bilateral filtering, guided filtering, etc. to consider edge information.

[0089] Alternatively, image registration can be performed after adjusting the spatial resolution of the second image IM2 to be equal to that of the first image IM1. Demosaicing can be performed to adjust the resolutions of the first image IM1 and the second image IM2 to be equal to each other. In this case, when the two optical systems for forming optical images on the first image sensor 100 and the second image sensor 200 have the same focal length and the same field of view, image registration can be performed by only considering translation without considering interpolation. For example, when the first image sensor 100 and the second image sensor 200 have the same focus and the same field of view, only translation may exist between the images obtained using the first image sensor 100 and the second image sensor 200, and the external camera parameters including the relative positions of the first image sensor 100 and the second image sensor 200 and the internal camera parameters including the focal lengths of the first image sensor 100 and the second image sensor 200 can be used to calculate the translation parameters.

[0090] Before image registration, the aberrations of the first image IM1 and the second image IM2 can be corrected. That is, image registration can be performed after correcting the effects of distortion, geometric aberration, chromatic aberration, etc. caused by the lenses of the imaging optical systems for obtaining the first image IM1 and the second image IM2.

[0091] The image registration unit 510 can extract edge feature information from the first image IM1 and the second image IM2, and can perform feature matching between the first image IM1 and the second image IM2. Since color distortion may occur when the image registration in the boundary region of the object is incorrect, image registration can be performed by aligning the edges of the two images using the extracted edge information, thereby preventing distortion in the boundary region between the two images. Image registration can be performed by using image features such as corner points instead of edge features.

[0092] In addition, the image registration unit 510 can perform image registration based on pixel groups rather than on pixels. For example, after classifying the pixels of the RGB image into groups, image registration can be performed by matching the corresponding groups of the MSI image with the pixel groups of the RGB image. Here, the pixel groups can be arranged such that each pixel group has a pattern of a given size and shape. For example, each pixel group can have a linear or rectangular block shape. For example, after dividing the RGB image into rectangular regions and finding the MSI regions corresponding to the rectangular regions, information about the MSI regions can be used to perform white balance, and the information found through white balance can be used to transform the rectangular regions of the RGB image. The pixel groups can be classified into foreground object regions and background regions. In addition, the pixel groups can be classified into regions of interest (hereinafter referred to as ROIs) and non-ROIs. In addition, the pixel groups can be classified by considering image segmentation or color distribution.

[0093] The illuminance estimation unit 520 estimates the illuminance value according to the second image IM2 acquired from the second image sensor 200. The illuminance estimation unit 520 can estimate the illuminance according to an image obtained by using an MSI sensor by using information about the spectra in a plurality of channels. For this purpose, the illuminance of the object and the surface reflectance can be represented by spectral decomposition. That is, the illuminance E can be represented by Equation 2 below, and the surface reflectance S of the object can be represented by Equation 3 below.

[0094] [Equation 2]

[0095]

[0096] [Equation 3]

[0097]

[0098] The product of the illuminance E and the surface reflectance S represents color, and the value sensed by the second image sensor 200 can be represented by Equation 4 below.

[0099] [Equation 4]

[0100]

[0101]

[0102] where m and n denote the number of basis vectors for spectral decomposition of the illumination spectrum and the object spectrum, x denotes the spatial position, ∈ i denotes the coefficient of the basis vector E i σ j denotes the coefficient of the basis vector S j and k denotes the channel index. The spectrum of the illumination light can be estimated by solving the linear equations by a method such as non-linear optimization.

[0103] In addition, optionally, the illuminance estimation unit 520 can estimate the illuminance by using a neural network. Regarding a predetermined illuminance value, an MSI sensor image can be used to train the neural network, and then the neural network can be used to estimate the illumination. After training the neural network, the MSI sensor image can be input into the neural network, and then the neural network can output an illuminance value.

[0104] The illuminance estimation unit 520 can divide the image into pixel groups and can estimate the illuminance in the area of each pixel group. In this case, the pixel groups can be the same as the pixel groups used in the image registration unit 510.

[0105] The illuminance estimation unit 520 can represent the result of the illuminance estimation as an illuminance-wavelength function. Alternatively, basis functions can be predefined, and the illuminance-wavelength function can be spectrally decomposed into basis functions multiplied by coefficients, and the coefficients of the basis functions can represent the result of the illuminance estimation. In addition, the color temperature index of the illumination can be used to represent the result of the illumination estimation. Here, the use of color temperature is a method of expressing the light of a light source in Kelvin (K) values. The color temperature varies depending on the type of illumination: the lower the color temperature, the redder the color; and the higher the color temperature, the bluer the color.

[0106] In addition, a set of illumination wavelength functions can be predefined, and the index of the most approximate function in the illumination wavelength functions can be output. Even when using a neural network, the output can be defined as described above, and the neural network can be trained on the defined output.

[0107] In addition, the illuminance estimation unit 520 can estimate the illuminance by using both the first image IM1 and the second image IM2 obtained by using the first image sensor 100 and the second image sensor 200. In this case, the illuminance estimation unit 520 can estimate the illuminance by using the first image IM1 and the second image IM2 separately, and then can combine the estimated results.

[0108] Alternatively, when the illuminance estimation unit 520 estimates the illuminance through multiple channels, the illuminance estimation unit 520 may consider all channels of the first image IM1 and the second image IM2.

[0109] In addition, when using a neural network, the neural network can be trained by considering both the first image IM1 (i.e., the RGB image) and the second image IM2 (i.e., the MSI image under specific illumination light). In this case, the RGB image and the MSI image can be input into the neural network, and then the neural network can output the estimated illuminance value.

[0110] The color conversion unit 530 performs color conversion on the registered image obtained from the image registration unit 510 by using the illuminance value estimated by the illuminance estimation unit 520.

[0111] The color conversion unit 530 performs color mapping for each pixel group used in the image registration unit 510 and the illuminance estimation unit 520. Here, the R, G, and B values of each pixel can be input for color mapping, and then the R', G', and B' values obtained by correcting the input R, G, and B values considering the illumination can be output. The 3x3 matrix operation as shown in Equation 5 below can be used to convert the input vector I = [R G B] T to the output vector I' = [R’ G’ B'] T .

[0112] [Equation 5]

[0113] I’ = MI

[0114] where M can be a diagonal matrix (e.g., a conversion matrix) prepared to independently consider R, G, and B. Alternatively, M can be a non - diagonal matrix for generating R', G', and B' by combining R, G, and B. Alternatively, M can be a matrix other than the 3x3 matrix, which considers the cross - terms or quadratic terms of R, G, and B. In this case, in addition to R, G, and B, the input vector I may also include cross - terms or quadratic terms. Multiple matrices M can be predetermined according to the color temperature. Alternatively, the matrix M can be predetermined according to the index of a predetermined illumination function. Alternatively, an optimized matrix can be prepared by using non - linear optimization of the illumination function to obtain more accurate results.

[0115] After the region - based color conversion, the color conversion unit 530 can prevent boundary artifacts from appearing at the boundaries between adjacent regions. To this end, the difference between the illumination functions of adjacent regions can be adjusted to be not greater than a predetermined value. In addition, after the color conversion, post - processing can be performed on the boundary regions to achieve smooth color conversion.

[0116] Figure 3is a conceptual diagram showing a schematic structure of an image acquisition device 1000 according to an embodiment, and Figure 4 is a view showing a circuit configuration of a first image sensor 100 and a second image sensor 200 provided in the image acquisition device 1000 according to an embodiment.

[0117] The image acquisition device 1000 includes: a first image sensor 100 configured to acquire a first image IM1 based on a first wavelength band; a second image sensor 200 configured to acquire a second image IM2 based on a second wavelength band; and a processor 500 configured to generate a third image IM3 by performing signal processing on the first image IM1 and the second image IM2. The image acquisition device 1000 may further include a first memory 300 configured to store data regarding the first image IM1, a second memory 310 configured to store data regarding the second image IM2, and an image output unit 700 configured to output an image.

[0118] The image acquisition device 1000 may further include: a first imaging optical system 190 configured to form an optical image of an object OBJ on the first image sensor 100; and a second imaging optical system 290 configured to form an optical image of the object OBJ on the second image sensor 200. Although each of the first imaging optical system 190 and the second imaging optical system 290 is shown as including one lens, this is merely a non-limiting example. The first imaging optical system 190 and the second imaging optical system 290 may be configured to have the same focal length and the same field of view, and in this case, the process of registering the first image IM1 and the second image IM2 to form the third image IM3 can be performed more easily. However, the embodiment is not limited thereto.

[0119] The first image sensor 100 includes a first pixel array PA1. The first pixel array PA1 includes: a first sensor layer 110 in which a plurality of first sensing elements are arranged; and a color filter 120 arranged on the first sensor layer 110. The color filter 120 may include color filters of red, green, and blue arranged alternately. A first microlens array 130 may be arranged on the first pixel array PA1. Various examples of the pixel arrangement of the first pixel array PA1 will be described later with reference to Figures 5 to 8 describe various examples of the pixel arrangement of the first pixel array PA1.

[0120] The second image sensor 200 includes a second pixel array PA2. The second pixel array PA2 includes: a second sensor layer 210 in which a plurality of second sensing elements are arranged; and a spectral filter 220 arranged on the second sensor layer 210. The spectral filter 220 includes a plurality of filter banks, and each of the plurality of filter banks may include a plurality of unit filters having different transmission bands. The spectral filter 220 may be configured to filter light in a band wider than the band of light filtered by the color filter 120. For example, the spectral filter 220 may be configured to filter light in a band ranging from the ultraviolet band to the infrared band by dividing the band into more sub-bands than the sub-bands of light filtered by the color filter 120. The first microlens array 230 may be arranged on the second pixel array PA2. An example of the pixel arrangement of the second pixel array PA2 will be described later with reference to Figures 10 to 12 an example of the pixel arrangement of the second pixel array PA2.

[0121] Each of the first sensor layer 110 and the second sensor layer 210 may include, but is not limited to, a charge-coupled device (CCD) sensor or a CMOS sensor.

[0122] The first pixel array PA1 and the second pixel array PA2 may be horizontally arranged on the same circuit board SU, for example, separated from each other in the X direction, as Figure 3 shown.

[0123] The circuit board SU may include: a first circuit element for processing signals from the first sensor layer 110; and a second circuit element for processing signals from the second sensor layer 210. However, the embodiments are not limited thereto, and the first circuit element and the second circuit element may be separately provided on different substrates.

[0124] Although the first memory 300 for storing data about the first image IM1 and the second memory 310 for storing data about the second image IM2 are shown as being separate from the circuit board SU, this is merely an example, and the first memory 300 and the second memory 310 may be arranged in the same layer as the first circuit element and the second circuit element of the circuit board SU, or may be arranged in a layer different from the layer in which the first circuit element and the second circuit element are arranged. Each of the first memory 300 and the second memory 310 may be a row memory configured to store an image row by row, or a frame buffer configured to store the entire image. Each of the first memory 300 and the second memory 310 may include a static random access memory (SRAM) or a dynamic random access memory (DRAM).

[0125] Various circuit elements required for the image acquisition device 1000 can be integrated into the circuit board SU. For example, the circuit board SU may include: a logic layer including various analog circuits and digital circuits; and a memory layer in which data is stored. The logic layer and the memory layer may be provided as different layers or the same layer.

[0126] Reference Figure 4 , the row decoder 102, the output circuit 103, and the timing controller (TC) 101 are connected to the first pixel array PA1. The row decoder 102 selects a row of the first pixel array PA1 in response to a row address signal output from the TC 101. The output circuit 103 outputs a photoelectric sensing signal from a plurality of pixels arranged along the selected row in units of columns. To this end, the output circuit 103 may include a column decoder and an analog-to-digital converter (ADC). For example, the output circuit 103 may include: a plurality of ADCs respectively arranged for columns between the column decoder and the first pixel array PA1, or may include one ADC arranged at the output end of the column decoder. The TC 101, the row decoder 102, and the output circuit 103 may be implemented as one chip or separate chips. At least some of the illustrated circuit elements may be provided on Figure 3 the circuit board SU shown in. A processor for processing the first image IM1 output through the output circuit 103 may be implemented as a single chip together with the TC 101, the row decoder 102, and the output circuit 103.

[0127] The row decoder 202, the output circuit 203, and the TC 201 are also connected to the second pixel array PA2, and may process signals from the second pixel array PA2 as described above. In addition, a processor for processing the second image IM2 output from the output circuit 203 may be implemented as a single chip together with the TC 201, the row decoder 202, and the output circuit 203.

[0128] Although the first pixel array PA1 and the second pixel array PA2 are shown in Figure 4 to have the same size and the same number of pixels, this is only a non-limiting example for ease of illustration.

[0129] When operating two different types of sensors, timing control may be required according to the different resolutions and output speeds of the sensors, as well as the size of the area required for image registration. For example, when reading an image column based on the first image sensor 100, the image column corresponding to that area in the second image sensor 200 may have been stored in the buffer or a new one may need to be read. Alternatively, the operations of the first image sensor 100 and the second image sensor 200 may be synchronized using the same synchronization signal. For example, the TC 400 may also be arranged to send a synchronization signal sync to the first image sensor 100 and the second image sensor 200.

[0130] Figure 5 is a graph showing a wavelength spectrum obtained by using the first image sensor 100 provided in the image acquisition device 1000 according to an embodiment, and Figures 6 to 8 is a view showing an example of the pixel arrangement of the first image sensor 100 provided in the image acquisition device 1000 according to an embodiment.

[0131] Refer to Figure 6 , in the color filter 120 provided in the first pixel array PA1, filters for filtering in the red (R), green (G), and blue (B) bands are arranged in a Bayer pattern. That is, one unit pixel includes sub-pixels arranged in a 2×2 array, and multiple unit pixels are arranged two-dimensionally in a repeating manner. The red color filter and the green color filter are arranged in the first row of the unit pixel, and the green color filter and the blue color filter are arranged in the second row. The pixels may be arranged in other patterns different from the Bayer pattern.

[0132] For example, refer to Figure 7 , a CYGM arrangement may also be implemented, in which a magenta pixel (M), a cyan pixel (C), a yellow pixel (Y), and a green pixel (G) form one unit pixel. In addition, refer to Figure 8 , an RGBW arrangement may also be implemented, in which a green pixel (G), a red pixel (R), a blue pixel (B), and a white pixel (W) form one unit pixel. In addition, although not shown in Figures 6 to 8 , the unit pixel may have a 3×2 array pattern. In addition, the pixels of the first pixel array PA1 may be arranged in various patterns according to the color characteristics of the first image sensor 100.

[0133] Figure 9 is a graph showing a wavelength spectrum obtained by using the second image sensor 200 provided in the image acquisition device 1000 according to an embodiment, and Figures 10 to 12 is a view showing an example of the pixel arrangement of the second image sensor 200 provided in the image acquisition device 1000 according to an embodiment.

[0134] Refer to Figure 10 , the spectral filter 220 provided in the second pixel array PA2 may include a plurality of filter groups 221 arranged in a two-dimensional form. Each group in the filter group 221 may include 16 unit filters, that is, the first unit filter F1 to the sixteenth unit filter F16 arranged in a 4×4 array.

[0135] The first unit filter F1 and the second unit filter F2 may have central wavelengths UV1 and UV2 in the ultraviolet region, and the third unit filter F3 to the fifth unit filter F5 may have central wavelengths B1 to B3 in the blue light region. The sixth unit filter F6 to the eleventh unit filter F11 may have central wavelengths G1 to G6 in the green light region, and the twelfth unit filter F12 to the fourteenth unit filter F14 may have central wavelengths R1 to R3 in the red light region. In addition, the fifteenth unit filter F15 and the sixteenth unit filter F16 may have central wavelengths NIR1 and NIR2 in the near-infrared region.

[0136] Figure 11 is a plan view showing one of the filter banks 222 provided in the spectral filter 220 according to an embodiment. Refer to Figure 11 , the filter bank 222 may include nine unit filters, i.e., the first unit filter F1 to the ninth unit filter F9 arranged in a 3×3 array. The first unit filter F1 and the second unit filter F2 may have central wavelengths UV1 and UV2 in the ultraviolet region, and the fourth unit filter F4, the fifth unit filter F5, and the seventh unit filter F7 may have central wavelengths B1 to B3 in the blue light region. The third unit filter F3 and the sixth unit filter F6 may have central wavelengths G1 and G2 in the green light region, and the eighth unit filter F8 and the ninth unit filter F9 may have central wavelengths R1 and R2 in the red light region.

[0137] Figure 12 is a plan view showing one of the filter banks 223 provided in the spectral filter 220 according to an embodiment. Refer to Figure 12, the filter bank 223 may include 25 unit filters, i.e., the first unit filter F1 to the twenty-fifth unit filter F25 arranged in a 5×5 array. The first unit filter F1 to the third unit filter F3 may have central wavelengths UV1 to UV3 in the ultraviolet region, and the sixth unit filter F6, the seventh unit filter F7, the eighth unit filter F8, the eleventh unit filter F11, and the twelfth unit filter F12 may have central wavelengths B1 to B5 in the blue light region. The fourth unit filter F4, the fifth unit filter F5, the ninth unit filter F9, the sixteenth unit filter F16, the seventeenth unit filter F17, the twenty-first unit filter F21, and the twenty-second unit filter F22 may have central wavelengths G1 to G3 in the green light region, and the tenth unit filter F10, the thirteenth unit filter F13, the fourteenth unit filter F14, the fifteenth unit filter F15, the eighteenth unit filter F18, and the nineteenth unit filter F19 may have central wavelengths R1 to R6 in the red light region. In addition, the twentieth unit filter F20, the twenty-third unit filter F23, the twenty-fourth unit filter F24, and the twenty-fifth unit filter F25 may have central wavelengths NIR1 to NIR4 in the near-infrared region.

[0138] The above unit filters provided in the spectral filter 220 may have a resonant structure having two reflector plates, and the transmission band of the spectral filter 220 may be determined according to the characteristics of the resonant structure. The transmission band can be adjusted according to the material of the reflector plates, the dielectric material in the cavity of the resonant structure, and the thickness of the cavity. In addition, other structures such as a structure using a grating or a structure using a distributed Bragg reflector (DBR) may be applied to the unit filters.

[0139] In addition, the pixels of the second pixel array PA2 may be arranged in various ways according to the color characteristics of the second image sensor 200.

[0140] Figure 13 is a flowchart schematically showing a method of controlling an image acquisition device according to an embodiment.

[0141] Referring to Figure 13 , in operation S1300, a first image and a second image are acquired. Here, the first image is an RGB image obtained using a general RGB or CMOS image sensor. The second image is obtained using an MSI sensor. The second image may have a wider band than the first image. In addition, the second image has more channels than the three channels (i.e., R, G, and B channels) of the first image.

[0142] In operation S1302, the first image is divided into regions. Here, the first image can be divided into pixel groups or patterns having a preset size and shape. The pattern can be a line or a square. Additionally, the pixel groups can be classified as foreground and background, or can be classified as ROI and non-ROI.

[0143] In operation S1304, the second image can be divided into regions corresponding to the regions of the first image respectively. In addition, the first image and the second image can be overlapped with each other through registration of the first image and the second image. Further, in operations S1302 and S1304, registration for superimposing the first image and the second image on each other can be performed in units of the regions into which the first image and the second image are divided. Although it has been described in operations S1302 and S1304 that image registration is performed in units of pixel groups, the embodiments are not limited thereto, and image registration can be performed in units of pixels or sub-pixels.

[0144] In operation S1306, an illuminance value is estimated for each region of the second image. The multi-channel spectral information included in the second image acquired using the MSI sensor can be used to estimate the illuminance value. Here, the illuminance value is estimated for each region into which the second image is divided.

[0145] In operation S1308, color conversion is performed using the illuminance values estimated for the regions of the first image respectively. Here, the first image can be an RGB image obtained using an RGB image sensor or a CMOS image sensor in operation S1300, or can be a registered image obtained by registering the first image and the second image in operations S1302 and S1304.

[0146] In the method of controlling an image acquisition device according to an embodiment, the image captured using an RGB camera is white-balanced using an RGB image sensor and an MSI sensor. For accurate white balance, the illumination can be separated from the color of the object to find the exact color of the object, and then the RGB image of the object can be white-balanced by color mapping based on the illumination information.

[0147] The above-described image acquisition device 1000 can be used in various high-performance optical devices or electronic devices. Examples of the electronic device can include a smart phone, a mobile phone, a cellular phone, a personal digital assistant (PDA), a laptop computer, a personal computer (PC), various portable devices, household appliances, a security camera, a medical camera, an automobile, an Internet of Things (IoT) device, and a mobile or non-mobile computing device, but are not limited thereto.

[0148] In addition to the image acquisition device 1000, the electronic device may further include a processor for controlling an image sensor disposed therein, such as an application processor (AP). The electronic device may control a plurality of hardware or software components by driving an operating system or an application in the processor, and may perform various data processing and operations. The processor may further include a graphics processing unit (GPU) and / or an ISP. When the processor includes an ISP, an image (or video) obtained using the image sensor may be stored and / or output using the processor.

[0149] Figure 14 is a block diagram schematically showing the structure of an electronic device ED01 according to an embodiment. Refer to Figure 14 , in a network environment ED00, the electronic device ED01 may communicate with another electronic device ED02 via a first network ED98 (such as a near-field wireless communication network), or may communicate with another electronic device ED04 and / or a server ED08 via a second network ED99 (such as a far-field wireless communication network). The electronic device ED01 may communicate with the electronic device ED04 via the server ED08. The electronic device ED01 may include a processor ED20, a memory ED30, an input device ED50, a sound output device ED55, a display device ED60, an audio module ED70, a sensor module ED76, an interface ED77, a haptic module ED79, a camera module ED80, a power management module ED88, a battery ED89, a communication module ED90, a user identification module ED96, and / or an antenna module ED97. Some of the components may be omitted from the electronic device ED01 (such as the display device ED60, etc.), or other components may be added to the electronic device ED01. Some of the components may be implemented in one integrated circuit. For example, the sensor module ED76 (fingerprint sensor, iris sensor, illuminance sensor, etc.) may be embedded in the display device ED76 (display, etc.). In addition, when the image sensor has a spectral function, some sensor module functions (color sensing, illumination sensing, etc.) may be implemented in the image sensor instead of in the sensor module ED76.

[0150] The processor ED20 can execute software (such as program ED40, etc.) to control one or more other components (hardware or software components, etc.) connected to the processor ED20 in the electronic device ED01, and can perform various data processing or operations. As part of the data processing or operations, the processor ED20 can load instructions and / or data received from other components (sensor module ED76, communication module ED90, etc.) into the volatile memory ED32, process the instructions and / or data stored in the volatile memory ED32, and can store the resulting data in the non-volatile memory ED34. The processor ED20 can include a main processor ED21 (central processing unit, application processor, etc.) and a co-processor ED23 (GPU, ISP, sensor hub processor, communication processor, etc.) that operates independently of or together with the main processor ED21. The co-processor ED23 can consume less power than the main processor ED21 and can perform specialized functions.

[0151] The co-processor ED23 can represent the main processor ED21 when the main processor ED21 is in an inactive (e.g., sleep) state, or can control functions and / or states related to some of the components (e.g., display device ED60, sensor module ED76, communication module ED90, etc.) in the components of the electronic device ED01 together with the main processor ED21 when the main processor ED21 is in an active (e.g., application execution) state. The co-processor ED23 (ISP, communication processor, etc.) can be implemented as part of other function-related components (camera module ED80, communication module ED90, etc.).

[0152] The memory ED30 can store various data required by the components (processor ED20, sensor module ED76, etc.) of the electronic device ED01. This data can include, for example, input data and / or output data of software (such as program ED40, etc.) and commands related thereto. The memory ED30 can include a volatile memory ED32 and / or a non-volatile memory ED34. The non-volatile memory ED32 can include an internal memory ED36 fixed to the electronic device ED01 and an external memory ED38 removable from the electronic device ED01.

[0153] The program ED40 can be stored in the memory ED30 as software, and can include an operating system ED42, middleware ED44, and / or an application ED46.

[0154] The input device ED50 can receive commands and / or data from outside the electronic device ED01 (user, etc.) to be used by the components (processor ED20, etc.) of the electronic device ED01. The input device ED50 can include a microphone, a mouse, a keyboard, and / or a digital pen (stylus, etc.).

[0155] The audio output device ED55 can output an audio signal to the outside of the electronic device ED01. The audio output device ED55 may include a speaker and / or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. The receiver can be set as a part of the speaker or can be implemented as a separate device.

[0156] The display device ED60 can visually provide information to the user of the electronic device ED01. The display device ED60 may include a display, a holographic device or a projector, and a control circuit for controlling the device. The display device ED60 may include a touch circuit configured to sense a touch and / or a sensor circuit (such as a pressure sensor) configured to measure the intensity of the force generated by the touch.

[0157] The audio module ED70 can convert sound into an electrical signal and vice versa. The audio module ED70 can obtain sound through the input device ED50, or can output sound through the sound output device ED55 and / or the speaker and / or headphones of another electronic device (such as the electronic device ED02) directly or wirelessly connected to the electronic device ED01.

[0158] The sensor module ED76 can detect the operating state (power, temperature, etc.) of the electronic device ED01 or the external environmental state (user state, etc.), and can generate an electrical signal and / or a data value corresponding to the detected state. The sensor module ED76 may include a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, and / or an illuminance sensor.

[0159] The interface ED77 can support one or more specified protocols that can be used to directly or wirelessly connect the electronic device ED01 to other electronic devices (such as the electronic device ED02). The interface ED77 may include a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, and / or an audio interface.

[0160] The connection terminal ED78 may include a connector through which the electronic device ED01 can be physically connected to other electronic devices (such as the electronic device ED02). The connection terminal ED78 may include an HDMI connector, a USB connector, an SD card connector, and / or an audio connector (such as a headphone connector).

[0161] The tactile module ED79 can convert an electrical signal into a mechanical stimulus (vibration, movement, etc.) or an electrical stimulus that a user can perceive through touch or kinesthesia. The tactile module ED79 may include a motor, a piezoelectric element, and / or an electrical stimulation device.

[0162] The camera module ED80 can capture still images and moving images. The camera module ED80 may include the above-described image acquisition device 1000, and may further include a lens assembly, an ISP, and / or a flash. The lens assembly included in the camera module ED80 can collect light emitted from an object to be imaged.

[0163] The power management module ED88 can manage the power supplied to the electronic device ED01. The power management module ED88 may be implemented as part of a power management integrated circuit PMIC.

[0164] The battery ED89 can supply power to the components of the electronic device ED01. The battery ED89 may include a non-rechargeable main battery, a rechargeable secondary battery, and / or a fuel cell.

[0165] The communication module ED90 can support the establishment of a direct (wired) communication channel and / or a wireless communication channel between the electronic device ED01 and other electronic devices (such as the electronic device ED02, the electronic device ED04, or the server ED08, etc.), and communicate through the established communication channel. The communication module ED90 can operate independently of the processor ED20 (such as an application processor), and may include one or more communication processors that support direct communication and / or wireless communication. The communication module ED90 may include a wireless communication module ED92 (such as a cellular communication module, a short-range wireless communication module, a global navigation satellite system (GNSS), etc.) and / or a wired communication module ED94 (such as a local area network (LAN) communication module or a power line communication module, etc.). The corresponding communication module among these communication modules can communicate with other electronic devices through a first network ED98 (such as a local area network such as Bluetooth, WiFi Direct, or Infrared Data Association (IrDA)) or a second network ED99 (such as a communication network such as a cellular network, the Internet, or a computer network (LAN, WAN, etc.)). These various types of communication modules may be integrated into a single component (such as a single chip), or may be implemented as multiple separate components (multiple chips). The wireless communication module ED92 can use the user information (such as an International Mobile Subscriber Identity (IMSI)) stored in the user identification module ED96 to identify and authenticate the electronic device ED01 in a communication network (such as the first network ED98 and / or the second network ED99).

[0166] The antenna module ED97 can transmit signals and / or power to the outside (other electronic devices, etc.) or receive signals and / or power from it. The antenna can include a radiator implemented by a conductive pattern formed on a substrate (such as a PCB). The antenna module ED97 can include one or more such antennas. When the antenna module ED97 includes multiple antennas, the communication module ED90 can select an antenna suitable for the communication method to be used in a communication network such as the first network ED98 and / or the second network ED99 from the multiple antennas. Signals and / or power can be transmitted or received between the communication module ED90 and other electronic devices through the selected antenna. Other components (such as RFICs) in addition to the antenna can be included as part of the antenna module ED97.

[0167] Some of the components can be connected to each other and exchange signals (commands, data, etc.) through a communication method between peripheral devices (such as a bus, general-purpose input and output (GPIO), serial peripheral bus (SPI), mobile industry processor interface (MIPI), etc.).

[0168] Commands or data can be transmitted or received between the electronic device ED01 and an external device such as the electronic device ED04 through a server ED08 connected to the second network ED99. The other electronic devices ED02 and ED04 can be the same as or different from the electronic device ED01. All or part of the operations of the electronic device ED01 can be executed by one or more of the other electronic devices ED02, ED04, and ED08. For example, when the electronic device ED01 needs to execute a specific function or service, the electronic device ED01 can request one or more other electronic devices to execute some or all of the functions or services instead of directly executing the function or service. One or more other electronic devices that have received the request can execute additional functions or services related to the request and can transmit the execution results to the electronic device ED01. For this purpose, cloud computing, distributed computing, and / or client-server computing technologies can be used.

[0169] Figure 15 is schematically shown Figure 14 a block diagram of the camera module ED80 included in the electronic device ED01 shown in. The camera module ED80 can include the above-described image acquisition device 1000 or can have a modified structure. Referring to Figure 15 , the camera module ED80 can include a lens assembly CM10, a flash CM20, an image sensor CM30, an image stabilizer CM40, a memory CM50 (such as a buffer memory), and / or an ISPCM60.

[0170] The image sensor CM30 may include a first image sensor 100 and a second image sensor 200 disposed in the above-described image acquisition device 1000. The first image sensor 100 and the second image sensor 200 may obtain an image corresponding to an object by converting light emitted or reflected from the object and transmitted through the lens assembly CM10 into an electrical signal. The first image sensor 100 may obtain an RGB image, and the second image sensor 200 may obtain a hyperspectral image in the ultraviolet to infrared wavelength range.

[0171] In addition to the above-described first image sensor 100 and second image sensor 200, the image sensor CM30 may further include one or more sensors selected from image sensors having different properties, such as other RGB image sensors, black and white (BW) sensors, infrared sensors, or ultraviolet sensors. Each of the sensors included in the image sensor CM30 may be implemented as a CCD sensor and / or a CMOS sensor.

[0172] The lens assembly CM10 may collect light emitted from an object to be imaged. The camera module ED80 may include a plurality of lens assemblies CM10, and in this case, the camera module ED80 may be a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies CM10 may have the same lens properties (field of view, focal length, autofocus, F-number, optical zoom, etc.) or different lens properties. Each of the lens assemblies CM10 may include a wide-angle lens or a telephoto lens.

[0173] The lens assembly CM10 may be configured and / or focused such that the two image sensors included in the image sensor CM30 may form an optical image of the object at the same position.

[0174] The flash CM20 may emit light for enhancing the light emitted or reflected from the object. The flash CM20 may include one or more light-emitting diodes (red-green-blue (RGB) LEDs, white LEDs, infrared LEDs, ultraviolet LEDs, etc.) and / or xenon lamps.

[0175] The image stabilizer CM40 may move one or more lenses included in the lens assembly CM10 or the image sensor CM30 in a specific direction in response to the movement of the camera module ED80 or the electronic device ED01 including the camera module ED80, or may control the operating characteristics (such as adjustment of the readout timing) of the image sensor CM30 to compensate for the negative effects caused by the movement. The image stabilizer CM40 may detect the movement of the camera module ED80 or the electronic device ED01 by using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module ED80. The image stabilizer CM40 may be an optical image stabilizer.

[0176] In the memory CM50, some or all of the data obtained by the image acquisition device 1000 can be stored for the next image processing operation. For example, when multiple images are obtained at high speed, the obtained raw data (Bayer pattern data, high-resolution data, etc.) can be stored in the memory CM50, and only the low-resolution images can be displayed. Then, the raw data of the selected image (user selection, etc.) can be transmitted to the ISP CM60. The memory CM50 can be integrated into the memory ED30 of the electronic device ED01, or can be configured as a separate memory that can operate independently.

[0177] The ISP CM60 can perform one or more image processes on the images obtained by the image sensor CM30 or the image data stored in the memory CM50. As referenced Figures 1 to 13 as described, the first image (e.g., RGB image) and the second image (e.g., MSI image) obtained using the two image sensors included in the image sensor CM30 are processed to form a third white balance-adjusted image. To this end, the components of the processor 500 can be included in the ISP CM60.

[0178] In addition, one or more image processes can include depth map generation, 3D modeling, panoramic generation, feature point extraction, image synthesis, and / or image compensation (noise reduction, resolution adjustment, brightness adjustment, blur, sharpening, softening, etc.). The ISP CM60 can control (exposure time control, readout timing control, etc.) the components (image sensor CM30, etc.) included in the camera module CM80. The images processed by the ISP CM60 can be stored again in the memory CM50 for further processing, or can be provided to external components of the camera module ED80 (memory ED30, display device ED60, electronic device ED02, electronic device ED04, server ED08, etc.). The ISP CM60 can be integrated into the processor ED20, or can be configured as a separate processor that operates independently of the processor ED20. When the ISP CM60 is separately provided from the processor ED20, the images processed by the ISP CM60 can be displayed on the display device ED60 after being further processed by the processor ED20.

[0179] The electronic device ED01 can include multiple camera modules ED80 with different attributes or functions. In this case, one of the multiple camera modules ED80 can be a wide-angle camera, while another of the multiple camera modules ED80 can be a telephoto camera. Similarly, one of the multiple camera modules ED80 can be a front camera, while another of the multiple camera modules ED80 can be a rear camera.

[0180] Figures 16 to 25It is a view showing various examples of an electronic device to which the image acquisition device 1000 according to an embodiment is applied.

[0181] According to an embodiment, the image acquisition device 1000 can be applied to Figure 16 the mobile phone or smartphone 5100m shown, Figure 17 the tablet computer or smart tablet 5200 shown, Figure 18 the digital camera or video camera 5300 shown, Figure 19 the laptop computer 5400 shown, or Figure 20 the television or smart TV 5500 shown. For example, the smartphone 5100m or the smart tablet computer 5200 may include a plurality of high-resolution cameras each having a high-resolution image sensor installed. The high-resolution cameras can be used to extract depth information of an object in an image, adjust the defocus of the image, or automatically identify an object in the image.

[0182] In addition, the image acquisition device 1000 can be applied to Figure 21 the smart refrigerator 5600 shown, Figure 22 the security camera 5700 shown, Figure 23 the robot 5800 shown, Figure 24 the medical camera 5900 shown, etc. For example, the smart refrigerator 5600 can automatically identify the food contained in the smart refrigerator 5600 by using the image acquisition device 1000, and can notify the user via the smartphone whether a specific food is contained in the smart refrigerator 5600, the type of food put into or taken out of the smart refrigerator 5600, etc. Due to the high sensitivity of the security camera 5700, the security camera 5700 can provide ultra-high-resolution images and can identify an object or a person in the ultra-high-resolution image even in a dark environment. The robot 5800 can be sent to a disaster or industrial site where humans cannot directly access, and can provide high-resolution images. The medical camera 5900 can provide high-resolution images for diagnosis or surgery, and can have a dynamically adjustable field of view.

[0183] In addition, the image acquisition device 1000 can be applied to, for example, Figure 25 the vehicle 6000 shown. The vehicle 6000 may include a plurality of in-vehicle cameras 6010, 6020, 6030, and 6040 arranged at different positions. Each of the in-vehicle cameras 6010, 6020, 6030, and 6040 may include an image acquisition device according to an embodiment. The vehicle 6000 can use the in-vehicle cameras 6010, 6020, 6030, and 6040 to provide various information about the interior or surrounding environment of the vehicle 6000 to the driver, and can provide the information required for autonomous driving by automatically identifying an object or a person in the image.

[0184] As described above, according to one or more of the above embodiments, the image acquisition device can perform accurate white balance without limitation by using two different types of image sensors even under one or more types of illumination light.

[0185] The image acquisition device can be used in various electronic devices.

[0186] It should be understood that the embodiments described herein should be considered only in a descriptive sense and not for purposes of limitation. The description of features or aspects in each embodiment should generally be regarded as available for other similar features or aspects in other embodiments. Although one or more embodiments have been described with reference to the accompanying drawings, those of ordinary skill in the art should understand that various changes in form and detail may be made without departing from the spirit and scope defined by the appended claims.

Claims

1. An image acquisition device, comprising: A first image sensor, having an R channel, a G channel, and a B channel, and configured to acquire a first image based on a first band; A second image sensor, having more channels than the first image sensor, and configured to acquire a second image based on a second band, the second band including the first band, wherein the bandwidth of each channel of the second image sensor is narrower than that of each of the R channel, the G channel, and the B channel, and the total bandwidth as the sum of the bandwidths of all channels of the second image sensor is wider than the sum of the bandwidths of the R channel, the G channel, and the B channel; and A processor, configured to: Divide the first image into one or more first regions, and divide the second image into one or more second regions respectively corresponding to the one or more first regions; Estimate the corresponding illuminance value of each second region in the one or more second regions; Perform color conversion on each of the one or more first regions using the estimated illuminance value; and When the difference between the illuminance values respectively estimated for adjacent second regions among the one or more second regions is greater than or equal to a first threshold, adjust any one of the illuminance values of the adjacent second regions to adjust the difference to be less than the first threshold.

2. The image acquisition device according to claim 1, wherein, The first image further includes a registered image obtained by registering the first image and the second image.

3. The image acquisition device according to claim 1, wherein, The processor is further configured to: after performing the color conversion, perform post-processing on the boundary portion between the adjacent second regions.

4. The image acquisition device according to claim 1, wherein, The processor is further configured to calculate parameters for registering the first image and the second image based on at least one of the resolution, field of view, and focal length of each of the first image sensor and the second image sensor.

5. The image acquisition device according to claim 1, wherein, The processor is further configured to estimate the illuminance value by using spectral information obtained by using a plurality of channels output from the second image sensor.

6. The image acquisition device according to claim 1, wherein, The processor is further configured to estimate the illuminance value by using a neural network trained with a plurality of second images associated with a predetermined illuminance value.

7. The image acquisition device according to claim 1, wherein, The processor is further configured to register the first image and the second image by respectively extracting a first feature from the first image and a second feature from the second image, and matching the extracted first feature with the extracted second feature.

8. The image acquisition device according to claim 1, wherein, The processor is further configured to register the first image and the second image in units of pixel groups, pixels, or sub-pixels of the first image and the second image.

9. The image acquisition device according to claim 1, wherein, the first image sensor includes a first pixel array, and the first pixel array includes: a first sensor layer in which a plurality of first sensing elements are arranged; and a color filter on the first sensor layer and including alternately arranged red, green, and blue color filters, wherein, the second image sensor includes a second pixel array, and the second pixel array includes: a second sensor layer in which a plurality of second sensing elements are arranged; and a spectral filter on the second sensor layer and in which a filter bank is repeatedly arranged, the filter bank including a plurality of unit filters, and each unit filter having a transmission band different from every other unit filter of the filter bank.

10. The image acquisition device according to claim 9, wherein, the combined transmission band of the plurality of unit filters of the filter bank includes the visible light band and is wider than the visible light band, and wherein, the plurality of unit filters includes 16 unit filters arranged in a 4×4 array.

11. The image acquisition device according to claim 9, wherein, the first pixel array and the second pixel array are horizontally separated from each other on the circuit board.

12. The image acquisition device according to claim 11, wherein, a first circuit element configured to process signals from the first sensor layer and a second circuit element configured to process signals from the second sensor layer are provided on the circuit board.

13. The image acquisition device according to claim 12, further comprising a timing controller configured to synchronize the operation of the first circuit element with the operation of the second circuit element.

14. The image acquisition device according to claim 11, further comprising: a first memory storing data about the first image; and a second memory storing data about the second image.

15. The image acquisition device according to claim 14, wherein, the first memory and the second memory are provided inside the circuit board.

16. The image acquisition device according to claim 1, further comprising: a first imaging optical system configured to form a first optical image of an object on the first image sensor, the first imaging optical system including at least one first lens; and a second imaging optical system configured to form a second optical image of the object on the second image sensor, the second imaging optical system including at least one second lens.

17. The image acquisition device according to claim 16, wherein, the first imaging optical system and the second imaging optical system have the same focal length and the same field of view.

18. An electronic device including the image acquisition device according to claim 1.

19. A method of controlling an image acquisition device including a plurality of image sensors, the method comprising: Obtain a first image and a second image from a first image sensor and a second image sensor respectively. The first image sensor has an R channel, a G channel, and a B channel and is configured to obtain a first image based on a first band. The second image sensor has more channels than the first image sensor and is configured to obtain a second image based on a second band, where the second band includes the first band. Each channel of the second image sensor has a bandwidth narrower than each of the R channel, the G channel, and the B channel, and the total bandwidth as the sum of the bandwidths of all channels of the second image sensor is wider than the sum of the bandwidths of the R channel, the G channel, and the B channel; Divide the first image into one or more first regions, and divide the second image into one or more second regions respectively corresponding to the one or more first regions; Estimate the corresponding illuminance value of each of the one or more second regions; Perform color conversion on each of the one or more first regions using the estimated illuminance value; And When the difference between the illuminance values estimated for adjacent second regions among the one or more second regions is greater than or equal to a first threshold, adjust any one of the illuminance values of the adjacent second regions to adjust the difference to be less than the first threshold.

20. An image acquisition device, Comprising: A first image sensor, including a first filter, the first image sensor having an R channel, a G channel, and a B channel and being configured to obtain a first image based on a first band; A second image sensor, including a second filter different from the first filter, the second image sensor having more channels than the first image sensor and being configured to obtain a second image based on a second band, where the second band includes the first band. Each channel of the second image sensor has a bandwidth narrower than each of the R channel, the G channel, and the B channel, and the total bandwidth as the sum of the bandwidths of all channels of the second image sensor is wider than the sum of the bandwidths of the R channel, the G channel, and the B channel; And A processor, configured to: Receive a first image of an object from the first image sensor and receive a second image of the object from the second image sensor; And Based on the received first image and the received second image, generate a white balance-adjusted first image of the object by eliminating the influence of the illumination reflected by the object, Wherein, the processor is further configured to: Divide the first image into one or more first regions, and divide the second image into one or more second regions respectively corresponding to the one or more first regions; Estimate the corresponding illuminance value of each of the one or more second regions; Perform color conversion on each of the one or more first regions using the estimated illuminance value; and When the difference between the illuminance values respectively estimated for adjacent second regions among the one or more second regions is greater than or equal to a first threshold, adjust any one of the illuminance values of the adjacent second regions to adjust the difference to be less than the first threshold.

21. The image acquisition device according to claim 20, wherein, the first image sensor includes a plurality of first pixels, wherein the first filter includes a plurality of first filter groups arranged in repetition, each first filter group includes a plurality of first unit filters, and each first unit filter corresponds to a corresponding first pixel, wherein the second image sensor includes a plurality of second pixels, wherein the second filter includes a plurality of second filter groups arranged in repetition, each second filter group includes a plurality of second unit filters, and each second unit filter corresponds to a corresponding second pixel, wherein each first filter group of the first filter corresponds to a corresponding second filter group of the second filter, and wherein the wavelength band of each first unit filter is larger than that of each second unit filter.

22. The image acquisition device according to claim 20, wherein, the processor is further configured to: generate a conversion matrix based on the second image for eliminating the influence of the illumination reflected by the object; and generate the white-balanced first image based on the conversion matrix and the first image.

Images