Image acquisition device and electronic device providing white balance function

Abstract

An image acquisition apparatus is provided, including: an image sensor configured to obtain an image; and a processor configured to: obtain a basis based on a surrounding environment of the image acquisition apparatus; estimate illumination information based on the obtained basis; and perform color conversion on the image based on the estimated illumination information.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based upon and claims the benefit of priority from Korean Patent Application No. 10-2021-0128352 filed on September 28, 2021, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] Example embodiments of the present disclosure relate to an image capturing apparatus providing white balance and an electronic apparatus including the image capturing apparatus. Background Art

[0004] An image sensor is a device that receives light incident from a subject and photoelectrically converts the received light to generate an electric signal.

[0005] The image sensor uses a color filter composed of an array of filter elements that selectively transmit red, green, and blue light for color representation, senses the amount of light passing through each filter element, and then forms a color image of the object through image processing.

[0006] Because the values ​​sensed by the image sensor are affected by lighting, the colors of the images captured by the camera are also affected by lighting. White balancing is a technique that eliminates these effects to capture the unique colors of the subject as closely as possible.

[0007] Related art white balancing techniques perform white balancing by capturing a red, green, and blue (RGB) image and analyzing the information within it. This approach requires a grayscale world assumption (i.e., assuming the average values ​​of the image's R, G, and B channels are the same) or other constraints, and therefore may not function properly if these constraints are not met. Summary of the Invention

[0008] One or more example embodiments provide an image capturing apparatus providing white balance and an electronic apparatus including the same.

[0009] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of example embodiments.

[0010] According to an aspect of an example embodiment, there is provided an image acquisition device including: an image sensor configured to acquire an image; and a processor configured to: acquire a base based on a surrounding environment of the image acquisition device; estimate illumination information based on the acquired base; and perform color conversion on the image based on the estimated illumination information.

[0011] The image acquisition device may further include: at least one sensor configured to sense environmental information of the surrounding environment, wherein the processor may be further configured to select a substrate from a plurality of pre-stored substrate sets based on the environmental information obtained from the at least one sensor.

[0012] The processor may be configured to periodically or non-periodically obtain environmental information through at least one sensor before the image sensor obtains an image.

[0013] The at least one sensor may include a GPS sensor, an IMU sensor, a barometer, a magnetometer, an illumination sensor, a proximity sensor, a distance sensor, or a three-dimensional scanner.

[0014] The image sensor can also be configured to sense images in multiple wavelength bands.

[0015] The processor may be further configured to: analyze an image obtained by the image sensor; extract environmental information from the analyzed image; and select a substrate from a plurality of pre-stored substrate sets based on the extracted environmental information.

[0016] The image acquisition apparatus may further include: a storage device configured to store a plurality of base sets including wavelength-based illumination and reflectivity, wherein the processor may further be configured to select a base set corresponding to the obtained environmental information from the plurality of pre-stored base sets.

[0017] The processor may be further configured to estimate illumination information by performing spectral decomposition on the obtained image based on the selected basis set.

[0018] The image sensor may include: a first image sensor configured to obtain an image in a first wavelength band; and a second image sensor configured to obtain an image in a second wavelength band.

[0019] The first image sensor may include: a first sensor layer in which a plurality of first sensing elements are arranged; and a first pixel array having a color filter arranged on the first sensor layer and including a red filter, a green filter and a blue filter arranged alternately, and wherein the second image sensor may include: a second sensor layer in which a plurality of second sensing units are arranged; and a second pixel array having a spectral filter in which a filter group including a plurality of unit filters with different transmission bands is repeatedly arranged, and the spectral filter is arranged on the second sensor layer.

[0020] Each of the transmission wavelength bands of the plurality of unit filters may include visible light and be included in a wavelength band larger than the visible light band, and the filter set may include 16 unit filters arranged in a 4×4 array.

[0021] The first pixel array and the second pixel array may be horizontally spaced apart from each other on the circuit board.

[0022] A first circuit unit configured to process a signal from the first sensor layer and a second circuit unit configured to process a signal from the second sensor layer may be provided on the circuit board.

[0023] The image acquisition device may further include: a timing controller configured to synchronize operations of the first circuit element and the second circuit element.

[0024] The image acquisition device may further include: a first memory configured to store data corresponding to the first image; and a second memory configured to store data corresponding to the second image.

[0025] The first memory and the second memory may be provided in a circuit board.

[0026] The image acquisition device may further include: a first imaging optical system configured to form an optical image of the object on the first image sensor and including one or more lenses; and a second imaging optical system configured to form an optical image of the object on the second image sensor and including one or more lenses.

[0027] The first imaging optical system and the second imaging optical system may have the same focal length and the same field of view.

[0028] According to another aspect of an example embodiment, there is provided an electronic device including an image acquisition device, the image acquisition device including: an image sensor configured to acquire an image; and a processor configured to: acquire a base based on a surrounding environment, estimate lighting information based on the acquired base, and perform color conversion on the image based on the estimated lighting information.

[0029] According to another aspect of example embodiments, there is provided a control method of an image acquisition device, the method comprising: acquiring an image; acquiring a base based on a surrounding environment of the image acquisition device; estimating illumination information based on the acquired base; and performing color conversion on the image based on the estimated illumination information.

[0030] A basis may be a data set used to estimate lighting information.

[0031] The surrounding environment may be an environment in which the image acquisition device is disposed.

[0032] According to another aspect of an example embodiment, there is provided an image acquisition device including: an image sensor configured to acquire an image; a sensor configured to acquire environmental information of a surrounding environment of the image acquisition device; and a processor configured to: select a substrate from a pre-stored plurality of substrate sets based on the acquired environmental information, estimate illumination spectrum information based on the acquired substrate, and perform color conversion on the image based on the estimated illumination spectrum information. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or other aspects, features and advantages of example embodiments will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0034] Figure 1 is a block diagram of a schematic structure of an image acquisition apparatus according to an example embodiment;

[0035] Figure 2 yes Figure 1 A detailed block diagram of the processor 500 is shown;

[0036] Figure 3 is a detailed block diagram of an image acquisition apparatus according to another example embodiment;

[0037] Figure 4A and Figure 4B is an example diagram used to illustrate the basis set;

[0038] Figure 5 yes Figure 3 A conceptual diagram showing a schematic structure of an image acquisition device is shown;

[0039] Figure 6 is Figure 3 A diagram showing a circuit configuration of a first image sensor and a second image sensor provided in the image acquisition device shown;

[0040] Figure 7 is a graph of a wavelength spectrum of a first image sensor provided in an image acquisition apparatus according to example embodiments;

[0041] Figure 8A 、 Figure 8B and Figure 8C is a diagram illustrating an example pixel arrangement of a first image sensor provided in an image acquisition apparatus according to an example embodiment;

[0042] Figure 9 is a graph of a wavelength spectrum of a second image sensor provided in an image acquisition apparatus according to example embodiments;

[0043] Figure 10A 、 Figure 10B and Figure 10Cis a diagram illustrating an example pixel arrangement of a second image sensor provided in an image acquisition apparatus according to an example embodiment;

[0044] Figure 11 is a flowchart illustrating a method of controlling an image acquisition apparatus according to another example embodiment;

[0045] Figure 12 is a block diagram of a schematic structure of an electronic device according to an example embodiment;

[0046] Figure 13 is Figure 12 A block diagram of a camera module provided in an electronic device; and

[0047] Figure 14 、 Figure 15 、 Figure 16 、 Figure 17 、 Figure 18 、 Figure 19 、 Figure 20 、 Figure 21 、 Figure 22 and Figure 23 are diagrams illustrating various examples of electronic devices to which the image acquisition device according to example embodiments is applied. DETAILED DESCRIPTION

[0048] Reference is now made in detail to the example embodiments shown in the accompanying drawings, in which the same reference numerals throughout the accompanying drawings refer to the same elements. In this regard, exemplary embodiments may have different forms and should not be construed as being limited to the description set forth herein. Therefore, the following description of exemplary embodiments is provided only by reference to the accompanying drawings to explain various aspects. As used herein, the term "and / or" includes any and all combinations of one or more of the relevant listed items. Statements such as "at least one of..." modify the entire list of elements when preceding a list of elements, rather than modifying the individual elements in the list. For example, the statement "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0049] Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings. The example embodiments described below are merely examples, and therefore, it should be understood that the example embodiments can be modified in various forms. The same reference numerals always represent the same elements. In the accompanying drawings, the sizes of the constituent elements may be exaggerated for clarity.

[0050] For example, when an element is referred to as being “on” or “over” another element, it can be directly on the other element or intervening elements may also be present.

[0051] It should be understood that although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. These terms do not limit the material or structure of the components.

[0052] As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, it will be understood that when an element is referred to as "comprising" another element, other elements may not be excluded and may be included unless clearly indicated otherwise.

[0053] In addition, the terms “device,” “component,” and “module” described in the specification refer to units for processing at least one function and / or operation and can be implemented by hardware components or software components and a combination thereof.

[0054] Use of the terms "a," "an," and "the" and similar referents are to be construed to cover both the singular and the plural.

[0055] The operations constituting a method may be performed in any suitable order unless it is explicitly stated that they should be performed in the order described. In addition, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the inventive concept and does not impose limitations on the scope of the present disclosure unless otherwise stated.

[0056] Generally, the value sensed by a camera can be expressed as the product of illumination, object color, and camera response, as shown in Equation 1 below.

[0057] [Equation 1]

[0058] ρ=∫E(λ)S(λ)R(λ)dλ

[0059] Here, ρ is the sensed value, and E(λ), S(λ), and R(λ) are spectral functions of the illumination, the object's surface reflectance, and the camera's response, λ. Because the sensed value is affected by the illumination, the color of the image captured by the camera is also affected by the illumination. White balancing eliminates these effects to capture the object's unique color as much as possible.

[0060] There are two main approaches: normal white balance or automatic white balance (AWB). Normal white balance directly estimates the illumination spectrum, while AWB estimates parameters related to the illumination. In AWB, in a scene with uniform color distribution, a grayscale world algorithm that assumes that the average value of all scene colors is the achromatic color can be used as an example. Next, using the estimated illumination information, a k×n matrix T is calculated to multiply the intensity of each pixel so that the entire scene to which the image belongs is placed under standard illumination. Here, k is the number of channels and n is the number of pixels. By multiplying all pixels in the image by T, a white balance-corrected image is obtained in which each pixel is a k-tuple.

[0061] In related white balancing methods, such as Grayscale World and Max-RGB, when the colors of all bands are unevenly distributed in the scene, the corrected image may produce results that appear to be under colored lighting rather than canonical lighting. In example embodiments, this shortcoming of related methods can be addressed by estimating the lighting spectrum using the high color resolution of a multispectral sensor.

[0062] In an example embodiment, an illumination spectrum is estimated in a relatively small wavelength unit region (eg, 5 nm or less), and a vector for color correction is estimated based on the estimated value of the illumination spectrum.

[0063] The white balance used to estimate the illumination spectrum according to the related method also has the following problems. When estimating the illumination spectrum, a basis (data basis) obtained by performing principal component analysis (hereinafter referred to as PCA) on the spectrum of a known illumination is used. This is based on the assumption that a linear combination of these bases can constitute the actual spectrum of the light. For example, in the case of the D light source series (including D65 illumination, which approximates the spectrum of sunlight at noon to 6500K, which is the temperature of a black body with the closest radiation spectrum), it is known that Figure 4A The linear combination of the three bases 207, 208 and 209 shown can approximate the distortion of the sunlight spectrum that may be caused by various effects such as cloudiness or fog. Figure 4A , the horizontal axis represents the wavelength of light, and the vertical axis represents the illuminance.

[0064] In addition, in the case of artificial lighting other than sunlight, the spectrum may include a peak depending on the characteristics of the fluorescent material used to emit light. When calculating the base including artificial lighting such as the F light source series, in the case of estimating a gentle and relatively flat lighting spectrum such as sunlight, due to the influence of the corresponding peak, such as Figure 4B As shown, the combination of bases 204, 205, and 206 may include peaks that should not be present. Figure 4B , the horizontal axis represents the wavelength of light, and the vertical axis represents the illuminance.

[0065] In example embodiments, to prevent this phenomenon, the illumination spectrum may be estimated more accurately by selecting an appropriate basis set according to surrounding conditions and estimating the illumination spectrum only within the selected basis set.

[0066] refer to Figure 1 The image acquisition device includes an image sensor 10, a sensor 20, a processor 500, and a storage unit 600. The image acquisition device according to example embodiments obtains a basis (i.e., a data set for estimating lighting information based on the surrounding environment in which the image acquisition device is installed), estimates the lighting information using the basis, and performs color conversion on the image by reflecting the estimated lighting information. The image acquisition device according to example embodiments performs white balancing by using a method of directly estimating the lighting spectrum rather than a grayscale world algorithm. Furthermore, the image acquisition device according to example embodiments can more accurately estimate the lighting spectrum by selecting an appropriate basis set based on the surrounding environment and estimating the lighting spectrum only within the selected basis set.

[0067] The image sensor 10 is configured to obtain a specific image. The image sensor 10 may be a multispectral sensor that senses images in multiple bands. The image sensor 10 may be multiple, and may be one RGB sensor and another multispectral image (MSI) sensor. Figure 3 etc. describe the configuration of multiple image sensors.

[0068] The sensor 20 is configured to sense environmental information surrounding the image acquisition device. There may be multiple sensors 20, and they may include sensors related to position and posture, such as GPS, IMU, barometer, and magnetometer. Furthermore, the sensor 20 may be an illumination sensor, a proximity sensor, a distance sensor, a 3D scanner, or the like.

[0069] The sensor 20 may sense surrounding environment information before obtaining an image through the image sensor 10. For example, when the image sensor 10 is not capturing an image, the sensing of the sensor 20 may be performed periodically or aperiodically to track environmental information or environmental variables and changes thereto.

[0070] The processor 500 is configured to control the image acquisition operation of the image sensor 10 and the sensing operation of the sensor 20. The processor 500 obtains ambient environment information from the sensor 20. The processor 500 may select an appropriate base set from a plurality of base sets based on the obtained ambient environment information. The processor 500 may estimate illumination information using the selected base set and perform color conversion on the image obtained from the image sensor 10 by reflecting the estimated illumination information.

[0071] In addition, the processor 500 may perform image processing on the image obtained from the image sensor 10. For example, the processor 500 may perform bad pixel correction, fixed pattern noise correction, crosstalk reduction, re-mosaicing, demosaicing, false color reduction, noise removal, chromatic aberration correction, etc. Here, it is described that the processor 500 performs image processing, but the embodiment is not limited thereto, and it should be understood that the image processing may be performed by a separate image signal processor (hereinafter referred to as ISP).

[0072] The storage unit 600 is configured to store specific bases suitable for indicating illumination and reflectivity. The bases representing illumination and reflectivity can be different bases, specifically for each of illumination and reflectivity. According to another exemplary embodiment, a common base for representing a universal signal can be used in common. The bases stored in the storage unit 600 can be stored by sampling the basis function values ​​of the spectrum. According to another exemplary embodiment, the basis functions can be generated by arithmetic operations. For example, the basis functions for Fourier transform, discrete cosine transform (DCT), wavelet transform, etc. can be determined and used.

[0073] The correspondence between base sets and environments can be preset. For example, the correspondence can be categorized according to various environments (e.g., indoor and outdoor environments, close-up and long-distance capture, complex scenes and monochrome scenes, etc.). After defining and storing the optimal base set for each environment, the base set corresponding to each environment can be read and used.

[0074] In an example embodiment, a basis set can be specified by machine learning. For example, a machine learning method can use a shallow neural network. A neural network can be established that takes the estimated environmental label as input and the label of each basis set calculated using PCA as output, and the neural network is trained using a pre-obtained ordered pair set (environmental label and basis label list). In this case, the output can include consideration of a one-to-many correspondence in the network by using a tuple such as (0, 1, 1, 0, ... 0) to indicate whether a specific basis is used by distinguishing between 0 and 1. By inputting the environmental label of the capture time point of the image acquisition device into the neural network and advancing once, the basis set required to estimate the lighting of the corresponding context can be estimated.

[0075] In an example embodiment, when determining a basis set for an environment, an illumination basis set and a reflectance basis set may be determined separately. According to another example embodiment, for an environment, an illumination basis set may be determined, and a reflectance basis set may use predefined fixed values.

[0076] The processor 500 can estimate the illumination spectrum by performing spectral decomposition using a basis or a set of basis selected in consideration of the environmental information of the acquired image. The transformation required for white balancing is obtained based on the estimated spectrum, and this transformation is used to transform each pixel value of the captured image. In this case, after the transformation generates an RGB vector representing the illumination based on the illumination spectrum, white balancing can be performed by dividing the R, G, and B values ​​of each pixel by the pixel value. According to another exemplary embodiment, white balancing can also be performed by a linear transformation that generates a matrix for RGB conversion based on the illumination spectrum. According to another exemplary embodiment, a nonlinear conversion method such as a neural network that uses the illumination spectrum to convert RGB values ​​can be used.

[0077] Figure 2 yes Figure 1 A detailed block diagram of processor 500 is shown.

[0078] refer to Figure 2 , the processor 500 includes an image processor 510 , an environment determiner 520 , a base generator 530 , an illumination estimator 540 , and a color converter 550 .

[0079] The image processor 510 performs image processing on the image obtained from the image sensor 10. For example, the image processing may include bad pixel correction, fixed pattern noise reduction, demosaicing, denoising, and the like.

[0080] The environment determiner 520 determines the environmental parameters to be used to determine the optimal substrate set. To this end, various sensors, including the image sensor 10 or sensor 20 of the image acquisition device, obtain surrounding environment data before capturing. The various sensors may include RGB image sensors and multispectral image sensors. The various sensors may include sensors related to position and posture, such as GPS, IMU, barometer, and magnetometer. The various sensors may also include sensors such as illumination sensors, proximity sensors, distance sensors, and 3D scanners. In addition, when images are not being captured, sensor sensing may be performed periodically or aperiodically in advance to track environmental variables and their changes.

[0081] The environment determiner 520 identifies environmental information including the environment in which the image acquisition device is currently located from the environmental information provided by the image sensor 10 or the sensor 20. The identified environmental information can be represented as a predefined environmental parameter value. The environmental information or environmental parameter can indicate whether the capture location is indoors or outdoors. According to another example embodiment, the environmental information or environmental parameter can indicate whether LED lighting of a specific band is used in the indoor lighting environment. In addition, the environmental information or environmental parameter can include illumination information. In addition, the environmental information or environmental parameter can include information about the distance to the object. In addition, the environmental information or environmental parameter can include information about the composition of the object within the field of view. In addition, the environmental information or environmental parameter can include information about the user's position and posture.

[0082] In another exemplary embodiment, the environment determiner 520 may analyze the captured image to calculate the environment parameter.The captured image may be an image captured by an RGB image sensor or a multi-directional image sensor, or both images may be used.

[0083] The basis generator 530 generates an appropriate basis based on the environmental information or environmental parameters provided by the environment determiner 520. The basis generator 530 can generate specific basis suitable for representing illumination and reflectivity in advance and store these basis in the storage unit 600. Subsequently, the predefined values ​​can be read from the storage unit 600 and used. In this case, the basis representing illumination and reflectivity can be different basis, specifically for each of illumination and reflectivity. According to another example embodiment, a common basis for representing a universal signal can be used in common. The stored basis can be stored by sampling the basis function values ​​of the spectrum. According to another example embodiment, the basis function can be generated by arithmetic operation.

[0084] The correspondence between the base set and the environment can be preset. For example, the correspondence can be classified according to various environments (such as indoor and outdoor environments, close-up and long-distance capture, complex scenes and monochrome scenes, etc.). After defining and storing the optimal base set for each environment, the base set corresponding to each environment can be read and used. In addition, the base set can be specified by machine learning. An example of a machine learning method that can be used is a shallow neural network. The input can be set to the determined environment label, and the output can be set to the label of each base set calculated using PCA, and an ordered pair set of pre-obtained environment labels and base label lists can be used for training. In this case, the output can include consideration of a one-to-many correspondence in the network by using tuples such as (0, 1, 1, 0, ... 0) to indicate whether a specific base is applicable by distinguishing between 0 and 1. By inputting the environment label of the capture time point of the image acquisition device into the neural network and advancing once, the base set required for estimating the lighting of the corresponding context can be estimated.

[0085] When determining a basis set for an environment, an illumination basis set and a reflectance basis set may be determined separately. According to another exemplary embodiment, for an environment, an illumination basis set may be determined, and a reflectance basis set may use predefined fixed values.

[0086] The illumination estimator 540 estimates an illumination spectrum by performing spectral decomposition using a basis generated by the basis generator 530 with respect to the image input from the image sensor 10 .

[0087] Illumination estimation is performed using spectral information composed of multiple channels in the image. To this end, the illumination and the surface reflectance of the object can be represented by spectral decomposition. The illumination E can be expressed as follows: Equation 2, and the color S of the object can be expressed as follows: Equation 3.

[0088] Equation 2

[0089]

[0090] Equation 3

[0091]

[0092] The product of the two represents a color, and a value sensed by the second image sensor 200 may be expressed by Equation 4 below.

[0093] Equation 4

[0094]

[0095]

[0096] Here, m and n are the number of bases or basis vectors used in the spectral decomposition of illumination and object color, respectively, x is the spatial position, and k is the index of each channel of the sensor, ∈ i is the basis vector E i The coefficient of j is the basis vector S j The illumination spectrum can be estimated by finding a solution to such a linear equation using a nonlinear optimization technique, etc. In an example embodiment, by using a basis or a basis set generated by the basis generator 530 as m and n in Equation 4, the illumination spectrum can be accurately estimated by reflecting environmental changes.

[0097] Alternatively, the lighting estimator 540 may use a neural network to perform lighting estimation. The neural network is constructed by learning images obtained for preset lighting values, and lighting estimation is performed. After learning, the light value is obtained based on the image obtained through the input and output of the neural network.

[0098] Color converter 550 generates the transformation required for white balancing based on the illumination spectrum estimated by illumination estimator 540, and uses this transformation to convert each pixel value of the captured image. In this case, after this transformation generates an RGB vector representing the illumination based on the illumination spectrum, white balancing can be performed by dividing the R, G, and B values ​​of each pixel by the pixel value. According to another exemplary embodiment, white balancing can also be performed using a linear transformation that generates a matrix for RGB conversion based on the illumination spectrum. According to another exemplary embodiment, nonlinear conversion methods such as neural networks that use the illumination spectrum to convert RGB values ​​can be used.

[0099] Figure 3 is a block diagram of a schematic structure of an image acquisition apparatus according to an example embodiment. Figure 3 An example embodiment of Figure 1 and Figure 2 The image sensor 10 shown is implemented as two image sensors 100 and 200, wherein the first image sensor 100 is an RGB image sensor and the second image sensor 200 is a multispectral image sensor. In an exemplary embodiment, image data obtained from the multispectral image sensor is used when estimating the illumination spectrum, and the image data obtained from the multispectral image sensor is not omitted. Figure 2 Same configuration.

[0100] refer to Figure 3, the image acquisition device includes a first image sensor 100, a second image sensor 200, and a processor 500. The image acquisition device according to an example embodiment more accurately performs white balance on an image captured by using multiple image sensors. The first image sensor 100 obtains a first image of a first wavelength band. The second wavelength band may include the first wavelength band and may be larger than the first wavelength band. The second image sensor 200 obtains a second image of the second wavelength band. The first image sensor 100 may be an RGB image sensor, and the second image sensor 200 may be an MSI sensor. The RGB image sensor has an R channel, a G channel, and a B channel. The MSI sensor senses light of more wavelength bands by having more channels than the RGB image sensor.

[0101] The processor 500 matches the first image and the second image outputted from the first image sensor 100 and the second image sensor 200 , respectively, and performs color conversion on the matched images by using an illumination value estimated from the second image.

[0102] In addition, the processor 500 can divide the first image into one or more regions, can estimate each illumination value of each region of the second image corresponding to each divided region of the first image, and can perform color conversion on each divided region of the first image by using each illumination value estimated for each region of the second image.

[0103] Furthermore, when the difference between the illumination values ​​of adjacent regions among the illumination values ​​estimated for the regions of the second image is equal to or greater than a first threshold, the processor 500 may adjust any of the illumination values ​​of the adjacent regions so that the difference is less than the first threshold. In this case, after performing the color conversion, the processor 500 may perform post-processing on the boundary portion of the adjacent regions.

[0104] Reference again Figure 3 The image acquisition device includes: a first image sensor 100 that obtains a first image IM1 based on a first wavelength band; a second image sensor 200 that obtains a second image IM2 based on a second wavelength band; and a processor 500 that performs signal processing on the first image IM1 and the second image IM2 to generate a third image IM3. The third image IM3 is a first image obtained from the first image sensor 100, or an image obtained by matching the first image obtained from the first image sensor 100 with the second image obtained from the second image sensor 200, and white-balanced.

[0105] The first image sensor 100 is a sensor used in a general RGB camera and may be a complementary metal oxide semiconductor (CMOS) image sensor using a Bayer color filter array. The first image IM1 obtained by the first image sensor 100 may be an RGB image based on red, green, and blue.

[0106] The second image sensor 200 is a sensor that senses more wavelengths of light than the first image sensor 100. The second image sensor 200 may use, for example, 16 channels, 31 channels, or another number of channels. The bandwidth of each channel is set to be smaller than the R, G, and B bands, and the total bandwidth, including the bandwidths of all channels, includes the RGB bandwidth (i.e., the visible light bandwidth) and may be larger than the visible light bandwidth. For example, the total bandwidth may be between approximately 350 nm and approximately 1000 nm. The second image IM2 obtained by the second image sensor 200 may be a multispectral image or a hyperspectral image, including wavelengths larger than the RGB bands (e.g., the visible light band), and may be a wavelength-based image that divides the ultraviolet to infrared band larger than the visible light band into 16 or more channels. The second image IM2 may be obtained by using all available channels of the second image sensor 200, or by selecting specific channels. The spatial resolution of the second image IM2 may be lower than that of the first image IM1, but is not limited thereto.

[0107] In an example embodiment, the first image sensor 100 may be an RGB image sensor, and the second image sensor 200 may be an MSI sensor. In this case, the RGB sensor may be a CMOS image sensor. The RGB sensor may sense spectra representing R, G, and B, respectively, by using a Bayer color filter array to generate an image of three channels. In addition, it should be understood that the RGB sensor may use other types of color filter arrays. The MSI sensor senses and displays light of wavelengths different from those of the RGB sensor. The MSI sensor is characterized in that it senses more wavelengths of light by having more channels. In a specific example, the number of channels may be 16. In another example, 31 channels may be used. Each channel may adjust the light transmission band, light transmission amount, and bandwidth to sense light in a desired band. The total bandwidth, which is the sum of the bandwidths of all channels, includes the bandwidth of the existing RGB sensor and may be greater than the bandwidth. The sensing spectrum or band of the RGB sensor and the MSI sensor will be referenced later below. Figures 7 to 10C Provide a description.

[0108] The first image sensor 100 and the second image sensor 200 may be configured as separate chips or a single chip.

[0109] In an exemplary embodiment, timing control can be performed based on the different resolutions and output speeds between different types of sensors, as well as the size of the area required for image matching. For example, when an RGB image stream is read while the RGB sensor is operating, the image stream of the MSI sensor corresponding to that area may already be stored in the buffer or may need to be read again. The sensing signals can be read out by calculating the timing. According to another exemplary embodiment, the operation of the two sensors can be synchronized using the same synchronization signal. In addition, focus control can be performed so that both sensors focus on the object at the same location.

[0110] In an exemplary embodiment, when an image is acquired using an MSI sensor, images of all channels (e.g., 16 channels) may be acquired, or images may be acquired by selecting only specific channels. Only desired channels may be used by merging sensor pixels or by selecting or synthesizing specific channels after image acquisition.

[0111] The first memory 300 stores the first image IM1 read from the first image sensor 100. The second memory 310 stores the second image IM2 read from the second image sensor 200.

[0112] In each sensor, the image is read line by line and stored sequentially. The first memory 300 and the second memory 310 may be line memories for storing images in units of lines, or frame buffers for storing the entire image.

[0113] In an exemplary embodiment, when outputting an image, only an RGB image may be output, and the RGB image may be stored in a frame buffer, the MSI image may be stored in a line buffer and processed line by line, and then the RGB image in the frame buffer may be updated. Memory 300 and memory 310 may use static random access memory (SRAM) or dynamic random access memory (DRAM), but the memory type is not limited thereto.

[0114] Each of the memory 300 and the memory 310 can be located outside the sensor or can be integrated inside the sensor. When integrated inside the sensor, the memory can be integrated with the sensor circuit. In this case, the pixel unit and the circuit unit and memory as other parts can be configured as a separate stack and integrated into two stacks to form a single chip. According to another example embodiment, the memory 300 and the memory 310 can be implemented as three stacks with three layers of pixel unit, circuit unit and memory.

[0115] In example embodiments, it has been described that the first image obtained from the first image sensor and the second image obtained from the second image sensor are stored in different memories, but embodiments are not limited thereto, and the first image and the second image may be stored in one memory.

[0116] In an example embodiment, the processor 500 separates illumination and color of an object using an MSI sensor to accurately perform white balancing to find the exact color of the object, and then performs color conversion on an image obtained from an RGB sensor or a matched image using the illumination value to adjust the white balance.

[0117] Figure 5 is a conceptual diagram of a schematic structure of an image acquisition apparatus according to an exemplary embodiment, and Figure 6 is a diagram of a circuit configuration of a first image sensor and a second image sensor provided in an image acquisition apparatus according to example embodiments.

[0118] Image acquisition device 1000 includes: a first image sensor 100 for acquiring a first image IM1 based on a first wavelength band; a second image sensor 200 for acquiring a second image IM2 based on a second wavelength band; and a processor 500 for performing signal processing on the first and second images IM1 and IM2 to form a third image IM3. Image acquisition device 1000 may also include: a first memory 300 for storing data related to the first image IM1; a second memory 310 for storing data related to the second image IM2; and an image output unit 700 for outputting images.

[0119] The image acquisition device 1000 may further include a first imaging optical system 190 for forming an optical image of the object OBJ on the first image sensor 100, and a second imaging optical system 290 for forming an optical image of the object OBJ on the second image sensor 200. Although each of the first and second imaging optical systems 190 and 290 is shown as including a single lens, this is illustrative and embodiments are not limited thereto. The first and second imaging optical systems 190 and 290 may be configured to have the same focal length and the same field of view. In this case, the process of registering the first and second images IM1 and IM2 to form the third image IM3 may be facilitated. However, embodiments are not limited thereto.

[0120] The first image sensor 100 includes a first pixel array PA1, and the first pixel array PA1 includes a first sensor layer 110 in which a plurality of first sensing elements are arranged and a color filter 120 arranged on the first sensor layer 110. The color filter 120 may include a red filter, a green filter, and a blue filter arranged alternately. A first micro lens array 130 may be on the first pixel array PA1. Figure 8A、 8B 8C to describe various examples of pixel arrangements applied to the first pixel array PA1.

[0121] The second image sensor includes a second pixel array PA2, and the second pixel array PA2 includes a second sensor layer 210 in which a plurality of second sensing elements are arranged, and a spectral filter 220 arranged on the second sensor layer 210. The spectral filter 220 includes a plurality of filter groups, and each of the plurality of filter groups may include a plurality of unit filters having different transmission bands. The spectral filter 220 may be configured to filter light in a band wider than the band of the color filter 120 in a more subdivided manner than the color filter 120, such as a band in the ultraviolet to infrared wavelength range. A second microlens array 230 may be on the second pixel array PA2. Reference will be made to Figure 10A 、 Figure 10B and Figure 10C An example of a pixel arrangement applied to the second pixel array PA2 will be described.

[0122] The first sensor layer 110 and the second sensor layer 210 may include a charge coupled device (CCD) sensor or a CMOS sensor, but the embodiment is not limited thereto.

[0123] The first pixel array PA1 and the second pixel array PA2 may be horizontally arranged on, for example, a circuit board SU, and spaced apart from each other in the X direction.

[0124] The circuit board SU may include a first circuit element for processing a signal from the first sensor layer 110 and a second circuit element for processing a signal from the second sensor layer 210. However, the embodiment is not limited thereto, and the first and second circuit elements may be provided on separate substrates, respectively.

[0125] Although the memory storing the data of the first image IM1 and the second image IM2 is shown separately from the circuit board SU, this is merely an example, and the memory may be arranged on the same layer as the circuit elements in the circuit board SU or as a separate layer. The memory may be a line memory that stores images line by line, or a frame buffer that stores the entire image. The memory may be an SRAM or a DRAM.

[0126] The various circuit elements required for the image acquisition device 1000 can be integrated and arranged on the circuit board SU. For example, a logic layer including various analog and digital circuits can be provided, and a memory layer for storing data can be provided. The logic layer and the memory layer can be configured as different layers or the same layer.

[0127] refer to Figure 6, the row decoder 102, the output circuit 103 and the timing controller (TC) 101 are connected to the first pixel array PA1. The row decoder 102 selects a row of the first pixel array PA1 in response to the row address signal output from the timing controller 101. The output circuit 103 outputs light sensing signals from a plurality of pixels arranged along the selected row in units of columns. To this end, the output circuit 103 may include a column decoder and an analog-to-digital converter (ADC). For example, the output circuit 103 may include a plurality of ADCs respectively arranged for each column between the column decoder and the first pixel array PA1, or an ADC arranged at the output end of the column decoder. The timing controller 101, the row decoder 102 and the output circuit 103 may be implemented as one chip or separate chips. At least some of the circuit elements shown may be arranged in Figure 5 The processor for processing the first image IM1 output from the output circuit 103 may be implemented as a single chip together with the timing controller 101, the row decoder 102, and the output circuit 103.

[0128] The row decoder 202, the output circuit 203, and the timing controller (TC) 201 are also connected to the second pixel array PA2 and, similarly to the above, can process signals from the second pixel array PA2. In addition, a processor for processing the second image IM2 output from the output circuit 203 can be implemented as a single chip together with the timing controller 201, the row decoder 202, and the output circuit 203.

[0129] Figure 6 It is shown that the first pixel array PA1 and the second pixel array PA2 have the same size and the same number of pixels, however, the embodiment is not limited thereto.

[0130] When operating two different types of sensors, timing control may be required based on the different resolutions and output speeds, as well as the area size required for image matching. For example, when reading an image column based on the first image sensor 100, the image column corresponding to the area of ​​the second image sensor 200 may already be stored in the buffer or may need to be read again. According to another example embodiment, the operation of the first image sensor 100 and the second image sensor 200 can be synchronized using the same synchronization signal. For example, the timing controller (TC) 400 can also be configured to send a synchronization signal sync to the first image sensor 100 and the second image sensor 200.

[0131] Figure 7 is a diagram of a wavelength spectrum of a first image sensor provided in an image acquisition apparatus according to example embodiments, and Figures 8A to 8C is a diagram of an exemplary pixel arrangement of a first image sensor provided in an image acquisition apparatus according to example embodiments.

[0132] refer to Figure 8A In the color filter 120 provided in the first pixel array PA1, filters for filtering the red (R), green (G), and blue (B) bands are arranged in a Bayer pattern. For example, one unit pixel includes sub-pixels arranged in a 2×2 array, and a plurality of unit pixels are repeatedly arranged two-dimensionally. Red filters and green filters are arranged in one row of unit pixels, and green filters and blue filters are arranged in a second row. Pixels may be arranged in a manner other than the Bayer pattern.

[0133] For example, reference Figure 8B , a CYGM arrangement in which a magenta pixel (M), a cyan pixel (C), a yellow pixel (Y), and a green pixel (G) constitute one unit pixel may also be performed. Figure 8C Alternatively, an RGBW arrangement may be performed in which a green pixel (G), a red pixel (R), a blue pixel (B), and a white pixel (W) constitute a unit pixel. Furthermore, the unit pixel may have a 3×2 array shape. Furthermore, the pixels of the first pixel array PA1 may be arranged in various ways according to the color characteristics of the first image sensor 100.

[0134] Figure 9 is a diagram of a wavelength spectrum of a second image sensor provided in an image acquisition apparatus according to example embodiments, 10A to 10C is a diagram of an exemplary pixel arrangement of a second image sensor provided in an image acquisition device according to example embodiments.

[0135] refer to Figure 10A The spectral filter 220 provided in the second pixel array PA2 may include a plurality of filter groups 221 arranged in a two-dimensional form. Each filter group 221 may include 16 unit filters F1 to F16 arranged in a 4×4 array.

[0136] The first and second unit filters F1 and F2 may have central wavelengths UV1 and UV2 in the ultraviolet region, and the third, fourth, and fifth unit filters F3, F4, and F5 may have central wavelengths B1, B2, and B3 in the blue region. The sixth, seventh, eighth, ninth, and eleventh unit filters F6, F7, F8, F9, F10, and F11 may have central wavelengths G1, G2, G3, G4, G5, and G6 in the green region, and the twelfth, thirteenth, and fourteenth unit filters F12, F13, and F14 may have central wavelengths R1, R2, and R3 in the red region. Furthermore, the fifteenth and sixteenth unit filters F15 and F16 may have central wavelengths NIR1 and NIR2 in the near-infrared region.

[0137] Figure 10B FIG. 2 is a plan view of another example of a filter group 222 provided in the spectral filter 220. Figure 10B , the filter group 222 may include nine unit filters F1 to F9 arranged in a 3×3 array. The first unit filter F1 and the second unit filter F2 may have central wavelengths UV1 and UV2 in the ultraviolet region, and the fourth unit filter F4, the fifth unit filter F5, and the seventh unit filter F7 may have central wavelengths B1, B2, and B3 in the blue light region. The third unit filter F3 and the sixth unit filter F6 may have central wavelengths G1 and G2 in the green light region, and the eighth unit filter F8 and the ninth unit filter F9 may have central wavelengths R1 and R2 in the red light region.

[0138] Figure 10C FIG. 2 is a plan view of another example of a filter group 223 provided in the spectral filter 220. Figure 12, the filter group 223 may include 25 unit filters F1 to F25 arranged in a 5×5 array. The first unit filter F1, the second unit filter F2, and the third unit filter F3 may have central wavelengths UV1, UV2, and UV3 in the ultraviolet region, and the sixth unit filter F6, the seventh unit filter F7, the eighth unit filter F8, the eleventh unit filter F11, and the twelfth unit filter F12 may have central wavelengths B1, B2, B3, B4, and B5 in the blue light region. The fourth unit filter F4, the fifth unit filter F5, and the ninth unit filter F9 may have center wavelengths G1, G2, and G3 in the green light region, and the tenth unit filter F10, the thirteenth unit filter F13, the fourteenth unit filter F14, the fifteenth unit filter F15, the eighteenth unit filter F18, and the nineteenth unit filter F19 may have center wavelengths R1, R2, R3, R4, R5, and R6 in the red light region. In addition, the twentieth unit filter F20, the twenty-third unit filter F23, the twenty-fourth unit filter F24, and the twenty-fifth unit filter F25 may have center wavelengths NIR1, NIR2, NIR3, and NIR4 in the near-infrared region.

[0139] The unit filter provided in the spectral filter 220 may have a resonant structure including two reflective plates, and the transmission band may be determined based on the characteristics of the resonant structure. The transmission band may be adjusted based on the material of the reflective plates, the material of the dielectric material in the cavity, and the thickness of the cavity. Furthermore, structures using gratings, distributed Bragg reflectors (DBRs), and the like may be applied to the unit filter. Furthermore, the pixels of the second pixel array PA2 may be arranged in various ways based on the color characteristics of the second image sensor 200.

[0140] Figure 11 is a flowchart illustrating a method of controlling an image acquisition apparatus according to another example embodiment.

[0141] refer to Figure 11 In operation 1100, an image is obtained from an image sensor. The image sensor may be a multispectral image sensor.

[0142] In operation 1102, a base is obtained based on the surrounding environment. Sensors used to sense the surrounding environment may include image sensors (including RGB image sensors and multispectral image sensors), sensors related to position and posture (such as GPS, IMU, barometer, and magnetometer), illumination sensors, proximity sensors, distance sensors, 3D scanners, and the like. The sensors used to sense the surrounding environment pre-sense environmental information before obtaining an image in operation 1100. Furthermore, the sensors used to sense the surrounding environment may track environmental information and changes in environmental information by performing periodic or aperiodic sensing before obtaining an image. Environmental information may be represented as predefined environmental parameter values. Environmental information or environmental parameters may indicate whether the capture location is indoors or outdoors. According to another example embodiment, the environmental information or environmental parameters may indicate whether LED lighting of a specific wavelength band is used in an indoor lighting environment. Furthermore, the environmental information or environmental parameters may include illumination information. Furthermore, the environmental information or environmental parameters may include information regarding the distance to an object. Furthermore, the environmental information or environmental parameters may include information regarding the composition of objects within the field of view. Furthermore, the environmental information or environmental parameters may include information regarding the user's position and posture.

[0143] In operation 1104, the obtained basis is used to estimate illumination information. The method of estimating illumination information using the basis has been described above with reference to Equations 1 to 4. In example embodiments, the method defined in Equations 1 to 4 is used, but the embodiment is not limited thereto, and various numerical methods for estimating illumination information or an illumination spectrum may be used.

[0144] In operation 1106 , color conversion is performed on the image by reflecting the estimated illumination information.

[0145] A method of controlling an image acquisition device according to an example embodiment may include: obtaining a specific image; obtaining a base according to a surrounding environment; estimating lighting information using the obtained base; and performing color conversion on the image by reflecting the estimated lighting information; thereby more accurately estimating lighting in various situations and maintaining color stability by performing accurate white balance.

[0146] The image acquisition device 1000 can be applied to various high-performance optical devices or high-performance electronic devices. The electronic device can be, for example, a smartphone, a mobile phone, a cellular phone, a personal digital assistant (PDA), a laptop computer, a personal computer (PC), various portable devices, home appliances, security cameras, medical cameras, automobiles, Internet of Things (IoT) devices, or other mobile or non-mobile computing devices, but is not limited thereto.

[0147] In addition to the image acquisition device 1000, the electronic device may further include a processor for controlling the image sensor provided therein, such as an application processor (AP), which can drive an operating system or application program to control multiple hardware or software components and perform various data processing and operations. The processor may also include a graphics processing unit (GPU) and / or an image signal processor. When the processor includes an image signal processor, the processor may be used to store and / or output images (or videos) obtained by the image sensor.

[0148] Figure 12 is a block diagram of a schematic structure of an electronic device according to an example embodiment. Figure 12 In a network environment ED00, an electronic device ED01 can communicate with another electronic device ED02 via a first network ED98 (a near-field wireless communication network, etc.), or can communicate with another electronic device ED04 and / or a server ED08 via a second network ED99 (a far-field wireless communication network, etc.). The electronic device ED01 can communicate with the electronic device ED04 via the server ED08. The electronic device ED01 may include a processor ED20, a memory ED30, an input device ED50, an audio output device ED55, a display device ED60, an audio module ED70, a sensor module ED76, an interface ED77, a tactile module ED79, a camera module ED80, a power management module ED88, a battery ED89, a communication module ED90, a subscriber identification module ED96, and / or an antenna module ED97. In the electronic device ED01, some components (such as the display device ED60) may be omitted, or other components may be added. Some of these components may be implemented in a single integrated circuit. For example, the sensor module ED76 (fingerprint sensor, iris sensor, illumination sensor, etc.) can be implemented by being embedded in the display device ED76 (display, etc.). In addition, when the image sensor 1000 includes a spectral function, some functions of the sensor module (color sensor or light sensor) can be implemented in the image sensor 1000 rather than in a separate sensor module.

[0149] Processor ED20 can execute software (program ED40, etc.) to control one or more other components (hardware or software components, etc.) of electronic device ED01 connected to processor ED20, and can perform various data processing or operations. As part of the data processing or operation, processor ED20 can load instructions and / or data received from other components (sensor module ED76, communication module ED90, etc.) into volatile memory ED32, can process the instructions and / or data stored in volatile memory ED32, and can store the resulting data in non-volatile memory ED34. Processor ED20 may include a main processor ED21 (central processing unit, application processor, etc.) and a coprocessor ED23 (graphics processing unit, image signal processor, sensor hub processor, communication processor, etc.) that can operate independently or together. Coprocessor ED23 uses less power than main processor ED21 and can perform dedicated functions.

[0150] Coprocessor ED23 can control functions and / or states related to some of the components of electronic device ED01 (display device ED60, sensor module ED76, communication module ED90, etc.) on behalf of main processor ED21 when main processor ED21 is inactive (e.g., sleeping), or together with main processor ED21 when main processor ED21 is active (e.g., executing an application). Coprocessor ED23 (image signal processor, communication processor, etc.) can be implemented as part of other functionally related components (camera module ED80, communication module ED90, etc.).

[0151] Memory ED30 can store various data required by the components of electronic device ED01 (processor ED20, sensor module ED76, etc.). This data may include, for example, software (program ED40, etc.) and input data and / or output data for commands associated therewith. Memory ED30 may include volatile memory ED32 and / or nonvolatile memory ED34. Nonvolatile memory ED32 may include internal memory ED36 fixed to electronic device ED01 and removable external memory ED38.

[0152] Program ED40 may be stored as software in memory ED30 and may include an operating system ED42 , middleware ED44 , and / or applications ED46 .

[0153] The input device ED50 may receive commands and / or data for components (processor ED20, etc.) of the electronic device ED01 from outside (user, etc.) of the electronic device ED01. The input device ED50 may include a microphone, a mouse, a keyboard, and / or a digital pen (stylus, etc.).

[0154] The audio output device ED55 can output audio signals to the outside of the electronic device ED01. The audio output device ED55 may include a speaker and / or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to answer incoming calls. The receiver may be integrated into the speaker or implemented as a separate device.

[0155] The display device ED60 can visually provide information to the outside of the electronic device ED01. The display device ED60 may include a display, a holographic device, or a projector, as well as control circuits for controlling these devices. The display device ED60 may include a touch circuit configured to sense touch and / or a sensor circuit (such as a pressure sensor) configured to measure the strength of the force generated by the touch.

[0156] The audio module ED70 can convert sound into electrical signals and vice versa. The audio module ED70 can obtain sound through the input device ED50, or can output sound through the audio output device ED55 and / or the speaker and / or headphones of another electronic device (such as the electronic device ED02) directly or wirelessly connected to the electronic device ED01.

[0157] The sensor module ED76 can detect the operating state (power, temperature, etc.) or external environmental state (user status, etc.) of the electronic device ED01 and can generate an electrical signal and / or data value corresponding to the detected state. The sensor module ED76 can include a gesture sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, and / or an illumination sensor.

[0158] Interface ED77 may support one or more designated protocols that can be used to connect electronic device ED01 directly or wirelessly to other electronic devices (such as electronic device ED02). Interface ED77 may include a High-Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital (SD) card interface, and / or an audio interface.

[0159] The connection terminal ED78 may include a connector through which the electronic device ED01 can be physically connected to other electronic devices (such as the electronic device ED02). The connection terminal ED78 may include an HDMI connector, a USB connector, an SD card connector, and / or an audio connector (such as a headphone connector).

[0160] The haptic module ED79 may convert electrical signals into mechanical stimulation (vibration, motion, etc.) or electrical stimulation that a user may sense through tactile or kinesthetic sensations. The haptic module ED79 may include a motor, a piezoelectric element, and / or an electrical stimulation device.

[0161] The camera module ED80 can capture both still and moving images. The camera module ED80 may include the image acquisition device 1000 described above, and may include an additional lens assembly, an image signal processor, and / or a flash. The lens assembly included in the camera module ED80 may collect light emitted from the object to be captured.

[0162] The power management module ED88 may manage power supplied to the electronic device ED01 and may be implemented as part of a power management integrated circuit PMIC.

[0163] The battery ED89 can supply power to the components of the electronic device ED01. The battery ED89 can include a non-rechargeable primary battery, a rechargeable secondary battery, and / or a fuel cell.

[0164] The communication module ED90 can support the establishment of direct (wired) communication channels and / or wireless communication channels between the electronic device ED01 and other electronic devices (electronic device ED02, electronic device ED04, server ED08, etc.), and communicate through the established communication channels. The communication module ED90 operates independently of the processor ED20 (application processor, etc.) and can include one or more communication processors that support direct communication and / or wireless communication. The communication module ED90 can include a wireless communication module ED92 (cellular communication module, short-range wireless communication module, global navigation satellite system (GNSS) module, etc.) and / or a wired communication module ED94 (local area network (LAN) communication module, power line communication module, etc.). Respective communication modules among these communication modules can communicate with other electronic devices via a first network ED98 (a local area network such as Bluetooth, Wi-Fi Direct, or Infrared Data Association (IrDA)) or a second network ED99 (a telecommunications network such as a cellular network, the Internet, or a computer network (LAN, WAN, etc.)). These various types of communication modules can be integrated into a single component (a single chip, etc.) or implemented as multiple separate components (multiple chips). The wireless communication module ED92 can identify and authenticate the electronic device ED01 within a communication network (eg, the first network ED98 and / or the second network ED99 ) using subscriber information (International Mobile Subscriber Identity (IMSI) etc.) stored in the subscriber identification module ED96 .

[0165] The antenna module ED97 can send signals and / or power to the outside (other electronic devices, etc.), and / or receive signals and / or power from the outside (other electronic devices, etc.). The antenna may include a radiator made of a conductive pattern formed on a substrate (PCB, etc.). The antenna module ED97 may include one or more antennas. When multiple antennas are included, the communication module ED90 can select an antenna suitable for the communication method used in the communication network (such as the first network ED98 and / or the second network ED99) from the multiple antennas. Signals and / or power can be sent or received between the communication module ED90 and other electronic devices through the selected antenna. Other components (RFIC, etc.) in addition to the antenna may be included as part of the antenna module ED97.

[0166] Some components can be connected to each other and exchange signals (commands, data, etc.) through a communication method between peripheral devices (bus, general-purpose input and output (GPIO), serial peripheral interface (SPI), mobile industry processor interface (MIPI), etc.).

[0167] Commands or data can be sent or received between the electronic device ED01 and the electronic device ED04 as an external device through the server ED08 connected to the second network ED99. The other electronic devices ED02 and ED04 may be the same as or different from the electronic device ED01. All or part of the operations performed by the electronic device ED01 may be performed by one or more of the other electronic devices ED02, ED04 and ED08. For example, when the electronic device ED01 needs to perform a specific function or service, the electronic device ED01 may request one or more other electronic devices to perform part or all of these functions or services instead of performing these functions or services directly. The one or more other electronic devices that receive the request may perform additional functions or services related to the request, and may transmit the results of the execution to the electronic device ED01. To this end, cloud computing, distributed computing and / or client-server computing technologies may be used.

[0168] Figure 13 yes Figure 12 The camera module ED80 may include the image acquisition device 1000 described above, or may have a structure modified from the image acquisition device 1000. Figure 13 The camera module ED80 may include a lens assembly CM10, a flash CM20, an image sensor CM30, an image stabilizer CM40, a memory CM50 (buffer memory, etc.), and / or an image signal processor CM60.

[0169] The image sensor CM30 may include the first image sensor 100 and the second image sensor 200 provided in the image acquisition device 1000 described above. The first image sensor 100 and the second image sensor 200 may obtain an image corresponding to an object by converting light emitted or reflected from the object and transmitted through the lens assembly CM10 into an electrical signal. The first image sensor 100 may obtain an RGB image, and the second image sensor 200 may obtain a hyperspectral image in the ultraviolet to infrared wavelength range.

[0170] In addition to the first image sensor 100 and the second image sensor 200 described above, the image sensor CM30 may further include one or more sensors selected from image sensors with different characteristics (e.g., another RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor). Each sensor included in the image sensor CM30 may be implemented as a CCD sensor and / or a CMOS sensor.

[0171] The lens assembly CM10 can collect light emitted from the object to be captured. The camera module ED80 may include multiple lens assemblies CM10, and in this case, it may be a dual camera, a 360-degree camera, or a spherical camera. Some of the multiple lens assemblies CM10 may have the same lens properties (field of view, focal length, autofocus, F number, optical zoom, etc.) or different lens properties. The lens assembly CM10 may include a wide-angle lens or a telephoto lens.

[0172] The lens assembly CM10 may be configured such that two image sensors included in the image sensor CM30 form optical images of an object at the same position and / or at the same focus.

[0173] The flash CM20 may emit light for enhancing light emitted or reflected from an object and may include one or more light emitting diodes (red, green, and blue (RGB) LEDs, white light LEDs, infrared LEDs, ultraviolet LEDs, etc.) and / or xenon lamps.

[0174] Image stabilizer CM40 can move one or more lenses included in lens assembly CM10 or image sensor 1000 in a specific direction in response to movement of camera module ED80 or electronic device CM01 including the camera module, or can control the operating characteristics of image sensor 1000 (such as readout timing adjustment) to compensate for the negative effects caused by the movement. Image stabilizer CM40 can use a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside camera module ED80 to detect movement of camera module ED80 or electronic device ED01. Image stabilizer CM40 can be implemented optically.

[0175] Memory CM50 can store some or all of the data acquired by image sensor 1000 for subsequent image processing operations. For example, when acquiring multiple images at high speed, the acquired raw data (Bayer pattern data, high-resolution data, etc.) can be stored in memory CM50, and only low-resolution images can be displayed. The raw data of a selected image (such as a user selection) can then be transferred to image signal processor CM60. Memory CM50 can be integrated into memory ED30 of electronic device ED01, or can be configured as a separate memory that operates independently.

[0176] The image signal processor CM60 may perform one or more image processing operations on the image obtained by the image sensor CM30 or the image data stored in the memory CM50. Figures 1 to 10C As described above, the first image (e.g., RGB image) and the second image (e.g., MSI image) obtained by the two image sensors included in the image sensor CM30 are processed to generate a third image on which white balance is performed. The configuration of the processor 500 for this purpose may be included in the image signal processor CM60.

[0177] In addition, one or more image processing may include depth map generation, three-dimensional modeling, panorama generation, feature point extraction, image synthesis and / or image compensation (noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, softening, etc.). The image signal processor CM60 can control (exposure time control, readout timing control, etc.) the components (image sensor CM30, etc.) included in the camera module CM80. The image processed by the image signal processor CM60 can be stored again in the memory CM50 for further processing, or can be provided to external components of the camera module ED80 (memory ED30, display device ED60, electronic device ED02, electronic device ED04, server ED08, etc.). The image signal processor CM60 can be integrated into the processor CM20, or can be configured as a separate processor that operates independently of the processor CM20. When the image signal processor CM60 is configured as a processor separate from the processor ED20, the image processed by the image signal processor CM60 can be displayed through the display device ED60 after further image processing by the processor ED20.

[0178] The electronic device ED01 may include multiple camera modules ED80 having corresponding properties or functions. In this case, one of the multiple camera modules ED80 may be a wide-angle camera, and the other camera modules ED80 may be telephoto cameras. Similarly, one of the multiple camera modules ED80 may be a front-facing camera, and the other camera modules ED80 may be a rear-facing camera.

[0179] Figures 14 to 23 are views illustrating various examples of electronic devices to which the image acquiring device according to example embodiments is applied.

[0180] The image acquisition apparatus according to example embodiments may be applied to Figure 14 The mobile phone or smart phone 5100m shown, Figure 15 The tablet computer or smart tablet computer 5200 shown, Figure 16 The digital camera or video camera 5300 shown, Figure 17 The laptop computer 5400 shown, or Figure 18 The television or smart TV 5500 shown is shown. For example, a smart phone 5100m or a smart tablet computer 5200 may include multiple high-resolution cameras, each of which has a high-resolution image sensor mounted thereon. The high-resolution camera can be used to extract depth information of objects in an image, adjust the focus of an image, or automatically identify objects in an image.

[0181] In addition, the image acquisition device 1000 can be applied to Figure 19 The smart refrigerator 5600 shown, Figure 20 The security camera 5700 shown, Figure 21 The robot 5800 shown, Figure 22 The medical camera 5900 shown in the figure is shown. For example, the smart refrigerator 5600 can automatically identify the food in the refrigerator using the image acquisition device 1000 and notify the user through the smartphone of the presence of specific food, the type of food received or delivered, etc. The security camera 5700 can provide ultra-high-resolution images and can identify objects or people in the image even in dark environments by using high sensitivity. The robot 5800 can provide high-resolution images by being input into disaster or industrial sites that cannot be directly accessed by humans. The medical camera 5900 can provide high-resolution images for diagnosis or surgery and can dynamically adjust the field of view.

[0182] In addition, the image acquisition device 1000 can be applied to Figure 23 Vehicle 6000 is shown. Vehicle 6000 may include multiple onboard cameras 6010, 6020, 6030, and 6040 arranged at various locations. Each of onboard cameras 6010, 6020, 6030, and 6040 may include an image acquisition device according to an example embodiment. Vehicle 6000 may use multiple onboard cameras 6010, 6020, 6030, and 6040 to provide the driver with various information about the interior and surroundings of vehicle 6000, and may provide information required for autonomous driving by automatically identifying objects or people in images.

[0183] An image acquisition apparatus according to example embodiments may more accurately estimate illumination in various situations and more accurately perform white balance to maintain color stability.

[0184] The image acquisition device described above can be applied to various electronic devices.

[0185] It should be understood that the example embodiments described herein should be considered to be descriptive only and not for purposes of limitation. The description of features or aspects in each exemplary embodiment should typically be considered to be applicable to other similar features or aspects in other embodiments. Although example embodiments have been described with reference to the accompanying drawings, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope defined by the appended claims and their equivalents.

Claims

1. An image acquisition device, comprising: an image sensor configured to acquire an image; at least one sensor configured to sense environmental information of a surrounding environment in which the image acquisition device is disposed; as well as The processor is configured to: selecting a basis set from a plurality of pre-stored basis sets based on the environmental information, wherein the basis set is a data set for estimating illumination information according to an ambient environment in which the image acquisition device is disposed; estimating illumination information by performing spectral decomposition on the obtained image based on a selected basis set; as well as Color conversion is performed on the image based on the estimated illumination information.

2. The image acquisition device according to claim 1, wherein: The processor is configured to periodically or non-periodically obtain the environmental information through the at least one sensor before the image sensor obtains the image.

3. The image acquisition device according to claim 1, wherein: The at least one sensor includes: a GPS sensor, an IMU sensor, a barometer, a magnetometer, an illumination sensor, a proximity sensor, a distance sensor, or a three-dimensional scanner.

4. The image acquisition device according to claim 1, wherein: The image sensor is further configured to sense images in multiple wavelength bands.

5. The image acquisition device according to claim 1, wherein: The processor is further configured to: analyzing the image acquired by the image sensor; extracting the environmental information from the analyzed image; and The base set is selected from the pre-stored plurality of base sets based on the extracted environment information.

6. The image acquisition device according to claim 1, further comprising: a storage device configured to store a plurality of base sets including wavelength-based illumination and reflectivity, The processor is further configured to select a basis set corresponding to the obtained environment information from the plurality of pre-stored basis sets.

7. The image acquisition device according to claim 1, wherein: The image sensor comprises: A first image sensor configured to obtain a first image in a first wavelength band; and a second image sensor configured to obtain a second image in a second wavelength band; The image includes the first image and the second image.

8. The image acquisition device according to claim 7, wherein: The first image sensor comprises: a first sensor layer having a plurality of first sensing elements disposed therein; and a first pixel array having color filters disposed on the first sensor layer, the color filters including red filters, green filters, and blue filters disposed alternately, and Wherein, the second image sensor comprises: a second sensor layer having a plurality of second sensing elements disposed therein; and The second pixel array has a spectral filter in which a filter group including a plurality of unit filters having different transmission wavelength bands is repeatedly arranged, and the spectral filter is provided on the second sensor layer.

9. The image acquisition device according to claim 8, wherein: Each of the transmission wavelength bands of the plurality of unit filters includes visible light and is included in a wavelength band larger than a visible light band, and The filter group includes 16 unit filters arranged in a 4×4 array.

10. The image acquisition device according to claim 8, wherein: The first pixel array and the second pixel array are horizontally spaced apart from each other on a circuit board.

11. The image acquisition device according to claim 10, wherein: A first circuit element configured to process a signal from the first sensor layer and a second circuit element configured to process a signal from the second sensor layer are provided on the circuit board.

12. The image acquisition device according to claim 11, further comprising: A timing controller is configured to synchronize operations of the first circuit element and the second circuit element.

13. The image acquisition device according to claim 10, further comprising: a first memory configured to store data corresponding to the first image; as well as The second memory is configured to store data corresponding to the second image.

14. The image acquisition device according to claim 13, wherein: The first memory and the second memory are provided in the circuit board.

15. The image acquisition device according to claim 7, further comprising: a first imaging optical system configured to form an optical image of an object on the first image sensor and comprising one or more lenses; as well as The second imaging optical system is configured to form an optical image of the object on the second image sensor and includes one or more lenses.

16. The image acquisition device according to claim 15, wherein: The first imaging optical system and the second imaging optical system have the same focal length and the same field of view.

17. An electronic device comprising an image acquisition device, the image acquisition device comprising: an image sensor configured to acquire an image; at least one sensor configured to sense environmental information of a surrounding environment in which the image acquisition device is disposed; as well as The processor is configured to: selecting a basis set from a plurality of pre-stored basis sets based on the environmental information, wherein the basis set is a data set for estimating illumination information according to an ambient environment in which the image acquisition device is disposed; estimating illumination information by performing spectral decomposition on the obtained image based on a selected basis set; as well as Color conversion is performed on the image based on the estimated illumination information.

18. A method for controlling an image acquisition device, the method comprising: Obtain an image; sensing environmental information of a surrounding environment in which the image acquisition device is disposed; selecting a basis set from a plurality of pre-stored basis sets based on the environmental information, wherein the basis set is a data set for estimating illumination information according to an ambient environment in which the image acquisition device is disposed; estimating illumination information by performing spectral decomposition on the obtained image based on a selected basis set; as well as Color conversion is performed on the image based on the estimated illumination information.

19. An image acquisition device, comprising: an image sensor configured to acquire an image; a sensor configured to obtain environmental information of a surrounding environment of the image acquisition device; as well as The processor is configured to: selecting a base set from a plurality of pre-stored base sets based on the obtained environmental information, wherein the base set is a data set for estimating illumination information according to an ambient environment in which the image acquisition device is disposed; estimating illumination spectrum information by performing spectral decomposition on the obtained image based on a selected basis set; as well as Color conversion is performed on the image based on the estimated illumination spectrum information.

Images