Image acquisition apparatus and method of acquisiting image

Abstract

There is provided an image acquisition apparatus including: a multispectral image sensor configured to acquire an image of at least one object in an environment in which at least one illumination source is present through eight or more channels in which an overlap between the channels is minimal; and a processor configured to estimate illumination spectrum data of the acquired image by using channel signals corresponding to the eight or more channels, and perform lens shading correction on the acquired image based on the estimated illumination spectrum data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2023-0079931 filed in the Korean Intellectual Property Office on June 21, 2023, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to an image acquisition device and an image acquisition method. Background Art

[0004] Cameras commonly used in electronic devices (e.g., smart phones) include RGB image sensors. The RGB image sensor may be configured to acquire an image using three channels including an R channel, a G channel, and a B channel. The camera or electronic device may be configured to generate or correct an image based on a signal acquired from the RGB image sensor. For example, the camera or electronic device may be configured to estimate the color temperature of the illumination source based on the signals acquired through the three channels of the RGB image sensor, and correct the image based on the estimated color temperature.

[0005] However, since illumination sources may have different spectral characteristics even at the same color temperature, it may be difficult to accurately distinguish different illumination sources using only color temperature. Therefore, a technique for accurately distinguishing and estimating illumination sources having different spectral characteristics and accurately correcting an image by using information about the estimated illumination sources may be required. Summary of the invention

[0006] One or more embodiments provide an image acquisition device and a method for acquiring an image. The technical objectives of the present disclosure are not limited to the aforementioned technical objectives, and other technical objectives can be derived from the following embodiments.

[0007] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the embodiments presented in the disclosure.

[0008] According to one aspect of the present disclosure, an image acquisition device includes: a multispectral image sensor configured to acquire an image of at least one object in an environment where at least one illumination source exists through eight or more channels with minimal overlap between the channels; and a processor configured to estimate illumination spectral data of the acquired image by using channel signals corresponding to the eight or more channels, and perform lens shading correction on the acquired image based on the estimated illumination spectral data.

[0009] The processor may be configured to generate a correction map for each of the eight or more channels by performing linear interpolation on a correction table corresponding to the estimated illumination spectral data.

[0010] The correction table may include correction data for each channel, and the correction data may include at least one of a scaling factor, an offset, an incident angle, and a curve fitting coefficient corresponding to the estimated illumination spectrum data.

[0011] The processor may be configured to perform lens shading correction on all RAW images acquired from the multispectral image sensor or images according to channels by using the generated correction map.

[0012] The processor may be configured to estimate the illumination spectral data based on eight or more basis vectors associated with reflectance characteristics of at least one object included in the acquired image, the number of channels, and the number of pixels in the acquired image.

[0013] The processor may be configured to estimate lighting spectral data by using a pre-trained machine learning model.

[0014] The processor may be configured to divide the acquired image into a plurality of regions and estimate illumination spectrum data for each of the plurality of regions.

[0015] The estimated illumination spectrum data may include at least one of: spectral data including intensity of channel signals according to wavelength, a color vector of the illumination source in a color space, a color temperature of the illumination source, and an index indicating a predetermined illumination type corresponding to the illumination source.

[0016] The processor is configured to further perform white balance correction on the image for which lens shading correction has been performed based on the estimated illumination spectrum data.

[0017] The multispectral image sensor may include a multispectral image sensor for which color calibration has been performed on at least one illumination source.

[0018] According to another aspect of the present disclosure, an image acquisition device includes: a multispectral image sensor configured to acquire a first image of at least one object in an environment where at least one illumination source exists through eight or more first channels with minimal overlap between channels; an image sensor configured to acquire a second image of at least one object in an environment where at least one illumination source exists through less than four second channels; and a processor configured to estimate illumination spectral data of the first image by using channel signals corresponding to the eight or more first channels, and perform lens shading correction on the second image based on the estimated illumination spectral data.

[0019] The processor may be configured to generate a correction map for each second channel by performing linear interpolation on a correction table corresponding to the estimated illumination spectrum data.

[0020] The processor may be configured to perform parallax correction on the first image and the second image.

[0021] The processor is configured to further perform white balance correction on the second image on which lens shading correction has been performed based on the estimated illumination spectrum data.

[0022] According to another aspect of the present disclosure, a method for acquiring an image includes: acquiring a first image of at least one object in an environment where at least one illumination source exists through eight or more first channels with minimal overlap between channels by using a multispectral image sensor; estimating illumination spectral data of the acquired first image by using channel signals corresponding to the eight or more first channels; and performing lens shading correction on the first image and a second image acquired by an image sensor different from the multispectral image sensor based on the estimated illumination spectral data.

[0023] The image sensor may be configured to acquire a second image of at least one object in an environment where at least one illumination source is present through less than four second channels, and performing lens shading correction may include generating a correction map for each first channel or each second channel by performing linear interpolation on a correction table corresponding to estimated illumination spectrum data, and performing lens shading correction on the first image or the second image by using the generated correction map.

[0024] Performing lens shading correction also includes: performing lens shading correction on all RAW images acquired from the multispectral image sensor or images according to the channels by using a correction map generated for each first channel, or performing lens shading correction on all RAW images acquired from the multispectral image sensor or images according to the channels by using a correction map generated for each second channel.

[0025] The correction table may include first correction data for each first channel, or second correction data for each second channel, and the first correction data and the second correction data may include at least one of a scaling factor, an offset, an incident angle, and a curve fitting coefficient corresponding to the estimated illumination spectrum data.

[0026] The method may further include performing parallax correction on the first image and the second image.

[0027] The method may further include performing white balance correction on the first image or the second image on which lens shading correction has been performed, based on the estimated illumination spectrum data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other aspects, features and advantages of some embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which:

[0029] Figure 1 is a cross-sectional view of a multispectral image sensor according to an embodiment;

[0030] Figure 2A is a block diagram showing the structure of an image acquisition device according to an embodiment;

[0031] Figure 2B is a block diagram showing the structure of an image acquisition device according to another embodiment;

[0032] Figure 3 is a graph of the wavelength spectrum of an RGB image sensor;

[0033] Figure 4A is a graph of a wavelength spectrum of a multispectral image sensor according to an embodiment;

[0034] Figure 4B is a graph of a wavelength spectrum of a multispectral image sensor according to an embodiment;

[0035] Figure 5 is a diagram of a pixel arrangement in a multispectral image sensor according to an embodiment;

[0036] Figure 6 is a diagram for describing a process of forming an image through channels of a multispectral image sensor according to an embodiment based on signals acquired from a plurality of channels;

[0037] Figure 7 is a diagram for comparing and describing images acquired for the same object in environments with different illumination sources according to an embodiment;

[0038] Figure 8 is a diagram for describing spectral data acquired based on intensities of signals acquired from a plurality of channels of a multispectral image sensor according to an embodiment;

[0039] Fig. 9 is a diagram for describing a lens shading correction method performed in the related art;

[0040] Fig.10 yes Figure 2A Detailed block diagram of the image acquisition device shown in ;

[0041] Fig.11 is a diagram for describing an interpolation method according to an embodiment;

[0042] Fig.12 is a diagram for describing a process of generating a correction map according to an embodiment;

[0043] Fig.13 is a flow chart of a method for performing lens shading correction on an image acquired from a multispectral image sensor according to an embodiment;

[0044] Fig.14 is a diagram for describing a method of performing white balance correction on an image acquired from a multispectral image sensor according to an embodiment;

[0045] Fig.15 yes Figure 2B Detailed block diagram of the image acquisition device shown in ;

[0046] Fig.16 is a flow chart of a method of performing lens shading correction on an image acquired from an image sensor according to an embodiment;

[0047] Fig.17 is a block diagram of a configuration of an electronic device according to an embodiment;

[0048] Fig.18 Is set in Fig.17 A block diagram of a camera module in an electronic device shown in ; and

[0049] Figures 19 to 28 : are diagrams showing various examples of electronic devices to which the image acquisition apparatus according to the exemplary embodiment is to be applied. DETAILED DESCRIPTION

[0050] Now will be referred to in detail embodiment, the example of embodiment is shown in the accompanying drawings, wherein similar reference numerals throughout the drawings represent similar elements. In this regard, the embodiment presented can have different forms and should not be construed as being limited to the description set forth herein. Accordingly, the embodiments are described below only by reference to the accompanying drawings to explain various aspects. The term "and / or" used herein includes any and all combinations of one or more of the associated listed items. Statements such as "at least one of ..." modify the entire element list when after the element list, rather than modifying the individual elements in the list.

[0051] The currently widely used general terms are selected in consideration of the functions of the structural elements in the various embodiments of the present disclosure. However, the meaning of the terms may change according to intention, judicial precedents, the emergence of new technologies, etc. In addition, in some cases, the terms may be arbitrarily selected according to the specific circumstances. In this case, the meaning of the terms will be described in detail in the corresponding part in the description of the present disclosure. Therefore, the terms used in the various embodiments of the present disclosure will be defined based on the meaning of the terms and the description provided herein.

[0052] In the description of the embodiments, when a part is connected to another part, the connection includes the case where the parts are electrically connected to each other with other components therebetween, and the direct connection between the parts. In addition, when a part is referred to as including components, this means that the part may also include other components, and does not exclude other components unless otherwise specifically described.

[0053] The terms “configured to” and “comprising” used in the embodiments should not be interpreted as necessarily including all components or steps described in the specification; some components or steps may not be included, or additional components or steps may be further included.

[0054] In addition, the terms "first", "second" may be used herein to describe various elements, but these elements will not be limited by these terms. These terms are only used to distinguish one component from another component.

[0055] The following description of the embodiment should not be interpreted as limiting the scope of the present disclosure, and those descriptions that are easily inferred by those skilled in the art should be interpreted as being within the scope of the embodiment. Hereinafter, embodiments that are merely examples will be described in detail with reference to the accompanying drawings.

[0056] Figure 1 is a cross-sectional view of a multispectral image sensor 100 according to an embodiment.

[0057] Figure 1 The multispectral image sensor 100 shown in FIG. 1 may include, for example, a complementary metal oxide semiconductor (CMOS) image sensor, or a charge coupled device (CCD) image sensor.

[0058] refer to Figure 1 The multispectral image sensor 100 may include a pixel array 65, and a spectral filter 83 disposed on the pixel array 65. Here, the pixel array 65 may include a plurality of pixels arranged two-dimensionally, and the spectral filter 83 may include a plurality of resonators disposed to correspond to the plurality of pixels. Figure 1 An example is shown in which the pixel array 65 includes four pixels and the spectral filter 83 includes four resonators.

[0059] Each pixel in the pixel array 65 may include a photodiode 62 as a photoelectric conversion element, and a driving circuit 52 configured to drive the photodiode 62. The photodiode 62 may be arranged to be buried in a semiconductor substrate 61. For example, a silicon substrate may be used as the semiconductor substrate 61. However, the present disclosure is not limited thereto. A wiring layer 51 may be provided at a bottom surface 61a of the semiconductor substrate 61, and in the wiring layer 51, for example, a driving circuit 52 such as a metal oxide semiconductor field effect transistor (MOSFET) may be provided.

[0060] The spectral filter 83 including a plurality of resonators is disposed on the top surface 61b of the semiconductor substrate 61. Each resonator can be configured to transmit light of a desired specific wavelength range. Each resonator can include reflective layers spaced apart from each other and a cavity disposed between the reflective layers. Each reflective layer can include a metal reflective layer or a Bragg reflective layer. Each cavity can be configured to resonate light of a desired specific wavelength range.

[0061] The spectral filter 83 may include one or more functional layers configured to increase the transmittance of light transmitted through the spectral filter 83 and incident on the photodiode 62. The functional layer may include a dielectric layer or a dielectric pattern having an adjusted refractive index. In addition, the functional layer may include, for example, an anti-reflection layer, a focusing lens, a color filter, a short wavelength absorption filter, a long wavelength blocking filter, etc. However, this is only an example.

[0062] Figure 2A is a block diagram showing the structure of the image acquisition device 10 according to the embodiment.

[0063] refer to Figure 2A According to the embodiment, the image acquisition device 10 may include a multispectral image sensor 100 and a processor 500. Figure 2A The image acquisition device 10 shown in FIG. 1 only shows components related to the present embodiment. Therefore, it is obvious to those skilled in the art that, except for Figure 2A In addition to the components shown in FIG. 1 , the image acquisition device 10 may further include other components.

[0064] The multispectral image sensor 100 may indicate a sensor configured to sense light of various wavelength bands. For example, the multispectral image sensor 100 may include sixteen channels or thirty-one channels. However, the present disclosure is not limited thereto, and the multispectral image sensor 100 may include any number of channels (e.g., four or more).

[0065] The multispectral image sensor 100 may be configured to adjust the central wavelength, bandwidth, and transmission of light absorbed by each channel so that each channel can sense light of a desired frequency band. The image acquired by the multispectral image sensor 100 may include a multispectral image or a hyperspectral image.

[0066] In an environment where there is at least one illumination source, the multispectral image sensor 100 may be configured to acquire an image of at least one object through a plurality of channels. For example, the multispectral image sensor 100 may be configured to acquire an image by dividing a relatively wide band including a visible light band, an infrared band, an ultraviolet band, etc. into a plurality of channels. The multispectral image sensor 100 may be configured to acquire an image using all available channels. Alternatively, the multispectral image sensor 100 may be configured to acquire an image by selecting certain channels.

[0067] The multi-spectral image sensor 100 may include a filter capable of transmitting light according to a frequency band in each channel. A multi-spectral filter array (MSFA) may indicate a filter corresponding to the entire frequency band sensed by the multi-spectral image sensor 100. The MSFA may have a one-dimensional or two-dimensional arrangement. For example, when the number of channels is sixteen, the MSFA may have a 4×4 arrangement.

[0068] The filter in each channel can be configured to transmit light of a specific frequency band. To this end, the filter can have a resonant structure. The transmission frequency band of the filter can be determined according to the resonant structure. For example, the transmission frequency band can be adjusted according to the material included in the reflective layer, the material included in the cavity, and the thickness of the cavity. The filter can be implemented by gratings, nanostructures, distributed Bragg reflectors (DBRs) or other methods.

[0069] The processor 500 may be configured to control the overall operation of the image acquisition device 10. The processor 500 may include a single core or multiple cores. The processor 500 may be configured to process or execute programs and / or data stored in the memory. For example, the processor 500 may be configured to control the functions of the image acquisition device 10 by executing a program stored in the memory.

[0070] The processor 500 according to the embodiment may be configured to acquire channel signals corresponding to four or more channels from the multispectral image sensor 100. The processor 500 may be configured to select at least some of the preset number of channels physically provided in the multispectral image sensor 100, and acquire channel signals from the selected channels. For example, the processor 500 may be configured to acquire channel signals from all channels of the preset number of channels physically provided in the multispectral image sensor 100. In addition, the processor 500 may also be configured to select only some of the preset number of channels physically provided in the multispectral image sensor 100, and acquire channel signals.

[0071] The processor 500 may be configured to obtain channel signals greater than or less than a preset number by synthesizing or interpolating channel signals obtained from a preset number of channels physically disposed in the multispectral image sensor 100. For example, the processor 500 may be configured to obtain channel signals of channels less than a preset number by performing merging on pixels or channels of the multispectral image sensor 100. In addition, the processor 500 may also be configured to obtain channel signals of channels greater than a preset number by generating new channel signals using interpolation of channel signals.

[0072] When the number of acquired channel signals decreases, each channel signal may correspond to a wide frequency band, and the signal may have increased sensitivity and reduced noise. On the contrary, when the number of acquired channel signals increases, the sensitivity of each channel signal may decrease, but a more accurate image may be acquired based on multiple channel signals. In this way, due to the trade-off between the increase and decrease in the number of acquired channel signals, the processor 500 may be configured to acquire an appropriate number of channel signals according to the application.

[0073] The processor 500 according to an embodiment may be configured to perform image processing before or after storing the image or signal acquired by the multispectral image sensor 100 in the memory. The image processing performed by the processor 500 may include bad pixel correction, fixed pattern noise correction, crosstalk reduction, re-mosaicing, demosaicing, false color reduction, denoising, chromatic aberration correction, etc. However, the present disclosure is not limited thereto, and the image processing performed by the processor 500 may also include lens shading correction.

[0074] The processor 500 according to an embodiment may be configured to acquire information related to the environment in which the multispectral image sensor 100 acquires an image. For example, the processor 500 may be configured to estimate information related to at least one illumination source that already exists in the environment in which the multispectral image sensor 100 acquires an image using channel signals corresponding to a plurality of channels of the multispectral image sensor 100.

[0075] The RGB image sensor used in the related art can be configured to sense the band within the visible light range through three channels and acquire channel signals corresponding to the three channels. However, considering that the illumination has a spectrum that changes continuously in the visible light band and is distributed over a wide range, it may be difficult to accurately estimate information related to at least one illumination source that already exists in the environment where the RGB image sensor acquires the image using only the information acquired from the three channels. In addition, in the case of an image of a scene including a simple background or a simple object, the data that can be sampled is further reduced, and the accuracy of estimating the illumination source may not be guaranteed.

[0076] The multispectral image sensor 100 may be configured to acquire channel signals corresponding to at least four channels. The image acquisition device 10 may be configured to more accurately estimate information related to at least one illumination source that already exists in the environment where the multispectral image sensor 100 acquires images based on the acquired four or more channel signals.

[0077] More specifically, when the multispectral image sensor 100 includes eight or more channels with minimal overlap between the channels, the image acquisition device 10 can be configured to estimate information related to all types of illumination sources available in the peripheral environment based on the eight or more channel signals that have been acquired. As the overlap between the channels of the multispectral image sensor 100 decreases, the accuracy of the information related to the illumination source estimated by the image acquisition device 10 can be improved.

[0078] In the present disclosure, minimizing the overlap between channels may indicate setting the channels of the multispectral image sensor 100 so that the bands sensed by the channels do not overlap with each other as much as possible. In addition, when the multispectral image sensor 100 includes a channel configured to sense the entire band, minimizing the overlap between channels may indicate setting other channels other than the channel so that the bands sensed by the other channels do not overlap with each other as much as possible.

[0079] The multispectral image sensor 100 may be configured to adjust the central wavelength and bandwidth of light absorbed by the channel so that the channel can sense light in a band that does not overlap with each other as much as possible. In an embodiment, the bands sensed by multiple channels of the multispectral image sensor 100 may not overlap with each other. The channels of the multispectral image sensor 100 may be configured to sense different bands divided from the entire band to be sensed by the multispectral image sensor 100. However, the present disclosure is not limited thereto. As another example, the area of ​​the band sensed by at least one channel of the multispectral image sensor 100 may overlap with the bands sensed by other channels. Even in this case, the band sensed by the at least one channel and the bands sensed by other channels may be set to not overlap with each other as much as possible. For example, the area of ​​the band sensed by the at least one channel may be 10% or less of the entire band area sensed by other channels. When the multispectral image sensor 100 includes multiple channels with minimal overlap between channels, the channels of the multispectral image sensor 100 may be configured to sense bands that do not overlap with each other as much as possible. Since the channels sense bands that do not overlap as much as possible, the multispectral image sensor 100 can be configured to acquire channel signals for bands that do not overlap with each other as much as possible. In addition, through multiple channels with minimal overlap between channels, the multispectral image sensor 100 can acquire channel signals for smaller bands. The channels of the multispectral image sensor 100 can be configured to sense small bands to avoid overlapping with each other as much as possible, and the multispectral image sensor 100 can be configured to acquire channel signals for bands that avoid overlapping with each other as much as possible. The information about the illumination source estimated by the image acquisition device 10 based on the acquired channel signals can be accurate.

[0080] More specifically, by including a multispectral image sensor 100 of eight or more channels with minimal overlap between the channels, the image acquisition device 10 can accurately estimate information related to all types of illumination sources available in the peripheral environment.

[0081] The processor 500 according to an embodiment may be configured to perform various types of corrections on the image acquired from the multispectral image sensor 100. For example, the processor 500 may be configured to perform color calibration, lens shading correction, white balance correction, etc. on the image acquired from the multispectral image sensor 100.

[0082] The processor 500 may be configured to perform the above-described various types of corrections on the acquired image using information related to at least one illumination source estimated by the multispectral image sensor 100. In the related art, lens shading correction or the like is performed on the acquired image based on color temperature. However, even illumination sources having the same color temperature may have different spectral characteristics. Therefore, an image on which lens shading correction is performed based only on color temperature may not be sufficient to accurately express the actual and unique colors of an object or background included in the image.

[0083] The multispectral image sensor 100 including eight or more channels with minimal overlap between the channels may be configured to acquire eight or more channel signals, and the processor 500 may be configured to accurately estimate information about at least one illumination source present in the environment in which the multispectral image sensor 100 acquires an image based on the signals of the eight or more channels with minimal overlap between the channels. The processor 500 may be configured to perform lens shading correction on the image acquired by the multispectral image sensor 100 based on the information about the at least one illumination source that has been accurately estimated. The image on which the lens shading correction has been performed may accurately express the actual and unique color of the object or background included in the image.

[0084] although Figure 2A Although not shown in the figure, the image acquisition device 10 may further include a memory. The memory may be configured to store images acquired from the multispectral image sensor 100. The memory may include a line memory configured to continuously store images in units of lines, and may also include a frame buffer configured to store the entire image. In addition, the memory may be configured to store applications and drivers to be driven by the image acquisition device 10.

[0085] The memory may include a random access memory (RAM) (e.g., a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a CD-ROM, a Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid-state drive (SSD), or a flash memory. However, the present disclosure is not limited thereto.

[0086] The memory may be configured to store information required to perform correction on images acquired from the multispectral image sensor 100. For example, the memory may be configured to store information required to perform lens shading correction on images captured in an environment where a specific illumination source is present.

[0087] The memory may be external to the multispectral image sensor 100, or may be integrated in the multispectral image sensor 100. When the memory is integrated in the multispectral image sensor 100, the memory may be integrated with the circuit (e.g., Figure 1The wiring layer 51 and / or the driving circuit 52 shown in FIG. 5 are integrated together. The pixel unit (for example, Figure 1 The semiconductor substrate 61 and / or the photodiode 62 shown in FIG. 1 and other parts (e.g., circuits and memories) may each correspond to a stack and may be integrated into two stacks. In this case, the multispectral image sensor 100 may include a chip having two stacks. However, the embodiment is not limited thereto, and the multispectral image sensor 100 may be implemented as three stacks including three layers of pixel units, circuits, and memories.

[0088] Figure 2B is a block diagram showing the structure of an image acquisition device 20 according to another embodiment.

[0089] refer to Figure 2B According to another embodiment, the image acquisition device 20 may include a multispectral image sensor 100, an image sensor 200, and a processor 500. Figure 2A Compared with the image acquisition device 10 shown in FIG. 1 , the image acquisition device 20 according to another embodiment may further include an image sensor 200. Figure 2B The multispectral image sensor 100 shown in FIG. Figure 2A The multispectral image sensor 100 shown in FIG. 1 is substantially the same as that in FIG. 1 , and thus the same description thereof will be omitted.

[0090] The multispectral image sensor 100 may be configured to acquire a first image of a first band, and the image sensor 200 may be configured to acquire a second image of a second band. The first band may include the second band and may be larger than the second band. However, the embodiment is not limited thereto, and the second band may include the first band and may be larger than the first band. In addition, the first band and the second band may be substantially the same as each other. The number of channels of the image sensor 200 may be less than the number of channels of the multispectral image sensor 100. For example, the image sensor 200 may include less than four channels to acquire an image of at least one object in an environment where at least one illumination source is present.

[0091] The image sensor 200 according to the embodiment may include an RGB image sensor. The RGB image sensor may include an R channel, a G channel, and a B channel. The image sensor 200 is a sensor used in a general RGB camera and may include a CMOS image sensor using a Bayer filter array. The second image acquired by the image sensor 200 may include an RGB image based on red, green, and blue.

[0092] The image sensor 200 according to another embodiment may include a monochrome image sensor. The monochrome image sensor includes a color channel. The second image acquired by the image sensor 200 may include a monochrome image.

[0093] Compared to the image sensor 200, the multispectral image sensor 100 may include a sensor configured to sense more different types of wavelengths of light. Each channel of the multispectral image sensor 100 may be configured to adjust the wavelength of the transmitted light, the amount of transmission, and the bandwidth of the light so as to sense light in a desired frequency band. The bandwidth of each channel of the multispectral image sensor 100 may be set to be smaller than the bandwidths of the R band, the G band, and the B band. The multispectral image sensor 100 may adjust the bandwidth of each channel to avoid overlap as much as possible.

[0094] Figure 3 is a graph of the wavelength spectrum of an RGB image sensor.

[0095] refer to Figure 3 , image sensors (e.g., Figure 2B The image sensor 200 shown in FIG. 2 may include an R channel, a G channel, and a B channel, and may be configured to sense light of a wavelength band corresponding to each of the aforementioned three channels.

[0096] Figure 4A is a graph of a wavelength spectrum of the multispectral image sensor 100 according to an embodiment, and Figure 4B is a diagram of a wavelength spectrum of a multispectral image sensor 100 according to another embodiment.

[0097] refer to Figure 4A and Figure 4B , the multispectral image sensor 100 may include sixteen channels or thirty-one channels. However, the present disclosure is not limited thereto, and the multispectral image sensor 100 may include any number of channels (eg, four or more).

[0098] The multispectral image sensor 100 may be configured to sense light of wavelength bands corresponding to a plurality of channels. The wavelength bands corresponding to the plurality of channels may be determined by dividing the entire frequency band sensed by the multispectral image sensor 100 at uniform intervals. However, the present disclosure is not limited thereto, and the wavelength bands of some of the plurality of channels may be larger or smaller than the wavelength bands of other channels in the plurality of channels. In addition, the wavelength bands of some of the plurality of channels may include the entire frequency band sensed by the multispectral image sensor 100.

[0099] The multispectral image sensor 100 may include a pixel array, and will be referred to below. Figure 5 Describes an array of pixels.

[0100] Figure 5 is a diagram of a pixel arrangement in a multispectral image sensor according to an embodiment.

[0101] refer to Figure 5, the spectral filter 120 provided in the pixel array PA1 may include a plurality of filter groups 121 arranged in two dimensions. Here, each of the plurality of filter groups 121 may include sixteen unit filters F1 to F16 arranged in the form of a 4×4 array. However, the present disclosure is not limited thereto. For example, each filter group 121 may include nine unit filters F1 to F9 arranged in the form of a 3×3 array, twenty-five unit filters F1 to F25 arranged in the form of a 5×5 array, or M×N unit filters arranged in the form of an M×N array (wherein each of M and N represents an arbitrary integer of 1 or greater).

[0102] For example, when each of the plurality of filter groups 121 is arranged in the form of a 4×4 array, the first unit filter F1 and the second unit filter F2 may have central wavelengths UV1 and UV2 in the ultraviolet range, and the third unit filter F3, the fourth unit filter F4, and the fifth unit filter F5 may have central wavelengths B1, B2, and B3 in the blue light range. The sixth unit filter F6, the seventh unit filter F7, the eighth unit filter F8, the ninth unit filter F9, the tenth unit filter F10, and the eleventh unit filter F11 may have central wavelengths G1 to G6 in the green light range, and the twelfth unit filter F12, the thirteenth unit filter F13, and the fourteenth unit filter F14 may have central wavelengths R1 to R3 in the red light range. The fifteenth unit filter F15 and the sixteenth unit filter F16 may have central wavelengths NIR1 and NIR2 in the near infrared range.

[0103] For another example, when each of the plurality of filter groups 121 is arranged in the form of a 3×3 array, the first unit filter F1 and the second unit filter F2 may have central wavelengths UV1 and UV2 in the ultraviolet range, and the fourth unit filter F4, the fifth unit filter F5, and the seventh unit filter F7 may have central wavelengths B1 to B3 in the blue light range. The third unit filter F3 and the sixth unit filter F6 may have central wavelengths G1 and G2 in the green light range, and the eighth unit filter F8 and the ninth unit filter F9 may have central wavelengths R1 and R2 in the red light range.

[0104] For another example, when each of the plurality of filter groups 121 is arranged in the form of a 5×5 array, the first unit filter F1, the second unit filter F2, and the third unit filter F3 may have central wavelengths UV1 to UV3 in the ultraviolet range, and the sixth unit filter F6, the seventh unit filter F7, the eighth unit filter F8, the eleventh unit filter F11, and the twelfth unit filter F12 may have central wavelengths B1 to B5 in the blue light range. The fourth unit filter F4, the fifth unit filter F5, and the ninth unit filter F9 may have central wavelengths G1 to G3 in the green light range, and the tenth unit filter F10, the thirteenth unit filter F13, the fourteenth unit filter F14, the fifteenth unit filter F15, the eighteenth unit filter F18, and the nineteenth unit filter F19 may have central wavelengths R1 to R6 in the red light range. The twentieth unit filter F20, the twenty-third unit filter F23, the twenty-fourth unit filter F24, and the twenty-fifth unit filter F25 may have central wavelengths NIR1 to NIR4 in the near infrared range.

[0105] The aforementioned unit filter provided in the spectrum filter 120 may have a resonant structure including two reflectors, and the transmission band may be determined according to the characteristics of the resonant structure. The transmission band may be adjusted according to the material of the reflector, the material of the dielectric material in the cavity, and the thickness of the cavity. In addition, a structure using a grating, a structure using a DBR, etc. may be applied to the unit filter.

[0106] The pixels in the pixel array PA1 may also be arranged in various methods according to the color characteristics of the multispectral image sensor 100 .

[0107] Figure 6 is a diagram for describing a process of forming an image through channels of the multispectral image sensor 100 according to an embodiment based on signals acquired from a plurality of channels.

[0108] refer to Figure 6 , shows a RAW (raw image format) image 610 acquired from the multispectral image sensor 100 and an image 620 for each channel after performing demosaicing. In the RAW image 610, small squares indicate pixels, and numbers in the squares indicate channel numbers. According to the channel numbers, the RAW image 610 may include an image acquired by a multispectral sensor including sixteen channels. Although the RAW image 610 includes all pixels corresponding to different channels, since pixels in the same channel are collected by demosaicing, an image 620 for each channel can be generated.

[0109] Figure 7is a diagram for comparing and describing images acquired for the same object in environments with different illumination sources according to an embodiment.

[0110] refer to Figure 7 , showing a multispectral image sensor 100 (eg, Figure 2A or Figure 2B The present invention relates to a process in which a multispectral image sensor 100 (shown in FIG. 1 ) acquires an image of an object 710 in an environment where different illumination sources (e.g., a first illumination source 720 and a second illumination source 730) are present. The first image 721 may correspond to an image acquired by capturing the object 710 in an environment where the first illumination source 720 is present, and the second image 731 may correspond to an image acquired by capturing the object 710 in an environment where the second illumination source 730 is present.

[0111] The color temperature, which uses absolute temperature to numerically express the color of an illumination source, is related to Planck's law and Wien's displacement law. The color of an illumination source from a black body appears different depending on the temperature, and as the wavelength of the illumination source that is emitted to the maximum extent at a specific temperature is longer, the color temperature is lower. Since the color temperature standardizes and specifies the color of an illumination source, illumination sources can be distinguished from each other using the color temperature. However, there may be different illumination sources that have different spectral characteristics even at the same color temperature. For example, some illumination sources with the same color temperature may have different spectral sensitivities.

[0112] Image acquisition device (e.g. Figure 2A The image acquisition device 10 and Figure 2B The image acquisition device 20 in the multi-spectral sensor may be configured to perform lens shading correction based on the illumination spectrum data. In the present disclosure, the expression "illumination spectrum data" may represent information indicating an environment in which an illumination source exists. For example, the illumination spectrum data may include information related to the intensity according to the wavelength of the channel signal acquired from the multi-spectral sensor. However, the present disclosure is not limited thereto, and any information for distinguishing environments in which different illumination sources exist may be referred to as illumination spectrum data.

[0113] In the present disclosure, lens shading correction may refer to performing correction to remove both vignetting and color shading.

[0114] Since the lens of the image acquisition device has a circular shape and the multispectral image sensor has a planar shape, the shape of the lens and the shape of the multispectral image sensor may be curved to different degrees. The multispectral image sensor is configured to acquire an image by sensing light received through the lens, and therefore, the image acquired by the multispectral image sensor may become darker toward the edge of the image. This is called vignetting.

[0115] Color changes may occur due to differences between the transmittance of the IR cut filter for illumination sources passing through the optical axis of the lens and illumination sources passing through the periphery of the optical axis. This is known as color shading.

[0116] In the related art, the color temperature of the illumination source is estimated from the acquired image, and lens shading correction is performed on the acquired image based on the estimated color temperature. Even illumination sources having the same color temperature may have different spectral characteristics. However, an image that does not take this factor into consideration and for which lens shading correction is performed only based on the color temperature of the illumination source may not accurately express the actual and unique color of an object or background included in the image.

[0117] For example, the first illumination source 720 and the second illumination source 730 may include illumination sources having the same color temperature and different spectral characteristics. When lens shading correction is performed based on the color temperature of the first illumination source 720 and the second illumination source 730, the color of the first image 721 and the color of the second image 731 may be achieved differently.

[0118] Therefore, the image acquisition device may be configured to estimate accurate illumination spectrum data from the acquired image, and perform lens shading correction based on the estimated illumination spectrum data, thereby achieving the same color for the color of the first image 721 and the color of the second image 731. Hereinafter, a method of estimating accurate illumination spectrum data from an image acquired by the image acquisition device will be described.

[0119] Figure 8 is a diagram for describing spectral data based on the intensity of signals acquired from a plurality of channels of a multi-spectral image sensor according to an embodiment. Figure 8 , shows an MSFA pattern 820 corresponding to a wavelength spectrum 810 for a particular illumination source, which is obtained by an image acquisition device (e.g., Figure 2A and Figure 2B The image acquisition device 10 and the image acquisition device 20 shown in are estimated.

[0120] The image acquisition device according to the embodiment may be configured to use a multispectral image sensor (e.g., Figure 2A Sixteen channel signals corresponding to sixteen or more channels of the multispectral image sensor 100 shown in FIG. 1 are used to estimate the illumination spectrum data of the image acquired by the multispectral image sensor. The number of channel signals corresponding to the number of channels of the multispectral image sensor is not limited to sixteen, and those skilled in the art can use a sufficient number of channel signals to estimate the illumination spectrum data of the image. For example, a multispectral image sensor having eight or more channels with minimal overlap between channels can be configured to estimate all types of illumination spectrum data available in the peripheral environment based on the signals of the eight channels corresponding to the eight or more channels.

[0121] The image acquisition device may acquire intensities of sixteen channel signals corresponding to sixteen or more channels of the multispectral image sensor for sixteen wavelength bands w1 to w16 divided from the wavelength spectrum 810 of the specific illumination source.

[0122] The image acquisition device may be configured to use the intensity of the acquired channel signal to generate the MSFA pattern 820. The MSFA pattern 820 may include sixteen unit color filters C1 to C16 arranged in a 4×4 array. The first unit color filter C1 to the sixteenth unit color filter C16 may be configured to sense the intensity of the channel signal for the first band w1 to the intensity of the channel signal for the sixteenth band w16.

[0123] Thus, the image acquisition device may be configured to estimate illumination spectral data of the acquired image using spectral data corresponding to the image acquired for at least one object in an environment in which at least one illumination source is present.

[0124] The image acquisition apparatus according to the embodiment may be configured to estimate the illumination spectrum data from a captured image, which is a mixture of the illumination spectrum data and the reflection spectrum data of the object, using a linear model.

[0125] [Equation 1]

[0126] D mxn =L mxn ×R mxn =L mxn ×B mxs ×C sxn .

[0127] In equation 1, D mxn is a matrix indicating the image acquired from the multispectral image sensor, L mxn is a diagonal matrix indicating the illumination sources present when the image was acquired, and R mxn is a matrix indicating the reflectance characteristics of an object. m indicates the number of channels included in the multispectral image sensor, and n indicates the number of pixels in an image acquired from the multispectral image sensor.

[0128] B mxn is a matrix indicating basis vectors associated with the reflective properties of the object, and C mxn is a matrix indicating the coefficients of basis vectors related to the reflection characteristics of the object. The reflection characteristics of the object can be expressed by a combination of eight or more basis vectors. s indicates the dimension of the local space.

[0129] The image acquisition device may be configured to calculate basis vectors related to the reflectance characteristics of the object based on Equation 1 using singular value decomposition, and in this way, the image acquisition device can estimate the illumination spectrum data. By calculating eight or more basis vectors related to the reflectance characteristics of the object, the image acquisition device can estimate all types of illumination spectrum data available in the surrounding environment.

[0130] According to another embodiment, the image acquisition device may be configured to estimate the illumination spectrum data using a neural network.The image acquisition device may be configured to estimate the illumination spectrum data using a pre-trained machine learning model.

[0131] The pre-trained machine learning model for illumination spectrum data may include a machine learning model that has been trained to receive channel signals corresponding to multiple channels of a multispectral image sensor and output illumination spectrum data. However, the machine learning model is not limited thereto, and it may also include a machine learning model that has been trained to simultaneously receive channel signals corresponding to multiple channels of a multispectral image sensor and an image sensor (e.g., Figure 2B A machine learning model that generates channel signals corresponding to less than four channels of the image sensor 200 shown in FIG. 2 and outputs illumination spectrum data.

[0132] The image acquisition device may be configured to estimate the illumination spectrum data using a swallow network. The image acquisition device may be configured to generate a plurality of images having a thumbnail size (640×480) from a RAW image, and estimate the illumination spectrum data for each of the plurality of images that have been generated. The image acquisition device may be configured to perform white balance correction on each of the plurality of images using a white balance coefficient corresponding to different illumination spectrum data. For example, the image acquisition device may be configured to estimate the illumination spectrum data by giving a high expected value to an image having the best white balance among the plurality of images for which white balance correction has been performed.

[0133] However, spectral data that can be used to estimate an environment in which at least one illumination source exists is merely an example of illumination spectral data, and illumination spectral data according to another example may include an index indicating a color vector of an illumination source in a color space, a color temperature of the illumination source, or a predetermined illumination type corresponding to the illumination source.

[0134] The image acquisition device according to the embodiment can divide the image acquired from the multi-spectral image sensor into multiple regions, and can estimate the illumination spectrum data for each of the multiple regions. However, the present disclosure is not limited thereto, and the image acquisition device can be configured to directly estimate the illumination spectrum data for each pixel of the image that has been acquired.

[0135] According to an example, each of the multiple regions can be divided based on the distance from the image acquisition device to the object (or the depth of the object in the frame of the image that has been acquired). However, the present disclosure is not limited to this. According to another example, object detection and / or image segmentation can be used to divide each of the multiple regions. In addition, each of the multiple regions that have been divided can be divided into the same size and the same shape. For example, each of the multiple regions that have been divided can be divided into square units with the same size and the same shape. However, the present disclosure is not limited to this. For another example, each of the multiple regions that have been divided can be divided into polygonal block units with uneven sizes and uneven shapes.

[0136] The image acquisition device according to the embodiment may be configured to perform preprocessing before estimating the illumination spectrum data. For example, the image processing performed by the image acquisition device before estimating the illumination spectrum data may include bad pixel correction, fixed pattern noise correction, crosstalk reduction, re-mosaicing, demosaicing, false color reduction, denoising, chromatic aberration correction, etc. For another example, the image acquisition device may also be configured to perform lens shading correction before estimating the illumination spectrum data. In this case, the lens shading correction may be performed without using the illumination spectrum data, or the lens shading correction may be performed using the illumination spectrum data estimated based on the previously acquired image. However, the image processing performed by the image acquisition device is not limited to the aforementioned examples. The above-mentioned image processing may also be performed after the image acquisition device estimates the illumination spectrum data.

[0137] Fig. 9 is a diagram for describing a lens shading correction method performed in the related art. Fig. 9 , a brightness curve graph 900 of an image shows a first brightness curve 910 of an image before performing lens shading correction, a second brightness curve 920 of the image after performing lens shading correction, and a correction curve 930 opposite to the first brightness curve 910 according to an embodiment. The x-axis of the brightness curve graph 900 indicates wavelength, and the y-axis indicates standardized brightness, so that the maximum brightness of the first brightness curve 910 is 1.

[0138] [Equation 2]

[0139] G=1-a(pc) 2

[0140] [Equation 3]

[0141] G=1+a(pc) 2

[0142] In Equation 2 and Equation 3, G indicates brightness, p indicates a pixel position, c indicates a position of a central pixel, and a indicates a coefficient related to lens shading correction. The first brightness curve 910 may be approximated in the form of a curve expressed by Equation 2, and the correction curve 930 may be approximated in the form of a curve expressed by Equation 3.

[0143] The lens shading correction may be performed on the image by straightening the first brightness curve 910 in the form of the second brightness curve 920 using the correction curve 930. However, in the above-mentioned lens shading correction, the illumination source existing in the environment in which the image is captured may not be considered. Hereinafter, a method of performing lens shading correction on the image in consideration of the environment in which the illumination source exists will be described.

[0144] Fig.10 yes Figure 2A A detailed block diagram of the image acquisition device 10 is shown in FIG.

[0145] refer to Fig.10 In addition to the multispectral image sensor 100 and the processor 500, the image acquisition device 10 may further include a memory 300. In addition, the processor 500 may further include an illumination estimator 510 and a corrector 520. Although the illumination estimator 510 and the corrector 520 are distinguished from each other according to the operation of the processor 500 for the sake of convenience of explanation, this distinction does not mean that the aforementioned units must be physically separated from each other. The aforementioned units may correspond to any combination of hardware and / or software included in the processor 500, and may be physically the same as or different from each other.

[0146] The multispectral image sensor 100 may include a sensor that has been color-calibrated for one or more illumination sources. Through the color space conversion matrix, the channel signals corresponding to the channels of the multispectral image sensor 100 may be converted into XYZ signals in the XYZ color space. The XYZ signals in the XYZ color space may be independent of the spectral sensitivity of the multispectral image sensor 100 and are affected by the spectral sensitivity recognized by the human eye.

[0147] The processor 500 may be configured to perform lens shading correction on the acquired image based on the estimated illumination spectrum data. In addition, the processor 500 may be configured to perform white balance correction on the image on which lens shading correction has been performed based on the estimated illumination spectrum data. Fig.14 A method of performing white balance correction will be described.

[0148] The correction table corresponding to the illumination spectrum data may be stored in the memory 300. The correction table may include correction data for each of the multiple channels of the multi-spectral image sensor 100. The image acquired from the multi-spectral image sensor 100 may include multiple segmented regions, and the correction table may include correction data corresponding to the corners of the grid for each of the multiple segmented regions. The grid may segment each of the multiple regions into uniform or non-uniform regions.

[0149] Correction data for performing lens shading correction using scaling factors and / or offsets corresponding to the estimated illumination spectrum data may be stored in the memory 300 .

[0150] [Equation 4]

[0151] E out =s i E in

[0152] [Formula 5]

[0153] E out =s i E in +o i

[0154] In Equation 4 and Equation 5, E in indicates input data related to an image acquired from the multispectral image sensor 100, and E out indicates output data related to an image for which lens shading correction has been performed. The subscript i indicates an index representing estimated illumination spectrum data, and s i indicates a normalization factor corresponding to the estimated illumination spectrum data, and o i Indicates the deviation corresponding to the estimated illumination spectrum data.

[0155] Input data (E) associated with an image acquired from the multispectral image sensor 100 in ) may indicate the strength of channel signals corresponding to the multiple channels of the multi-spectral image sensor 100. The processor 500 may be configured to convert the input data (E in ) is input to Equation 4 and / or Equation 5 to perform lens shading correction on the acquired image, and outputs output data (E out ).

[0156] Correction data for performing lens shading correction using an incident angle corresponding to the estimated illumination spectrum data may be further stored in the memory 300 .

[0157] [Equation 6]

[0158] Eout =s i E in cos 4 θ i

[0159] In Equation 6, θ i Indicates the incident angle of the illumination source corresponding to the estimated illumination spectrum data.

[0160] Input data (E) associated with an image acquired from the multispectral image sensor 100 in )) may indicate the intensity of a channel signal corresponding to each channel of the multi-spectral image sensor 100, and the processor 500 may be configured to convert the input data (E in )) is input to Equation 6 and outputs output data (E) associated with the image for which lens shading correction has been performed out )) to perform lens shading correction on the acquired image.

[0161] In addition to correction data for performing lens shading correction using scaling factors and / or offsets corresponding to estimated illumination spectrum data, the memory 300 according to another embodiment may also store correction data for performing lens shading correction using curve fitting coefficients corresponding to estimated illumination spectrum data.

[0162] However, the correction data stored in the memory 300 is not limited to the aforementioned embodiment. A memory according to another embodiment may store correction data including a suitable combination of scaling factors, offsets, incident angles, and / or curve fitting coefficients corresponding to the estimated illumination spectrum data.

[0163] The processor 500 may be configured to receive a correction table corresponding to the estimated illumination spectrum data from the memory 300, and may also be configured to perform lens shading correction on the image acquired from the multi-spectral image sensor 100 using the received correction table. Due to the limitation of the storage space of the memory 300, the memory 300 may not be able to store the correction table corresponding to all illumination spectrum data. Fig.11 A method of acquiring a correction table not previously stored in the memory 300 is described.

[0164] Fig.11 is a diagram for describing an interpolation method according to an embodiment, and Fig.12 is a diagram for describing a method of generating a correction map according to an embodiment.

[0165] Processor (e.g. Figure 2A , Figure 2B or Fig.10The processor 500 shown in FIG. 5 may be configured to generate a correction map by performing linear interpolation on a correction table corresponding to the estimated illumination spectrum data. The processor may be configured to generate a correction map relative to a multispectral image sensor (e.g., Figure 2A , Figure 2B or Fig.10 Lens shading correction is performed on each channel of the multispectral image sensor 100 shown in FIG.

[0166] refer to Fig.11 , showing a region 1100 segmented in blocks from an image acquired from a multispectral image sensor. Figure 2A , Figure 2B and Fig.10 The memory 300 shown in the figure may store only correction data corresponding to the corners (corners c1, c2, c3, c4, etc.) of the grid that evenly divides the area 1100, but the processor may be configured to obtain correction data corresponding to any coordinate (e.g., coordinate p) among the corners of the grid (i.e., angles c, c2, c3, c4, etc.) by performing linear interpolation on the limited correction data stored in the memory.

[0167] [Equation 7]

[0168] cx1=c1+(c2-c1)×a>>n

[0169] cx2=c3+(c4-c3)×a>>n

[0170] p=c1+(cx2-cx1)×b>>m

[0171] In Equation 7, p indicates an arbitrary coordinate among the corners of the grid, cx1 indicates an arbitrary coordinate between the first corner c1 and the second corner c2 where p coincides with the x-axis coordinate, cx2 indicates an arbitrary coordinate between the third corner c3 and the fourth corner c4 where p coincides with the x-axis coordinate, and n and m indicate integers for reducing computational complexity. Based on Equation 7, the processor 500 may be configured to acquire correction data corresponding to the arbitrary coordinate p.

[0172] The processor 500 may be configured to generate a correction map for each channel of the multi-spectral image sensor 100 by performing linear interpolation on a correction table corresponding to the acquired illumination spectrum data.

[0173] Fig.12 It shows the use of Fig.11 The correction map generated in the embodiment shown in FIG. 1 is an image of a case where lens shading correction is performed on an acquired image.

[0174] refer to Fig.12, the first image 1210 is an image before lens shading correction is performed, and the second image 1230 is an image after lens shading correction is performed. By performing lens shading correction on the first image 1210 using the correction map 1220, the image acquisition device (e.g., Figure 2A , Figure 2B and Fig.10 The image acquisition device 10 or 20 shown in can acquire a second image 1230.

[0175] Fig.12 The correction diagram 1220 shown in FIG. 1 may include Fig.11 The correction map 1220 may include a first correction map M1, a second correction map M2 to an nth correction map Mn for n channels of the multispectral image sensor.

[0176] The processor may be configured to use the generated correction map 1220 to correct all RAW images (eg, Figure 6 ) or images according to channels (e.g., Figure 6 Lens shading correction is performed based on the image 620 of the channel as shown in .

[0177] Fig.13 is a flowchart of a method of performing lens shading correction on an image acquired from the multispectral image sensor 100 according to an embodiment.

[0178] Fig.13 The method shown in can utilize an image acquisition device (e.g., Figure 2A , Figure 2B and Fig.10 The image acquisition device 10 or 20 shown in FIG. 1 is executed, and the above reference Figures 1 to 12 The description can be applied to Fig.13 Even if the description is omitted below, the method shown in FIG.

[0179] In operation 1310, the image acquisition device may use a multispectral image sensor (e.g., Fig.10 The multispectral image sensor 100 shown in FIG. 1 acquires a first image for at least one object through eight or more first channels with minimal overlap between channels in an environment where at least one illumination source exists. The multispectral image sensor may include a sensor that has previously performed color calibration.

[0180] In operation 1320, the image acquisition device may be configured to estimate illumination spectrum data of the acquired first image using channel signals corresponding to eight or more first channels. The illumination spectrum data may include information related to the intensity of the channel signals acquired from the multi-spectral image sensor according to the wavelength. For example, the image acquisition device may be configured to generate an MSFA pattern corresponding to the estimated illumination spectrum data (e.g., Figure 8 MSFA pattern 820 shown in FIG.

[0181] In operation 1330, the image acquisition device may be configured to perform lens shading correction on the first image based on the estimated illumination spectrum data. The image acquisition device may be configured to generate a correction map for eight or more first channels of the multispectral image sensor by performing linear interpolation on a correction table corresponding to the estimated illumination spectrum data. The image acquisition device may be configured to perform lens shading correction on all RAW images acquired from the multispectral image sensor or images according to channels using the correction map generated for each first channel.

[0182] The correction table may include first correction data for each first channel, and the first correction data may include a scaling factor, an offset, an incident angle, and / or a curve fitting coefficient corresponding to the estimated illumination spectrum data.

[0183] although Fig.13 , but the image acquisition device may be configured to further perform white balance correction on the first image on which lens shading correction has been performed. Fig.14 A method of performing white balance correction will be described.

[0184] Fig.14 is a diagram for describing a method of performing white balance correction on an image acquired from a multispectral image sensor according to an embodiment.

[0185] Since the multispectral image sensor is configured to sense light passing through a channel, the color of the image may change according to at least one illumination source present in the environment in which the multispectral image sensor acquires the image. That is, the color of an image about a piece of white paper acquired by the multispectral image sensor in an environment in which at least one illumination source exists may not be realized as white. White balance correction indicates a correction method for preventing changes in the color of the image due to at least one illumination source and realizing the inherent color of the object as much as possible in the acquired image.

[0186] Performing white balance correction on an image acquired by an RGB image sensor may indicate determining a gain for each of a pixel corresponding to a red band and a pixel corresponding to a blue band with reference to a pixel corresponding to a green band. A signal-to-noise ratio (SNR) of the RGB image sensor with respect to a green band may significantly affect the quality of an image acquired by the RGB image sensor. Therefore, the image acquisition device may set the gain of a pixel corresponding to a green band to 1, and set the gain of a pixel corresponding to a band other than the green band to less than 1, and by doing so, the image acquisition device may correct the image acquired from the RGB image sensor so that it is implemented as similarly as possible to the unique color of the image.

[0187] The aforementioned method of performing white balance correction on an image acquired by an RGB image sensor may be partially used in a method of performing white balance correction on an image acquired by a multispectral image sensor.

[0188] refer to Fig.14 , an image acquisition device (e.g., Figure 2A , Figure 2B and Fig.10 The image acquisition device 10 or 20 shown in FIG. 1 may be configured to estimate illumination spectrum data based on the acquired image and perform white balance correction based on the estimated illumination spectrum data. The image acquisition device may perform white balance correction using a white balance coefficient according to pixels, and may determine the white balance coefficient based on the MSFA pattern 1420 corresponding thereto.

[0189] Fig.14 The MSFA pattern 1420 shown in FIG. Figure 8 For example, the eighth unit color filter C8 of the MSFA pattern 1420 may be configured to sense the intensity of a channel signal corresponding to a channel of the multi-spectral image sensor for a green band of 550 nm.

[0190] The white balance coefficient according to the pixel may indicate the gain corresponding to the unit filters C1 to C16. The image acquisition device may be configured to set the value of the eighth gain gain8 corresponding to the eighth unit filter C8 to 1, give a gain greater than the eighth gain gain8 to the unit filter whose channel signal intensity is less than the intensity of the channel signal of the eighth unit filter C8, and give a gain less than the eighth gain gain8 to the unit filter whose channel signal intensity is greater than the intensity of the channel signal of the eighth unit filter C8. However, the white balance coefficient according to the pixel is not limited thereto. As another example, the sixteenth unit filter C16 may bypass the white balance correction, and in this case, the channel corresponding to the sixteenth unit filter C16 may include a channel configured to sense the entire band from 400nm to 700nm.

[0191] The image acquisition device may be configured to estimate illumination spectrum data from an image acquired by a multispectral image sensor, and determine a white balance coefficient according to the estimated illumination spectrum data. The image acquisition device may use the determined white balance coefficient according to the pixel to perform white balance correction on the image for which lens shading correction has been performed.

[0192] Fig.15 yes Figure 2B A detailed block diagram of the image acquisition device 20 is shown in FIG.

[0193] refer to Fig.15 ,and Fig.10 Compared with the image acquisition device 10 shown in Fig.15 The image acquisition device 20 shown in FIG. 1 may also include an image sensor 200. In the following, the multispectral image sensor 100, the processor 500 and the memory 300 will be omitted. Fig.10 The description of the image acquisition device 10 shown in FIG. 1 is the same as the description of FIG.

[0194] The multispectral image sensor 100 may be configured to acquire a first image for an object through multiple first channels in an environment where a specific illumination source exists, and the image sensor 200 may be configured to acquire a second image for an object through fewer than four second channels in an environment where a specific illumination source exists.

[0195] The multispectral image sensor 100 may include a sensor that has been color-calibrated for one or more illumination sources. Through the color space conversion matrix, the channel signals corresponding to the channels of the multispectral image sensor 100 may be converted into XYZ signals in the XYZ color space. The XYZ signals in the XYZ color space may be independent of the spectral sensitivity of the multispectral image sensor 100 and are affected by the spectral sensitivity recognized by the human eye.

[0196] The processor 500 may be configured to estimate the illumination spectrum data of the first image using the channel signal corresponding to the first channel. The processor 500 may be configured to perform lens shading correction on the second image based on the estimated illumination spectrum data. In addition, the processor 500 may be configured to perform white balance correction on the image on which lens shading correction has been performed based on the estimated illumination spectrum data. The processor 500 may be configured to perform parallax correction on the first image and the second image. For example, the processor 500 may be configured to align the first image and the second image, or to perform an affine transformation on the first image and the second image.

[0197] The correction table corresponding to the illumination spectrum data may be stored in the memory 300. The correction table may include correction data for each of the multiple channels of the multi-spectral image sensor 100. The image acquired from the multi-spectral image sensor 100 may include multiple segmented regions, and the correction table may include correction data corresponding to the corners of the grid for each of the multiple segmented regions. The grid may segment each of the multiple regions into uniform or non-uniform regions.

[0198] Correction data for performing lens shading correction using a scaling factor and / or an offset corresponding to the estimated illumination spectrum data may be stored in the memory 300. The input data associated with the image acquired from the multispectral image sensor 100 may indicate the intensity of a channel signal corresponding to each of a plurality of channels of the multispectral image sensor 100. The processor 500 may be configured to perform lens shading correction on the acquired image by inputting the input data to Equation 4 and / or Equation 5, and output output data associated with the image on which lens shading correction has been performed.

[0199] Correction data for performing lens shading correction using an incident angle corresponding to the estimated illumination spectrum data may be further stored in the memory 300. The processor 500 may be configured to perform lens shading correction on the acquired image by inputting the input data into Equation 6, and outputting output data related to the image on which lens shading correction has been performed.

[0200] In addition to correction data for performing lens shading correction using scaling factors and / or offsets corresponding to estimated illumination spectrum data, the memory 300 according to another embodiment may also store correction data for performing lens shading correction using curve fitting coefficients corresponding to estimated illumination spectrum data.

[0201] However, the correction data stored in the memory 300 is not limited to the aforementioned embodiment. A memory according to another embodiment may store correction data including a suitable combination of scaling factors, offsets, incident angles, and / or curve fitting coefficients corresponding to the estimated illumination spectrum data.

[0202] The processor 500 may be configured to receive a correction table corresponding to the estimated illumination spectrum data from the memory 300 , and may also be configured to perform lens shading correction on the image acquired from the multispectral image sensor 100 using the received correction table.

[0203] The processor 500 may be configured to generate a correction map for each channel of the multispectral image sensor 100 by performing linear interpolation on the correction table corresponding to the estimated illumination spectrum data. The processor 500 may be configured to use the generated correction map 1220 to correct all RAW images (e.g., Figure 6 ) or images according to channels (e.g., Figure 6 Lens shading correction is performed based on the image 620 of the channel as shown in .

[0204] The processor 500 may be configured to estimate illumination spectrum data from an image acquired by the multispectral image sensor 100, and determine a white balance coefficient according to a pixel based on the estimated illumination spectrum data. The processor 500 may perform white balance correction on an image on which lens shading correction has been performed using the determined white balance coefficient according to a pixel.

[0205] Fig.16 is a flowchart of a method of performing lens shading correction on an image acquired from an image sensor according to an embodiment.

[0206] Fig.16 The method shown in can be performed by an image acquisition device (e.g., Fig.15 The image acquisition device 20 shown in FIG. 1 is used to perform the above reference. Figures 1 to 15 The description can be applied to Fig.16 Even if the description is omitted below, the method shown in FIG.

[0207] In operation 1610, a processor (e.g., Fig.15 The processor 500 shown in FIG. 5 may use a multispectral image sensor (e.g., Fig.15 The multispectral image sensor 100 shown in FIG. 1 acquires a first image of at least one object in an environment where at least one illumination source exists through eight or more first channels with minimal overlap between the channels. The multispectral image sensor may include a sensor that has previously performed color calibration.

[0208] Alternatively, the processor may use an image sensor (e.g., Fig.15 The image sensor 200 shown in FIG. 2 additionally acquires a second image for an object in an environment where an illumination source exists through less than four second channels. The processor may be configured to perform parallax correction on the first image and the second image. For example, the processor may be configured to align the first image and the second image, or perform an affine transformation on the first image and the second image.

[0209] In operation 1620, the processor may be configured to estimate illumination spectrum data of the acquired first image using channel signals corresponding to eight or more first channels. In addition, in addition to the channel signals corresponding to eight or more first channels, the processor may be configured to additionally estimate illumination spectrum data using channel signals corresponding to less than four second channels.

[0210] The illumination spectrum data may include information related to the intensity of channel signals acquired from the multi-spectral image sensor according to wavelength. For example, the image acquisition device may be configured to generate an MSFA pattern corresponding to the estimated illumination spectrum data (e.g., Figure 8 MSFA pattern 820 shown in FIG.

[0211] In operation 1630, the image acquisition device may be configured to perform lens shading correction on the second image based on the estimated illumination spectrum data. The image acquisition device may be configured to generate a correction map for four or more second channels of the image sensor by performing linear interpolation on a correction table corresponding to the estimated illumination spectrum data. The image acquisition device may be configured to perform lens shading correction on all RAW images acquired from the multispectral image sensor or images according to channels using the correction map generated for each second channel.

[0212] The correction table may include first correction data for each first channel, and the first correction data may include a scaling factor, an offset, an incident angle, and / or a curve fitting coefficient corresponding to the estimated illumination spectrum data.

[0213] although Fig.16 Although not shown in the figure, the image acquisition device may be configured to further perform white balance correction on the second image on which lens shading correction has been performed. The method of performing white balance correction on the second image on which lens shading correction has been performed may be the same as that in reference Fig.14 The method described is the same as that of performing white balance correction on the first image on which lens shading correction has been performed.

[0214] Fig.17 is a block diagram of a configuration of an electronic device ED01 according to the embodiment.

[0215] refer to Fig.17In the network environment ED00, the electronic device ED01 can be configured to communicate with another electronic device ED02 through a first network ED98 (near field wireless communication network, etc.), or to communicate with another electronic device ED04 and / or a server ED08 through a second network ED99 (far field wireless communication network, etc.). The electronic device ED01 can be configured to communicate with the electronic device ED04 through the server ED08. The electronic device ED01 may include a processor ED20, a memory ED30, an input device ED50, an audio output device ED55, a display device ED60, an audio module ED70, a sensor module ED76, an interface ED77, a tactile module ED79, a camera module ED80, a power management module ED88, a battery ED89, a communication module ED90, a subscriber identification module ED96 and / or an antenna module ED97. In the electronic device ED01, some of the aforementioned components (for example, the display device ED60, etc.) may be omitted, or other components may be added. Some components may be implemented as an integrated circuit. For example, the sensor module ED76 (fingerprint sensor, iris sensor, brightness sensor, etc.) can be embedded and implemented in the display device ED60 (display, etc.).

[0216] The processor ED20 may be configured to control one or more other components (hardware, software components) connected to the processor ED20, and to perform various data processing or calculations by executing software such as program ED40. As part of the data processing or operation, the processor ED20 may be configured to load instructions and / or data received from other components (e.g., sensor module ED76, communication module ED90, etc.) into the volatile memory ED32, process the instructions and / or data stored in the volatile memory ED32, and store the resulting data in the non-volatile memory ED34. The processor ED20 may include a main processor ED21 (central processing unit, application processor, etc.), and an auxiliary processor ED23 (graphics processing unit, image signal processor, sensor hub processor, communication processor, etc.) that may operate independently of the main processor ED21 or simultaneously with the main processor ED21. The auxiliary processor ED23 may have less power consumption than the main processor ED21, and may be configured to perform a specified function.

[0217] While the main processor ED21 is in an inactive state (sleep state) or an active state (application execution state), the auxiliary processor ED23 can replace the main processor ED21 or control functions and / or states related to some components of the electronic device ED01 (display device ED60, sensor module ED76, communication module ED90, etc.) simultaneously with the main processor ED21. The auxiliary processor ED23 (image signal processor, communication processor, etc.) can also be implemented as a part of another component (camera module ED80, communication module ED90, etc.) that is functionally related to the auxiliary processor ED23.

[0218] The memory ED30 can store various types of data required by the components of the electronic device ED01 (processor ED20, sensor module ED76, etc.). The data may include, for example, software (e.g., program ED40, etc.), and input data and / or output data about instructions related to the software. The memory ED30 may include a volatile memory ED32 and / or a non-volatile memory ED34. The non-volatile memory ED34 may include an internal memory ED36 installed in a fixed manner in the electronic device ED01 and a detachable external memory ED38.

[0219] The program ED40 may be stored as software in the memory ED30 , and may include an operating system ED42 , middleware ED44 , and / or applications ED46 .

[0220] The input device ED50 may be configured to receive instructions and / or data from outside the electronic device ED01 (e.g., a user) to be used for components of the electronic device ED01 (e.g., processor ED20, etc.). The input device ED50 may include a microphone, a mouse, a keyboard, and / or a digital pen (stylus, etc.).

[0221] The audio output device ED55 can be configured to output an audio signal to the outside of the electronic device ED01. The audio output device ED55 can include a speaker and / or a receiver. The speaker can be used for general purposes, such as replaying multimedia or recording, and the receiver can be used to receive incoming calls. The receiver can be combined with the speaker as part of it, or can be implemented as a separate device.

[0222] The display device ED60 can provide information to the outside of the electronic device ED01 in a visual manner. The display device ED60 may include a display, a holographic device or a projector, and a control circuit configured to control the aforementioned devices. The display device ED60 may include a touch circuit configured to sense a touch and / or a sensor circuit configured to measure the magnitude of the force generated by the touch (pressure sensing, etc.).

[0223] The audio module ED70 may be configured to convert sound into an electrical signal, and vice versa. The audio module ED70 may be configured to obtain sound through the input device ED50, or output sound through the audio output device ED55 and / or a speaker and / or earphone of another electronic device (e.g., electronic device ED02) directly or wirelessly connected to the electronic device ED01.

[0224] The sensor module ED76 may be configured to sense the operating state (power, temperature, etc.) of the electronic device ED01 or the external environmental state (user's state, etc.), and generate an electrical signal and / or data value corresponding to the sensed state. The sensor module ED76 may include a gesture sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared sensor, a biometric sensor, a temperature sensor, a humidity sensor, and / or a brightness sensor.

[0225] The interface ED77 may support one or more specified protocols that can be used for direct connection or wireless connection between the electronic device ED01 and another electronic device (e.g., the electronic device ED02). The interface ED77 may include a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, and / or an audio interface.

[0226] The connection terminal ED78 may include a connector through which the electronic device ED01 may be physically connected to another electronic device (eg, the electronic device ED02, etc.). The connection terminal ED78 may include an HDMI connector, a USB connector, an SD card connector, and / or an audio connector (headphone connector, etc.).

[0227] The haptic module ED79 may be configured to convert an electrical signal into a mechanical stimulus (vibration, motion, etc.) or an electrical stimulus that can be recognized by a user through a sense of touch or motion. The haptic module ED79 may include a motor, a piezoelectric device, and / or an electrical stimulation device.

[0228] The camera module ED80 may be configured to capture a still image or a video. The camera module ED80 may include the above-mentioned image acquisition device 10 and / or the image acquisition device 20, and may further include a lens assembly, an image signal processor, and / or a flash. The lens assembly included in the camera module ED80 may be configured to collect light emitted from an object as an image capture object.

[0229] The power management module ED88 may be configured to manage power provided to the electronic device ED01. The power management module ED88 may be implemented as a part of a power management integrated circuit (PMIC).

[0230] The battery ED89 may be configured to provide power to the components of the electronic device ED01. The battery ED89 may include a disposable primary battery, a rechargeable secondary battery, and / or a fuel cell.

[0231] The communication module ED90 can support the establishment of a direct (wired) communication channel and / or a wireless communication channel between the electronic device ED01 and another electronic device (e.g., electronic device ED02, electronic device ED04, server ED08, etc.), and communicate through the established communication channel. The communication module ED90 may include one or more communication processors that operate independently of the processor ED20 (e.g., an application processor) and support direct communication and / or wireless communication. The communication module ED90 may include a wireless communication module ED92 (cellular communication module, near-field wireless communication module, global navigation satellite system (GNSS) communication module, etc.) and / or a wired communication module ED94 (local area network (LAN) communication module, power line communication module, etc.). In the aforementioned communication modules, the corresponding communication module can be configured to communicate with other electronic devices through a first network ED98 (a near-field communication network such as Bluetooth, WiFi Direct or Infrared Data Association (IrDA)) or a second network ED99 (a far-field communication network such as a cellular network, the Internet or a computer network (LAN, WAN, etc.)). The various types of communication modules mentioned above may be integrated into one component (e.g., a single chip) or multiple separate components (e.g., multiple chips). The wireless communication module ED92 may be configured to confirm and authenticate the electronic device ED01 in a communication network such as the first network ED98 and / or the second network ED99 using the subscriber information (e.g., IMSI) stored in the subscriber identification module ED96.

[0232] The antenna module ED97 can be configured to send signals and / or power to / receive signals and / or power from the outside (e.g., another electronic device). The antenna may include a radiator including a conductive pattern formed on a substrate (e.g., PCB). The antenna module ED97 may include one or more antennas. When the antenna module ED97 includes multiple antennas, an antenna suitable for communication used in a communication network such as a first network ED98 and / or a second network ED99 can be selected from the multiple antennas by the communication module ED90. Signals and / or power can be sent or received between the communication module ED90 and another electronic device through the selected antenna. In addition to the antenna, other components (e.g., RFIC) may also be included as part of the antenna module ED97.

[0233] Some components may be connected to each other, and signals (instructions, data, etc.) may be exchanged between peripheral devices through a communication scheme (bus, general purpose input and output (GPIO), serial peripheral interface (SPI), mobile industry processor interface (MIPI), etc.).

[0234] Instructions and data can be sent or received between the electronic device ED01 and the external electronic device ED04 through the server ED08 connected to the second network ED99. Other electronic devices (for example, electronic device ED02 and electronic device ED04) may include devices of the same or different types as the electronic device ED01. All or some operations performed by the electronic device ED01 may be performed by one or more devices among other electronic devices (for example, electronic device ED02, electronic device ED04 and server ED08). For example, when the electronic device ED01 needs to perform certain functions or services, the electronic device ED01 may request one or more other electronic devices to perform some or all of these functions or services instead of performing these functions or services autonomously. After receiving the request, one or more other electronic devices perform additional functions or services related to the request, and send the execution results to the electronic device ED01. To this end, cloud computing technology, distributed computing and / or client-server computing technology can be used.

[0235] Fig.18 Is set in Fig.17 8 is a block diagram of a camera module ED80 in an electronic device shown in FIG.

[0236] The camera module ED80 may include the above-described image acquisition device 10 and / or the image acquisition device 20, or may have a structure modified from the image acquisition device 10 and / or the image acquisition device 20. Fig.18 The camera module ED80 may include a lens assembly CM10, a flash CM20, an image sensor CM30, an image stabilizer CM40, a memory CM50 (eg, a buffer memory), and / or an image signal processor CM60.

[0237] The image sensor CM30 may include a multispectral image sensor 100 disposed in the aforementioned image acquisition device 10. The multispectral image sensor 100 may be configured to acquire an image corresponding to an object by converting light emitted or reflected from the object and transmitted through the lens assembly CM10 into an electrical signal. The multispectral image sensor 100 may be configured to acquire a hyperspectral image within an ultraviolet or infrared wavelength range.

[0238] In addition to the above-mentioned multispectral image sensor 100, the image sensor CM30 may also include one or more sensors selected from image sensors having different characteristics, such as another image sensor (eg, Figure 2Bor Fig.15 ), a black and white sensor, an IR sensor, or a UV sensor. Each sensor included in the image sensor CM30 may be implemented as a charge coupled device (CCD) sensor and / or a complementary metal oxide semiconductor (CMOS) sensor.

[0239] The lens assembly CM10 may be configured to collect light emitted from an object as an image capture object. The camera module ED80 may also include a plurality of lens assemblies CM10, and in this case, the camera module ED80 may include a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies CM10 may have the same lens features (e.g., quadrant, focal length, autofocus, F number, optical zoom), or may have other lens features. The lens assembly CM10 may include a wide-angle lens or a telephoto lens.

[0240] The lens assembly CM10 may be configured and / or focus-controlled so that two image sensors provided in the image sensor CM30 may form an optical image of an object at the same position.

[0241] The flash CM20 may be configured to emit light for enhancing light emitted or reflected from an object. The flash CM20 may include one or more light emitting diodes (red, green, and blue (RGB) LEDs, white LEDs, infrared LEDs, ultraviolet LEDs, etc.) and / or xenon lamps.

[0242] The image stabilizer CM40 may be configured to move one or more lenses included in the lens assembly CM10 or the multispectral image sensor 100 in response to the control of the operating characteristics of the multispectral image sensor 100 by the camera module ED80 or the electronic device ED01 (e.g., adjustment of the readout timing), thereby removing the negative effects caused by the movement. The image stabilizer CM40 may be configured to sense the movement of the camera module ED80 or the electronic device ED01 using a gyroscope sensor (not shown) or an acceleration sensor (not shown) arranged inside or outside the camera module ED80. The image stabilizer CM40 may be implemented optically.

[0243] The memory CM50 may be configured to store some or all of the data of the image acquired by the multispectral image sensor 100 for subsequent image processing operations. For example, when acquiring multiple images at a high rate, the acquired raw data (Bayer pattern data, high-resolution data, etc.) may be stored in the memory CM50, only low-resolution data may be displayed, and the raw data of the selected image (e.g., selected by the user) may be sent to the image signal processor CM60 using the low-resolution data. The memory CM50 may be integrated as the memory ED30 of the electronic device ED01, or may be configured as a memory that operates separately.

[0244] The image signal processor CM60 may be configured to perform image processing on an image acquired by the image sensor CM30 or image data stored in the memory CM50. The image signal processor CM60 may be configured to perform image processing on an image acquired by the image sensor CM30 or image data stored in the memory CM50. Figure 3 , Fig.10 or Fig.15 The processor 500 described is the same or different. Image processing may include the generation of depth maps, three-dimensional modeling, the generation of panoramas, the extraction of key points, the synthesis of images and / or compensation of images (noise reduction, adjustment of resolution, adjustment of brightness, blurring, sharpening, softening, etc.). The image signal processor CM60 may perform control (control of exposure time, control of readout timing, etc.) on the components (e.g., image sensor CM30) included in the camera module ED80. The image processed by the image signal processor CM60 may be stored again in the memory CM50 for additional processing or provided to an external component (e.g., memory ED30, display device ED60, electronic device ED02, electronic device ED04, or server ED08) of the camera module ED80. The image signal processor CM60 may be integrated into the processor ED20, or configured as a separate processor that operates independently of the processor ED20. When the image signal processor CM60 includes the processor ED20 and another processor, the image processed by the image signal processor CM60 may be additionally processed by the processor ED20 and then displayed by the display device ED60.

[0245] The electronic device ED01 may include a plurality of camera modules ED80 having different features or functions, respectively. In this case, one of the plurality of camera modules ED80 may include a wide-angle camera, and another of the plurality of camera modules ED80 may include a telephoto camera. Similarly, one of the plurality of camera modules ED80 may include a front camera, and another of the plurality of camera modules ED80 may include a rear camera.

[0246] Figures 19 to 28 are diagrams showing various examples of electronic devices to which the image acquisition apparatus 10 according to the exemplary embodiment is to be applied.

[0247] The image acquisition device 10 according to the embodiment can be applied to Fig.19 The mobile phone or smart phone 5100m shown in Fig. 20 The tablet computer or smart tablet computer 5200 shown in Fig.21 The digital camera or video camera 5300 shown in Fig. 22 Laptop 5400 or Fig.23 5500. For example, a smart phone 5100m or a smart tablet 5200 may include a plurality of high-resolution cameras, each of which includes a high-resolution image sensor. The high-resolution camera may be used to extract depth information of an object in an image, adjust the defocus of an image, or automatically identify an object in an image.

[0248] The image sensor 200 may be arranged near the image acquisition device 20 including the image sensor, and may be configured to support a front camera or a rear camera. The smart phone 5100m may include multiple rear cameras, and the multiple rear cameras may have the same or different focal lengths from each other. The image acquisition device may be configured to estimate the illumination spectrum data using the multispectral image sensor 100, and send the estimated illumination spectrum data to all of the multiple rear cameras. The image acquisition device 10 may be configured to perform the same white balance correction on images acquired from the multiple rear cameras using the estimated illumination spectrum data. Accordingly, the image acquisition device 10 may be configured to stably perform white balance correction on images acquired from the multiple rear cameras having different focal lengths.

[0249] In addition, the image acquisition device 10 can be applied to Fig.24 The smart refrigerator 5600 shown in Fig.25 The security camera 5700 shown in Fig.26 The robot 5800 shown in Fig. 27 The medical camera 5900 shown in , etc. For example, the smart refrigerator 5600 can be configured to automatically identify the food in the refrigerator using the image acquisition device 10, and notify the user through the smart phone whether there is a certain food in the refrigerator, the type of food put in or taken out, etc. The security camera 5700 can provide ultra-high resolution images, and can use high sensitivity to help users identify objects or people in the image in a dark environment. The robot 5800 can be introduced to disaster or industrial sites that cannot be directly reached by humans to provide high-resolution images. The medical camera 5900 can be configured to provide high-resolution images for diagnosis or surgery, and dynamically adjust the viewing angle.

[0250] In addition, the image acquisition device 10 can be applied to the vehicle 6000, such as Fig.28As shown. The vehicle 6000 may include a plurality of vehicle-mounted cameras 6010, 6020, 6030, and 6040 arranged at various locations, and each of the vehicle-mounted cameras 6010, 6020, 6030, and 6040 may include an image acquisition device according to an embodiment. The vehicle 6000 may be configured to provide a user with various information related to the interior or peripheral area of ​​the vehicle 6000 using the plurality of vehicle-mounted cameras 6010, 6020, 6030, and 6040, and may also be configured to automatically recognize objects or people in an image and provide information required for autonomous driving.

[0251] It should be understood that the embodiments described herein should be considered in a descriptive sense only and not for restrictive purposes. The description of features or aspects in each embodiment should generally be considered as applicable to other similar features or aspects in other embodiments. Although one or more embodiments have been described with reference to the accompanying drawings, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope defined by the appended claims.

Claims

1. An image acquisition device, comprising: a multispectral image sensor configured to acquire an image of at least one object in an environment in which at least one illumination source is present through eight or more channels with minimal overlap between the channels; as well as A processor is configured to estimate illumination spectrum data of an acquired image by using channel signals corresponding to the eight or more channels, and perform lens shading correction on the acquired image based on the estimated illumination spectrum data.

2. The image acquisition device according to claim 1, wherein: The processor is configured to generate a correction map for each of the eight or more channels by performing linear interpolation on a correction table corresponding to the estimated illumination spectral data.

3. The image acquisition device according to claim 2, in, The correction table includes correction data for each of the eight or more channels, and The correction data includes at least one of a scaling factor, an offset, an incident angle, and a curve fitting coefficient corresponding to the estimated illumination spectrum data.

4. The image acquisition device according to claim 2, wherein: The processor is configured to perform lens shading correction on all RAW images or images according to channels acquired from the multispectral image sensor by using the generated correction map.

5. The image acquisition device according to claim 1, in, The processor is configured to estimate the illumination spectral data based on eight or more basis vectors associated with reflectance characteristics of the at least one object included in the acquired image, a number of channels, and a number of pixels in the acquired image.

6. The image acquisition device according to claim 1, wherein: The processor is configured to estimate the lighting spectrum data by using a pre-trained machine learning model.

7. The image acquisition device according to claim 1, wherein: The processor is configured to segment the acquired image into a plurality of regions and estimate illumination spectral data for each of the plurality of regions.

8. The image acquisition device according to claim 1, wherein: The estimated illumination spectral data includes at least one of: spectral data including intensities of channel signals according to wavelengths, a color vector of an illumination source in a color space, a color temperature of the illumination source, and a predetermined illumination type corresponding to the illumination source.

9. The image acquisition device according to claim 1, wherein: The processor is configured to further perform white balance correction on the image on which the lens shading correction has been performed based on the estimated illumination spectrum data.

10. The image acquisition device according to claim 1, wherein: In the multispectral image sensor, a color calibration is performed on the at least one illumination source.

11. An image acquisition device, comprising: a multispectral image sensor configured to acquire a first image of at least one object in an environment in which at least one illumination source is present through eight or more first channels with minimal overlap between the channels; an image sensor configured to acquire, through less than four second channels, a second image of the at least one object in the environment in which the at least one illumination source is present; as well as A processor is configured to estimate illumination spectrum data of the first image by using channel signals corresponding to the eight or more first channels, and perform lens shading correction on the second image based on the estimated illumination spectrum data.

12. The image acquisition device according to claim 11, wherein: The processor is configured to generate a correction map for each of the second channels by performing linear interpolation on a correction table corresponding to the estimated illumination spectral data.

13. The image acquisition device according to claim 11, wherein: The processor is configured to perform parallax correction on the first image and the second image.

14. The image acquisition device according to claim 11, wherein: The processor is configured to further perform white balance correction on the second image on which the lens shading correction has been performed based on the estimated illumination spectrum data.

15. A method for acquiring an image, the method comprising: Acquiring a first image of at least one object in an environment in which at least one illumination source is present by using a multispectral image sensor through eight or more first channels with minimal overlap between the channels; estimating illumination spectrum data of the acquired first image by using channel signals corresponding to the eight or more first channels; as well as Based on the estimated illumination spectrum data, lens shading correction is performed on the first image, or a second image acquired by an image sensor different from the multi-spectral image sensor.

16. The method according to claim 15, in, The image sensor is configured to acquire the second image of the at least one object in the environment where the at least one illumination source is present through less than four second channels, and Performing the lens shading correction comprises: generating a correction map for each of the first channels or each of the second channels by performing linear interpolation on a correction table corresponding to the estimated illumination spectrum data; and Lens shading correction is performed on the first image or the second image by using the generated correction map.

17. The method according to claim 16, Performing the lens shading correction comprises: performing lens shading correction on all RAW images acquired from the multispectral image sensor or on images according to channels by using a correction map generated for each of the first channels, or Using the correction map generated for each of the second channels, lens shading correction is performed on all RAW images acquired from the multispectral image sensor or on images according to channels.

18. The method according to claim 16, in, The correction table includes first correction data for each of the first channels, or second correction data for each of the second channels, and The first correction data and the second correction data include at least one of a scaling factor, an offset, an incident angle, and a curve fitting coefficient corresponding to the estimated illumination spectrum data.

19. The method according to claim 15, further comprising: Parallax correction is performed on the first image and the second image.

20. The method of claim 15, further comprising: Based on the estimated illumination spectrum data, white balance correction is performed on the first image or the second image on which the lens shading correction has been performed.

Images